Deleting the similarity threshold is what made retrieval work

The assistant would not answer questions it demonstrably had the answer to. Ask it something the documents covered plainly and it would say "I don't know." The chunk was in the corpus. The embedding existed. The question was clear.

I spent an embarrassing amount of time treating this as a model problem — rewriting the system prompt, trying different phrasings, considering a bigger model. It was a retrieval problem, and it was one line.

The line

// what I had
const candidates = scored
  .filter((chunk) => chunk.score >= SIMILARITY_THRESHOLD)
  .slice(0, 4);

A fixed cosine-similarity cutoff. Anything below it never reaches the prompt. It looks obviously sensible — why send the model chunks that are not similar enough? — and it is where the bug lived.

// what it is now
export function topK(
  queryVec: number[],
  items: EmbeddedChunk[],
  k = 4,
): ScoredChunk[] {
  const scored: ScoredChunk[] = items.map((item) => ({
    ...item,
    score: cosineSimilarity(queryVec, item.embedding),
  }));
 
  scored.sort((a, b) => b.score - a.score);
 
  return scored.slice(0, k);
}

No threshold. Score everything, sort, take the best four, hand them to the model. Relevance is judged by the thing that can read, not by a number chosen in advance.

Why a fixed cutoff is the wrong shape

The mistake is treating cosine similarity as if it were an absolute measure of relevance. It is not. It is a relative measure, and what counts as a high score depends on things that vary per query:

• Question length. A short question — "What is the refund window?" — produces a vector whose top matches sit noticeably lower than a long, wordy question does. Same corpus, same documents, different scale entirely.
• Vocabulary overlap. A question phrased in the document's own words scores high. The same question in a user's words scores lower, and is exactly as relevant.
• Corpus homogeneity. In a tightly-themed document set, everything scores moderately well and the spread between the best chunk and the tenth-best is tiny. In a diverse one, the spread is large.

A single number cannot be right across all of those. Whatever value you pick is too strict for some queries and too loose for others, and the failure is asymmetric in the worst possible way: too strict fails silently. The answer never arrives and nothing logs a reason. Too loose fails loudly — you see irrelevant citations and know immediately.

Top-N is not "no filter", it is a different filter

The thing that took me longest to accept is that removing the threshold does not mean sending the model junk. It means changing which filter does the work.

Top-N is scale-invariant. It does not care whether today's best chunk scored 0.61 or 0.88 — it cares which chunks are best relative to each other, which is the only question the embedding is actually equipped to answer.

The model is allowed to say the context is useless

Removing the threshold only works because the prompt does the rejecting:

const SYSTEM_PROMPT =
  "Answer ONLY using the provided context. If the answer isn't in the " +
  "context, say you don't know. Cite sources by filename.";

Three constraints, and each earns its place:

• "ONLY using the provided context" — closes the door on parametric knowledge, so an answer that sounds right but is not in the documents does not get through.
• "say you don't know" — makes abstention a legal output. Without this, a model handed four marginally relevant chunks will construct something from them, because producing an answer is the implicit default.
• "cite sources by filename" — makes the retrieval auditable from the answer itself. When something is wrong, you can see immediately whether it was the retrieval or the reading.

That second one is what makes top-N safe. If the four best chunks genuinely do not contain the answer, the model is instructed to say so — which is precisely what the threshold was trying to achieve, done at the layer that can actually tell.

The threshold and the prompt were solving the same problem, and only one of them could read the text.

Verify it with questions you know are out of scope

The way to check you have not just traded silence for hallucination is to ask deliberately out-of-scope questions and confirm the abstention still fires.

I keep a handful in the test set: questions that are plausible for the domain, phrased confidently, and answered nowhere in the corpus. If any of them get a confident answer, the constraint has stopped working — usually because someone loosened the system prompt or raised n far enough that the context turned into noise.

That test is cheap and it is the one that would have caught the original bug from the other direction.

The retrieval maths is about forty lines

Worth saying, because the ecosystem does not advertise it: none of this needed a framework. The whole retrieval layer is pure, dependency-free TypeScript — chunking with configurable overlap, cosine similarity, top-N ranking — and it is unit tested without a network call:

describe("cosineSimilarity", () => {
  it("returns 1 for identical direction", () => { /* ... */ });
  it("returns 0 for orthogonal vectors", () => { /* ... */ });
  it("returns 0 when either vector is all zeros", () => { /* ... */ });
  it("throws on length mismatch", () => { /* ... */ });
});

Writing it by hand is what made the threshold bug findable. When retrieval is a function you own that takes vectors and returns a sorted list, you can put a breakpoint in it. When it is a retriever object configured with a score_threshold keyword, the same bug is a config value you have no particular reason to doubt.

What transfers

• A tuning constant that silently drops data is a bug generator. Prefer filters that fail visibly.
• Cosine similarity is a ranking signal, not a relevance measure. Compare scores to each other, not to a constant.
• Push the relevance judgement to the layer that can read. The model can tell whether a paragraph answers a question. A float cannot.

The library is src/lib/rag.ts — small enough to read in one sitting, which was the point.