A stateless MCP tool re-embeds its corpus on every call

I wanted an agent to be able to search text by meaning mid-reasoning, without an embedding API in the loop. A local sentence encoder, four tools, an MCP server — a weekend's work.

The first version took 1,040 ms per search over 200 documents. Not because the model is slow. Because of something structural about MCP tools that I had not thought through.

The tool signature is the problem

def semantic_search(query: str, documents: list[str], k: int = 5): ...

Look at where the documents come from. They arrive as an argument, on every call. MCP tools are stateless — the server holds no session, and the host sends the whole payload each time.

So the obvious implementation embeds all 200 documents, embeds the query, and takes the top k. Then the agent asks a second question about the same 200 documents, and it embeds all 200 again.

The realistic agent pattern is one corpus, many questions. The naive implementation is quadratic in exactly the dimension that matters.

The fix is four lines

def encode(self, texts: list[str]) -> np.ndarray:
    missing = [t for t in dict.fromkeys(texts) if t not in self._cache]
 
    if missing:
        model = self._load()
        fresh = model.encode(missing, normalize_embeddings=True)
        for text, vector in zip(missing, fresh, strict=True):
            self._cache[text] = vector
        while len(self._cache) > self.cache_size:
            self._cache.popitem(last=False)
    ...

Keyed on the exact string. Embeddings are deterministic for a given model, so a cache hit is not an approximation — it is the same vector, bit for bit. There is no accuracy tradeoff to reason about, which is unusual and worth noticing.

Measured on CPU with all-MiniLM-L6-v2, 200 documents, median of 20 calls. The first call still pays full price; every subsequent one encodes only the query.

Two details that are not decoration:

dict.fromkeys(texts) rather than set(texts) — it de-duplicates while preserving order, so a batch containing the same string twice encodes it once and the output rows still line up with the input.

The bound. An unbounded cache is a slow leak in a long-lived stdio server that an agent keeps feeding new text to. Least-recently-used, 4096 entries, evicted from the front.

This is the same shape as prompt caching

The pattern generalises, and I have now hit it twice in a month.

One cache_control breakpoint cut an agent loop's input cost 47% because the loop re-sent an identical system prompt and tool list on every iteration. Here a local cache cuts search latency 893× because the tool re-encodes an identical corpus on every call.

Both are the same observation: in a loop, the expensive part of the call is usually the part that did not change. The work is finding the boundary between what varies and what does not, and putting the cache exactly there.

For prompt caching the boundary is the largest stable prefix. For an embedding tool it is the individual document, because the set changes between calls but the members do not.

Lazy loading, for a different reason than usual

def _load(self):
    if self._model is None:
        from sentence_transformers import SentenceTransformer
        self._model = SentenceTransformer(self.model_name)
    return self._model

The model takes about 21 seconds to load on this machine. A stdio MCP server is launched by the host, often at startup, and may never have a tool called on it during a session.

Loading eagerly means every host launch pays 21 seconds for a capability the user might not use. Loading lazily means the server starts instantly, advertises its tools, and pays only if something actually calls one.

Worth stating plainly because the usual argument for lazy loading is memory, and here it is not — it is that an MCP server's startup cost is paid by a user who has not asked for anything yet.

The tool description is doing work you might not notice

description=(
    "Rank a list of documents by how well each answers a question, using "
    "meaning rather than shared keywords. Use this whenever you have a pile "
    "of text and a question about it: finding the relevant passage in notes, "
    "picking which of several answers fits, or narrowing a long list before "
    "reading it. Runs locally, so it is free to call and safe to call "
    "repeatedly."
)

That last sentence is the one I would defend hardest. It tells the model something true about the tool's cost, and cost changes willingness. An agent reasons differently about a tool it believes is expensive — it batches, it hesitates, it tries to get by without one more call.

Making the tool free and then saying so is what turns semantic search from a resource to be rationed into an operation the agent uses freely mid-reasoning. That is the actual point of running it locally; the money saved is rounding error.

Testing something that needs a 90 MB model

The encoder sits behind a four-line interface:

class Encoder(Protocol):
    dimensions: int
    def encode(self, texts: list[str]) -> np.ndarray: ...

Deliberately narrower than anything sentence-transformers offers — an interface shaped like the vendor SDK is a second SDK you now maintain, whereas one shaped like your own call site cannot drift.

The test implementation is a signed hashed bag-of-words. It has no semantic understanding at all — "car" and "automobile" are orthogonal to it — and that is fine, because the suite is asserting properties of the arithmetic: that top-k orders correctly, that MMR de-duplicates, that clustering converges and is deterministic given a seed.

36 tests, 0.42 seconds, no network. CI does not depend on a model host being reachable.

One thing that changed under me

The MCP Python SDK is now on v2, where FastMCP was renamed to MCPServer and moved:

# v1
from mcp.server.fastmcp import FastMCP
# v2
from mcp.server.mcpserver import MCPServer

The error message is unusually good — it names the rename, links the migration guide, and tells you to pin mcp<2 if you would rather not move. I moved. If you are following an MCP tutorial written before the rename, that import is why nothing works.

What transfers

• Read your tool signature for what it forces you to recompute. Statelessness is a protocol property, and it has a cost that only shows up under the real usage pattern.
• Cache at the boundary between what varies and what does not. Here that is the individual document, not the request.
• Bound every cache in a long-lived server.
• Lazy-load anything expensive in a stdio server, because startup is paid by someone who has not asked for anything.
• Tell the model what a tool costs. It changes how freely the tool gets used.

The server is mcp-local-semantics — four tools, python scripts/bench.py reproduces the table, and MCP_SEMANTICS_ENCODER=hash runs the whole thing without torch if you want to smoke-test a host connection first.