"Add a cloud fallback" sounds like a configuration flag. For a retrieval system it is closer to a data-integrity problem wearing a configuration flag as a disguise.
Text generation is stateless — if one provider is slow, use another, and nobody can tell afterwards. Embedding is not. Embedding writes into a table you will query for months, and a query is only meaningful if the encoder that embedded the query agrees with the encoder that embedded the rows.
Fantom MCP Server's cloud integration is mostly the machinery required to make that safe.
Five roles, routed separately
There is no single "use the cloud" switch, because the five inference roles have genuinely different requirements. Each is routed on its own policy:
The four policies:
| Policy | Behaviour |
|---|---|
local |
Local GPU and sidecar hosts only. Nothing leaves the network. |
cloud |
OpenRouter only. No GPU required at all. |
aggregate |
Both, in one fan-out pool, pulling from the same queue. (default) |
backup |
Local first, cloud held in reserve. |
aggregate is the default because it is the one that uses hardware you have already paid for while still absorbing a burst.
The gate that makes this safe
Under aggregate, local and cloud vectors land in the same table, interleaved, during the same run. If the two serving stacks do not produce interchangeable vectors, cosine distance stops being comparable and retrieval degrades silently — with no way to tell which rows came from where.
So compatibility is not a warning. It is a hard precondition: a provider that has not passed must never serve a single text.
The check is deliberately unsubtle: embed the same probe texts through both routes, take pairwise cosine similarity on every corresponding pair, require the minimum to clear 0.99, and require the returned dimension to be exactly the configured width.
The interesting number is the one that failed. The measured cosine between the local model and the cloud model is 0.976 — close enough to look fine in a spot check, comfortably under the floor, and precisely why the cloud policy exists. Those two encoders are not interchangeable, so if you want to use the cloud one, the entire index has to be built from it. That is not a workaround; that is the policy doing its job.
Which means the reference is not always local. Under the three policies where GPUs serve, the reference must be local, because agreement with the rows already in the table is exactly the property being tested. Under cloud, comparing against a local model would be meaningless — no local vector is ever written — so the reference becomes another cloud provider, and the gate is otherwise unchanged. It is never disabled.
The limit, stated rather than buried
When exactly one cloud provider exists there is no peer, and cross-provider consistency is vacuous: one provider cannot disagree with itself, and it alone writes every row. The gate then checks dimension and the upstream pin, and nothing else.
That is a real reduction in coverage. A single pinned upstream can still change its serving stack behind the same pin over time — a host redeploying a different quantisation of the same model slug — and under a sole-provider cloud regime nothing catches it. Worth knowing if you run that configuration.
Why embedding pins and reranking does not
Two OpenRouter upstreams serving the same model slug do not guarantee identical vectors. An unpinned embedding route can therefore split one vector space between requests, which is the same corruption by a quieter path. So every embedding route must name its upstream provider, and the system fails closed if it does not.
Reranking is stateless — it scores a candidate list and writes nothing — so it needs no pin.
One budget, held above every provider
Here is the trap that shapes the whole cloud path: OpenRouter rate-limits per key, not per caller.
Every sidecar gets the same key. Three providers each capped at eight in-flight requests is twenty-four requests against one shared budget. A per-provider cap cannot protect the limit — the cap has to live above all of them.
A few details in there are worth pulling out.
The permit unit is concurrency, not requests per minute. A discovered rate_limit is an RPM figure, so it gets converted using the pool's own measured round-trip latency — Little's Law — which means it re-derives itself as latency drifts rather than being pinned to a number from boot time.
A believed RPM is also enforced directly, as a sliding-window gate, so a latency estimate that is briefly too high cannot overrun the account. But that second gate is only armed for an RPM anyone trusts. A funded account whose key reports a degenerate rate limit — or none at all, which is the modern shape — is sized from its credit balance instead and gets no window gate. Arming it on a vestigial "1 per minute" was strictly worse: it held the entire cloud fan-out to one request a minute.
Backoff is adaptive, never an immediate retry. A 429 halves the effective budget and starts a jittered cooldown; a sustained clean window ramps it back one permit at a time toward the ceiling.
The defaults are deliberately timid. Four permits when nothing could be discovered, because assuming headroom you could not read is how you get a 429 storm and a half-written table. And a hard ceiling of 256, because a typo in an override box should not saturate the account.
Local GPUs are exempt entirely. Work is pulled per provider, so a blocked cloud provider simply stops pulling and the local ones keep going. A cloud rate limit slows the cloud; it does not idle your hardware.
Key handling
The OpenRouter key is write-only from Fantom MCP Server's perspective. It arrives on an admin request, goes straight out over the WebSocket frame to the sidecar that will use it, and is never written to config, never logged, never returned by any endpoint, and never held past the call.
What gets persisted is only what is safe to persist: which roles have a model mapped, which upstream is pinned, the routing modes, and the fact that a push happened. Re-pushing without a key is the normal case for changing a mode or a model — the sidecar merges a partial push onto the entry it already holds.
What runs where, by default
- Reranker —
qwen/qwen3-reranker-8b - Code assistant —
poolside/laguna-s-2.1 - RLM sandbox —
deepseek/deepseek-v4-flash, with a fallback mode of local-only
Embedding models are chosen per deployment, because that choice is the one that determines what your index is. The code vector table is a single space at 2560 dimensions; everything above exists to keep it that way.
The point
None of this makes the cloud faster. What it buys is the ability to say something specific: that every row in the index was written by an encoder proven interchangeable with the one that will embed your query, that no single project can eat the account, and that when the cloud throttles, the GPUs keep working.
"Add a cloud fallback" really is a configuration flag. Everything in this post is what has to be true underneath before that flag is safe to flip.
Fantom MCP Server is source-available at github.com/Project-SandStar/FantomMcpServer. Previous posts: the Sidecar fleet and the RLM · cross-language code intelligence.