Distributed Indexing on the Edge: Performing RAG Without Moving Data to a Cloud Vector DB

The short answer

Running RAG entirely on the appliance means the embedding model, the vector index, and the retrieval logic all execute inside the same sealed hardware boundary as the LLM. Raw documents, chunked text, and the resulting vectors never leave the box to reach a hosted vector database. This is the architecture Forge uses by default: FAISS or a comparable ANN index sits on local disk, embeddings come from a model running on the appliance's own GPUs, and every query resolves through a retrieval loop with no outbound connection. For defense and regulated-industry deployments, that's not a performance optimization. It's the difference between a system that can sit on a classified or air-gapped network and one that architecturally cannot.

Why cloud vector DBs break the air gap

Most production RAG stacks are built around a managed vector database — Pinecone, Weaviate Cloud, Qdrant Cloud, or a vector-enabled extension on a hosted Postgres instance. The pattern is consistent: an application server chunks documents, calls an embedding API, and upserts the resulting vectors into a remote index. Retrieval at query time sends the user's query text (or its embedding) back out to that same remote service.

Each of those hops is a data egress point. Embeddings aren't the anonymized artifacts people sometimes assume they are. Dense vector representations can be inverted to recover substantial portions of the source text, a fact demonstrated repeatedly in embedding-inversion research since 2020. For a defense contractor working with export-controlled technical data, or a hospital system handling clinical notes, sending embeddings to a third-party API is functionally equivalent to sending the underlying document, whatever the vendor's data processing agreement says about not training on customer data. That agreement governs intent. It doesn't change what crossed the network boundary, and it does nothing for an environment with no network boundary to cross in the first place, such as a SCIF or a shipboard system operating under IL5/IL6 handling requirements.

There's also a simpler operational problem: cloud vector DBs assume connectivity. An appliance deployed on a forward-operating network, a submarine, or an isolated plant floor doesn't have a reliable path to any external service, managed or not. If the vector store lives outside the appliance, retrieval either fails when disconnected or requires periodic sync windows that reopen the exact exposure the deployment was built to avoid.

What local indexing actually requires

Building a vector index that runs entirely on-appliance means solving three problems without external infrastructure: embedding generation, index storage, and index maintenance as documents change.

Embedding generation happens with a model resident on the appliance, typically a smaller, purpose-built embedding model in the 100M–1B parameter range rather than the LLM itself, since embedding and generation have different compute profiles and can run on separate GPU allocations without contention. This model is fixed at deployment and versioned alongside the rest of the appliance image, so re-indexing after a model update is a deliberate, auditable operation. It doesn't happen silently the way it can with hosted embedding APIs, where an unannounced model version bump quietly invalidates the geometry of an existing index — a real failure mode, and one we've seen catch teams off guard.

Index storage uses an on-disk ANN structure, usually FAISS's HNSW or IVF variants, held on encrypted local storage. Because there's no network hop between the retriever and the index, latency is bound by disk I/O and local GPU throughput rather than API round-trip time. In practice, that makes local retrieval faster than most cloud-hosted alternatives once the corpus is warm in memory.

Handling index growth and staleness

The harder problem is keeping the index current without a managed service doing background maintenance. A cloud vector DB handles sharding, re-balancing, and incremental upserts as a service; on the appliance, that logic has to be part of the product rather than someone else's infrastructure. Forge handles this with an incremental indexing job that watches designated document stores for changes and re-embeds only the delta, rather than requiring a full corpus re-index on every update. For corpora in the tens of thousands of documents, full re-indexing on a modern GPU takes minutes, not hours. That's fast enough that incremental updates end up being the better default rather than a hard requirement.

Retrieval quality without a managed service

The natural objection: managed vector databases exist because they solve real problems, including approximate nearest neighbor search at scale, hybrid dense/sparse retrieval, and metadata filtering. True, but none of that is unique to a hosted service. FAISS, the library underlying most managed vector DBs' actual search implementation, is itself a local library. Meta open-sourced it in 2017, and it remains the reference implementation most commercial vector databases wrap with a hosted API layer and a billing system. Running it locally doesn't sacrifice the algorithm. It removes the network layer wrapped around it.

Hybrid retrieval, combining dense vector search with BM25 or another sparse keyword method, works the same way on-appliance, since both retrieval paths operate against local indexes. Metadata filtering, chunk-level access control, and re-ranking all run as local post-processing steps on the retrieved candidate set before anything reaches the LLM's context window.

The one genuine tradeoff is horizontal scale. A cloud vector DB can shard a billion-vector index across a cluster elastically. An appliance is bounded by the hardware in the chassis. For most Forge deployments, technical documentation, contract archives, classified reporting sets in the range of thousands to low millions of documents, a single appliance's local NVMe and GPU memory comfortably hold the index. Organizations with genuinely web-scale retrieval needs and no data sovereignty constraint are a different customer than the one buying a sealed appliance, and that's fine.

Where this fits in a governance posture

None of this substitutes for a certified compliance program, and Element 31 doesn't claim it does. What it is: an architecture where the vector index, like the model weights and the inference engine, is a component of the sealed appliance rather than a dependency on an external service. For governance teams evaluating RAG systems against data residency requirements, export control regimes like ITAR, or internal policies that prohibit any transit of source material off-premises, the relevant question isn't how a vendor encrypts data in transit to their vector database. It's whether that transit has to happen at all. On Forge, it doesn't.