Reference Architecture: Document Ingestion Router¶
RAG pipelines generally treat every document the same way: chunk it, embed it, index it. That works fine until you're reasoning over a corpus where a controlled engineering spec, a Slack thread, and a raw telemetry log all end up carrying equal weight in retrieval — even though they carry very different truth value and very different shelf life.
The Document Ingestion Router (MIT license, open source) is a reference architecture worth studying for exactly this problem. A few of its ideas directly inform how ctxEco thinks about ingestion.
Truth-value classification¶
Documents are classified before extraction, not after:
| Class | Truth value | Decay | Examples |
|---|---|---|---|
| A | Immutable truth | Very slow | Specs, standards, manuals |
| B | Ephemeral stream | Moderate | Emails, slide decks, wiki pages |
| C | Operational pulse | Fast | Logs, CSVs, telemetry |
The insight isn't the three-bucket taxonomy specifically — it's that decay rate is a first-class retrieval signal, not an afterthought. A five-year-old engineering standard and a five-minute-old log line shouldn't compete on equal footing just because they both matched the query embedding. Retrieval that's aware of how fast a document's truth value decays ranks meaningfully better than retrieval that treats every chunk as equally durable.
Hybrid retrieval, fused¶
The project's search layer (TriSearch™) combines keyword search, vector search, and a graph-knowledge layer built from named-entity co-occurrence, merging the three via Reciprocal Rank Fusion. Keyword search alone misses semantic matches; vector search alone misses exact identifiers (part numbers, error codes, case IDs) that keyword search finds trivially. Fusing multiple retrieval modalities — rather than picking one and living with its blind spots — is a pattern worth adopting regardless of which vector store or SQL engine sits underneath.
Access control as a first-class design constraint¶
The project implements resource-level access scoping (private / team / project / tenant) enforced at the query layer, and maps its controls explicitly to recognized frameworks (NIST AI RMF, FedRAMP). Writing that mapping down as part of the architecture — not as an afterthought compliance document — is itself a useful habit: it forces the access model to be precise enough to state which control it satisfies, not just vaguely "secure."
Where the reference stops and the platform starts¶
ctxEco doesn't run this project's code directly — the platform's own tenant-isolation, policy-enforcement, and retrieval-substrate choices follow a different architecture. What we did take: the classification + decay concept, the routed-extraction-by-document-type pattern, and hybrid retrieval with rank fusion, all reimplemented against ctxEco's own stack. That's the right way to use a strong reference architecture — study the ideas, verify them against your own constraints, and build the version that actually fits your system.
If you're studying retrieval architecture, the DIR repository and its published paper are worth reading directly: github.com/derekbmoore/documentIngestionRouter.