The pipelines, indexes and knowledge graphs that everything else depends on. This is where most AI projects are actually won or lost, and where the least attention usually goes.
Before a model can answer anything useful, something has to get your data out of the systems it lives in, into a shape that can be searched, kept current as it changes, and constrained so the wrong person cannot reach the wrong record. That is most of the work on most projects, and it is the part that is invisible in a demo.
The answer is in a help centre, a shared drive, a wiki, a ticket history and one long email thread, and three of those contradict each other. Deciding which source wins is an organisational decision that has to be made before it can be encoded, and it is usually the first thing a project surfaces.
Contracts, invoices, statements of work, shipping documents and clinical notes carry meaning in tables, headers, footnotes and position on the page. Naive text extraction turns a table into a stream of numbers with no columns. Layout aware extraction is the difference between a working system and a plausible looking one.
A policy is updated and the assistant keeps citing last year's version for two months. Freshness needs incremental reindexing driven by change events, not a full rebuild on a nightly cron that quietly starts failing.
The moment you copy content into a search index you have created a second access control surface. If the index does not carry and enforce the source permissions, you have built an efficient way to leak documents across teams.
Which entities are related to which, who is party to what, which shipment belongs to which order. Questions about relationships are answered badly by passage retrieval and well by a graph. Knowing which questions are which shapes the whole architecture.
Once content flows into embeddings, logs, caches and prompts, personal data is in more places than the original system of record. Redaction, retention and residency have to be designed at ingestion, because retrofitting them across an index and a log store is genuinely painful.
Connectors to your systems of record with incremental updates driven by change events, so the index reflects reality rather than the state of the world on the day it was built.
CDC, incremental indexingTables, headers, footnotes and multi column layouts preserved through extraction, with structure aware chunking per document type rather than one fixed size for everything.
Document parsing, structured extractionHybrid semantic and keyword indexes with metadata filters, tenancy boundaries and permission tags carried from the source, so access control survives the copy.
Vector and hybrid search, ACL propagationEntity and relationship extraction into a graph, queried alongside passage retrieval, for the questions that are about connections rather than content.
Entity resolution, graph retrievalPersonal and sensitive data classified, redacted or tokenised as it enters, with residency respected across index, cache and log. Designed in at the boundary, where it is cheap.
Data protection by designEvery indexed passage traceable to its source document, version and ingestion run, so a wrong answer can be traced to the content that caused it and fixed at source.
Lineage, provenanceA thirty minute call. Tell us where your content lives and we will tell you honestly what state it needs to be in before any of the rest is worth building.