Building Production RAG: From Document Ingestion to Cited Answers at 50K Queries/Day

Everyone's building RAG. Most of it doesn't work well in production. The gap between a demo that retrieves relevant chunks and a system that reliably answers domain-specific questions with source citations at enterprise scale is enormous. Here's how we closed that gap building Woby, the AI assistant behind Wokay's unified workspace.

Why Most RAG Demos Fall Apart in Production

The standard tutorial RAG pipeline — chunk documents, embed them, retrieve top-k, stuff into prompt — works for demos. It breaks in production for three reasons:

1. Chunking destroys context. A fixed 512-token split in the middle of a policy document produces chunks that are individually meaningless.
2. Semantic search isn't enough. "What's our refund policy for enterprise clients?" requires metadata filtering (client tier = enterprise), not just embedding similarity.
3. No citation means no trust. Enterprise users won't trust an answer unless they can verify the source document and page number.

Document Ingestion Pipeline

Woby handles PDFs, Notion exports, Google Docs, Confluence pages, and Slack message archives. Each source has its own parser, but they all output the same intermediate format: a list of sections with metadata.

Chunking Strategy: Semantic Boundaries First

We chunk at semantic boundaries, not token counts. The pipeline first identifies section breaks using heading detection and paragraph clustering. Within a section, we split at paragraph boundaries if the section exceeds 800 tokens. Each chunk retains the section title and preceding context as a preamble.

The key insight: every chunk should be independently meaningful. If you read a chunk in isolation and can't understand what it's about without seeing the surrounding text, your retrieval will suffer.

We also generate a summary embedding per document. When a query is broad ("tell me about our refund policies"), the document-level embedding catches it even if no individual chunk is a perfect match.

Hybrid Retrieval: Vector + Metadata Filtering

Pure vector search over embeddings works for open-ended questions. But enterprise queries are often scoped: "What's the SLA for Tier 2 customers?" needs to filter to documents tagged with tier=2 before doing semantic similarity.

Our retrieval pipeline runs in two stages:

1. Metadata pre-filter — extract structured filters from the query using a lightweight Haiku call (doc_type, audience, date range)
2. Vector search within the filtered set — cosine similarity over the remaining chunks, top-8 retrieved

This two-stage approach reduced irrelevant retrievals by 40% compared to pure vector search in our A/B tests. The metadata filter is cheap — it's a database index scan, not an LLM call.

Citation-Grounded Generation

Every answer Woby generates must cite its sources. The system prompt instructs the model to reference chunks by their source ID, and we post-process the response to convert these into clickable links that jump to the exact page in the original document.

We also run a lightweight faithfulness check: does the answer contain claims that aren't grounded in any retrieved chunk? If so, the response is flagged and either regenerated with stricter instructions or escalated to a human reviewer.

Scale: 50K Queries Per Day

At peak, Woby handles ~50K AI-assisted queries per day across all Wokay workspaces. The bottleneck isn't the LLM — it's embedding generation during document ingestion. We batch embedding requests and process them asynchronously using a worker queue, so uploading a 200-page PDF doesn't block the user.

What I'd Do Differently

Invest in chunk quality evaluation earlier. We spent weeks tuning retrieval parameters when the real problem was bad chunks. Build a chunk quality scorer that checks for standalone comprehensibility before worrying about embedding models.

Metadata extraction is worth the extra Haiku call. Extracting structured tags from documents during ingestion (not just at query time) makes retrieval dramatically more precise. The cost is negligible.

Don't skip the faithfulness check. The 3% of responses that hallucinate are the ones that erode user trust the fastest. A post-generation faithfulness filter is cheap insurance.