How I built a production RAG pipeline in 48 hours

A client came to me on a Tuesday evening with a problem: their support team was drowning in repetitive questions that could be answered by their own documentation — thousands of pages of it. They'd heard about RAG and wanted a demo by Thursday. I said yes, knowing it was tight. Here's exactly what happened and what I learned.

This isn't a tutorial about RAG theory. There are plenty of those. This is about the decisions that matter when shipping something real — and fast — under constraints that don't care about your ideal architecture.

The brief

Five thousand pages of internal documentation, a support team fielding 400+ tickets a week, and a deadline of 48 hours for a working demo. The client wanted to ask questions in plain English and get precise, cited answers from their own knowledge base. Simple to say, non-trivial to get right.

The biggest mistake in RAG systems isn't the retrieval — it's trusting the retrieval. You need ground truth, not just semantic similarity.

What I actually built

The stack: Python, LangChain, ChromaDB for the vector store, OpenAI embeddings, and a thin FastAPI wrapper. The frontend was a single HTML file with streaming fetch. Deliberately minimal — complexity is the enemy of a 48-hour deadline.

The three decisions that mattered most

Every RAG project lives or dies on chunking strategy, embedding model choice, and re-ranking. Most people get the first wrong, almost everyone skips the third.

• Chunking: Fixed-size chunks are almost always wrong for structured documents. I used a hybrid: section-level splits for legal/policy docs, sentence-level for FAQs. Overlap of 15% on all chunks.
• Embeddings: text-embedding-3-small was 5× cheaper and within 3% accuracy of large for this domain. Budget matters in production.
• Re-ranking: Added a Cohere re-ranker in the last six hours. Jumped accuracy from 71% to 89% on our test set. It's never optional if you care about quality.

What went wrong

The document ingestion pipeline failed silently on PDFs with complex table layouts — a class of document the client hadn't mentioned but represented 20% of the knowledge base. I caught it at hour 36 during final testing. Two hours of emergency parsing fixes later, we were back on track.

The lesson: always audit your input data before you start building the pipeline around it. What you think you're ingesting and what you're actually ingesting are almost always different things.

The demo and what came next

The demo landed. The client extended the engagement to a full production build over three months. The 48-hour prototype became the foundation — almost unchanged in its core architecture. That's the other lesson: build the right thing fast, then make it robust. Don't build the robust thing slowly and never ship.

If you're considering a similar build, feel free to reach out or read more on how I approach production ML. The architecture decisions change, but the principles don't.