Designing Production RAG Systems

While building several RAG systems for production use, here are the patterns that consistently work and the ones that don't.

Chunking Strategy Matters More Than You Think

The most common mistake is naive fixed-size chunking. A 512-token chunk with no overlap breaks logical paragraphs, loses context, and produces worse retrieval results than a well-designed recursive splitter.

I use a recursive character text splitter with:

This respects document structure while maintaining enough context for meaningful retrieval.

Hybrid Search Catches Edge Cases

Pure semantic search misses exact-match queries (product names, SKUs, version numbers). Adding BM25 keyword retrieval and fusing results with a weighted combination (0.7 semantic + 0.3 keyword) catches 95% of edge cases that either method alone would miss.

Re-ranking Is Not Optional

The jump from top-5 to top-1 precision is where the real quality gain lives. A cross-encoder re-ranker (e.g., BAAI/bge-reranker-v2-m3) consistently improves top-1 precision by 30-40% over pure embedding similarity.

Evaluation Strategy

I evaluate RAG systems on three dimensions:

1. **Retrieval precision**: Are the right documents being retrieved?

2. **Answer groundedness**: Is the answer supported by retrieved context?

3. **Citation accuracy**: Does the answer cite the correct source documents?

Each dimension needs its own test set. Groundedness and citation accuracy are best evaluated with LLM-as-judge, not human annotation at scale.