Building a RAG Pipeline That Admits What It Doesn't Know

The hardest part of building a retrieval-augmented chatbot isn't the retrieval — it's deciding what to do when retrieval comes back empty.

Chunking with overlap

Splitting content at paragraph boundaries with a small overlap between chunks preserves context that a hard cutoff would destroy. Frontmatter metadata (title, tags, source file) travels with every chunk so the final answer can cite where it came from.

chunk_size = 500 tokens
overlap = 50 tokens
metadata = { source, title, tags }

Relevance thresholds

A similarity search always returns something — the question is whether "something" is good enough to answer from. Setting a minimum cosine-similarity threshold, and having the assistant say "I don't have information about that" when nothing clears it, matters more for trustworthiness than any prompt-engineering trick.

The honest answer is the product

A chatbot that confidently invents a project that doesn't exist is worse than no chatbot at all. Constraining the LLM to answer only from retrieved chunks — and being explicit when nothing relevant was found — is the actual engineering challenge here, not the vector search itself.