Building a RAG Pipeline That Admits What It Doesn't Know

The hardest part of building a retrieval-augmented chatbot isn't the retrieval — it's deciding what to do when retrieval comes back empty.

Chunking with overlap

Splitting content at paragraph boundaries with a small overlap between chunks preserves context that a hard cutoff would destroy. Frontmatter metadata (title, tags, source file) travels with every chunk so the final answer can cite where it came from.

chunk_size = 500 tokens
overlap = 50 tokens
metadata = { source, title, tags }

Relevance thresholds

A similarity search always returns something — the question is whether "something" is good enough to answer from. Setting a minimum cosine-similarity threshold, and having the assistant say "I don't have information about that" when nothing clears it, matters more for trustworthiness than any prompt-engineering trick.

Why the honest answer matters

A chatbot that confidently invents a project that doesn't exist is worse than no chatbot at all. The real engineering challenge turned out to be constraining the LLM to answer only from retrieved chunks and being explicit when nothing relevant came back — the vector search itself was the easy part.