u/Far-Catch-3324

RAG apps can look fine while using the wrong documents.

For example, the source an AI pulled from changed from the refund policy to some random pricing page.

So, I built a small open-source tool called `rag-contract` to catch that.

The idea is we save a few questions and the documents they should find. Before new code gets merged, we run those questions again. If the right document is missing or buried too low, the check fails.

Curious if other people building RAG systems have hit this. I also look forward to open-source contributors. Together, I believe we can truly make an impact in RAG reliability with this project if it goes far.

Repo: https://github.com/volkthienpreecha/rag-contract
(for usage: pip install rag-contract)