Evaluation metric example: RAG document relevance

Say goodbye to uncertainty in AI document retrieval!

This automation adds evaluation power to your n8n workflows by measuring how relevant the documents retrieved from your vector store are to user questions. It runs test datasets to score your AI’s accuracy, so you know exactly where it performs well—and where it needs improvement.

Ideal for anyone building AI search, Q&A, or knowledge management tools that rely on vector databases.

What it does:

🔎 Takes questions and compares retrieved documents for relevance

⚙️ Runs in parallel with your main workflow via an evaluation trigger

🧠 Uses AI to calculate a relevance metric, showing how well retrieved info matches queries

📈 Sends evaluation scores back to n8n for easy monitoring and improvement

💡 Supports initial document insertion into vector stores as part of setup

💰 Smartly skips scoring during regular runs to save on costs

✅ Setup guide & importable automation included

Need help setting it up? We offer full configuration and testing for a one-time fee.