Deepeval

DeepEval is an open-source framework that brings unit testing to LLM applications. It provides metrics like faithfulness, answer relevancy, hallucination detection, contextual recall, and toxicity that integrate directly with pytest for CI/CD pipeline testing.

Key use cases include:

• RAG evaluation — measuring faithfulness, context precision, and answer relevancy
• Hallucination detection — verifying LLM outputs are grounded in provided context
• Toxicity testing — screening outputs for harmful or inappropriate content
• Regression testing — catching quality degradation when prompts or models change
• CI/CD integration — running LLM tests automatically in your deployment pipeline with pytest
• Benchmarking — comparing models and prompts with standardized metrics

DeepEval is used by developers who want to test LLM applications with the same rigor as traditional software testing. The pytest integration means LLM evaluation slots naturally into existing testing workflows and CI/CD pipelines.

All metrics provide explanations for their scores, making it easy to understand why a test passed or failed.

1. Install DeepEval:

   pip install deepeval

2. Set your OpenAI key (used for metric computation):

   export OPENAI_API_KEY='sk-...'

3. Write a test:

   from deepeval import assert_test
   from deepeval.test_case import LLMTestCase
   from deepeval.metrics import FaithfulnessMetric
   
   test_case = LLMTestCase(
       input='What is the capital of France?',
       actual_output='Paris is the capital of France.',
       retrieval_context=['France is a country in Europe. Its capital is Paris.']
   )
   metric = FaithfulnessMetric(threshold=0.7)
   assert_test(test_case, [metric])

4. Run with pytest:

   deepeval test run test_llm.py

Pricing: DeepEval is free and open source. Confident AI (the hosted platform) offers free and paid tiers for dashboards, collaboration, and dataset management. Metric computation uses LLM API calls (~$0.01-0.05 per test case).