Building with AI
Generative AI system evaluation course
Measure your AI application's quality with data, not impressions.
“I tried it with a few questions and it works fine” is not a way to evaluate a system that will serve customers or make decisions. Evaluation is the discipline that separates prototypes from AI applications you can trust.
In this course you will learn to build representative test sets, choose metrics, use models as evaluators without falling for their biases, evaluate RAG systems and agents and build it all into the development cycle to catch regressions before they reach users.
What you will learn
- Design representative evaluation sets
- Choose the right metrics for each type of task
- Use models as evaluators reliably
- Evaluate RAG systems and agents
- Build evaluation into development and production
Syllabus
8 modules, from the fundamentals to the final project.
-
Why evaluate
- The "it seems to work" problem
- Model evaluation versus application evaluation
-
Evaluation sets
- Real, edge and adversarial cases
- Labelling and reference answers
- Synthetic data
-
Metrics
- Accuracy, coverage and extraction metrics
- Faithfulness, relevance and tone
- Cost and latency
-
LLM as a judge
- Rubrics and pairwise comparisons
- Evaluator bias and how to calibrate it
- Agreement with human evaluators
-
Evaluating RAG
- Retrieval quality
- Faithfulness to the context
-
Evaluating agents
- Task success and trajectory
- Tool use and errors
-
Continuous integration
- Regression tests for prompts and models
- Comparisons before switching models
-
In production
- Sampling and human review
- User feedback
- Degradation alerts
Tools you will practise with
- Python
- promptfoo
- Ragas
- DeepEval
- Langfuse
Hands-on project
You create the evaluation system for an AI application (your own or a sample one), with its set of cases, automatic metrics, a calibrated evaluator and a regression test in continuous integration.
Requirements
- Python programming.
- Previous experience with language model APIs.
Other courses
-
AI in apps and digital products course
AI features users understand, use and trust.
- Duration: 12 hours
- Level: Intermediate
-
AI-assisted programming course
Code faster with coding assistants and agents, without losing quality.
- Duration: 12 hours
- Level: Intermediate
-
LLM application development course
From the first API call to a generative AI application ready for production.
- Duration: 20 hours
- Level: Intermediate
Want this course for your team?
We adapt the content, length and format to your company.