Evaluation, observability, and improvement
Build evaluation datasets for retrieval, generation, agents, safety, and end-to-end workflows. Use automated metrics, LLM-as-judge methods, human review, and regression testing appropriately. Instrument applications for tracing, latency, cost, quality, tool usage, and user feedback.
Hands-onBuild an AI quality and observability platform for the capstone system: Golden dataset and regression suite. Retrieval metrics: precision@k, recall@k, MRR, and nDCG. Generation metrics: groundedness, relevance, completeness, and hallucination rate. Agent metrics: task success rate, tool-call accuracy, recovery rate, and cost per completed task. Trace visualization, alerting thresholds, and weekly quality report.