Udemy – Production LLM Evaluation and Observability

This course teaches you how to evaluate, test, and monitor real world LLM applications before and after they reach production. You will build an Agentic RAG chatbot and use DeepEval for LLM evaluation and Langfuse for LLM observability, tracing, and monitoring. Build a Practical LLM Evaluation and Observability Workflow.

In this course, you will:

• 1. Learn the fundamentals of LLM evaluation and why traditional testing is not enough for AI applications.
• 2. Understand human vs automated evaluation and when to use each approach.
• 3. Learn offline and online evaluation for LLM applications.
• 4. Create and work with golden datasets for systematic evaluation.
• 5. Understand reference based and reference free evaluation.
• 6. Build and use different LLM evaluation metrics.
• 7. Create custom evaluation metrics for application-specific requirements.
• 8. Use DeepEval to build an automated LLM evaluation pipeline.
• 9. Evaluate an Agentic RAG chatbot using practical evaluation techniques.
• 10. Learn LLM observability with Langfuse.
• 11. Trace LLM calls, retrieval steps, tool calls, and application workflows.
• 12. Monitor evaluation scores and use observability data to understand application behavior.
• 13. Connect evaluation and observability to create a more reliable production workflow.