
Production AI Agents with MLflow: AgentOps Techniques for Evaluation, Observability, and Deployment
Published 11/2026
By Jonathan Bown
MP4 | Video: h264, 1280x720 | Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning | Language: English + subtitle | Duration: 6h 48m | Size: 12.2 GB
Build, evaluate, and ship production-grade AI agents with MLflow's open-source observability, evaluation, and deployment stack
Overview
Production AI Agents with MLflow: AgentOps Techniques for Evaluation, Observability, and Deployment is a hands-on, code-driven course that takes you from a single LLM call to a LangGraph workflow to a deployed ReAct agent, using MLflow tracing, evaluation, Prompt Registry, AI Gateway, and Model Registry to make every stage observable, measurable, and production-ready.
Learn How To
⚡ Apply Evaluation-Driven Development and the Agent Development Lifecycle to move agents from proof-of-concept to production
⚡ Set up a local MLflow tracking server and trace LLM calls, RAG pipelines, and tool usage in the MLflow UI
⚡ Run structured evaluations with built-in scorers and custom LLM-as-judge scorers
⚡ Capture human feedback and ground truth annotations on traces to build alignment datasets
⚡ Align automated judges to expert human feedback using the SIMBA optimizer
⚡ Version and alias prompts in the MLflow Prompt Registry using a champion/challenger pattern
⚡ Evolve a single LLM call into a LangGraph workflow with node-level evaluation, memory, and memory governance
⚡ Build and evaluate ReAct tool-calling agents, including the shift from offline to online evaluation
⚡ Route traffic, track usage, and set budgets with the MLflow AI Gateway
⚡ Automate prompt optimization and evaluate multi-turn conversations with simulated user personas
⚡ Package, register, and serve agents locally with the MLflow Model Registry
Who Should Take This Course
AI and GenAI engineers, machine learning engineers, and data scientists working on agents or LLM applications, as well as product engineers and product leaders who want to understand the Agent Development Lifecycle and lead engineering teams building agents. This course is ideal for practitioners who have built LLM or agent prototypes and want the operational discipline to make them observable, measurable, versioned, and deployable.
Course
Requirements
⚡ Intermediate Python proficiency (functions, classes, environment variables, virtual environments)
⚡ Basic familiarity with LLMs and the OpenAI API or an equivalent (e.g., Gemini, Anthropic)
⚡ LLM API Key recommended but not required - Gemini Free Tier suitable for many aspects of course
⚡ Comfort working in and reading Jupyter notebooks
⚡ No prior MLflow experience required; all MLflow concepts are introduced from scratch
Homepage
https://www.oreilly.com/videos/production-ai-agents/9780136173045/
We need your support!
Make a donation to help us stay online
Bitcoin (BTC)
bc1q08g9d22cxkawsjlf8etuek2pc9n2a3hs4cdrld
Bitcoin Cash (BCH)
qqvwexzhvgauxq2apgc4j0ewvcak6hh6lsnzmvtkem
Ethereum (ETH)
0xb55513D2c91A6e3c497621644ec99e206CDaf239
Litecoin (LTC)
ltc1qt6g2trfv9tjs4qj68sqc4uf0ukvc9jpnsyt59u
USDT (ERC20)
0xb55513D2c91A6e3c497621644ec99e206CDaf239
USDT (TRC20)
TYdPNrz7v1P9riWBWZ317oBgJueheGjATm