ealeng
Menu
BlogLearnVideosAboutStart learning

Learn

Learn eval engineering by doing it

Practice-first tracks. Each lab has runnable code, a metric to compute, real measured results, and a checklist to test what you understood.

AvailableBeginner → Intermediate

AI Engineering: Hands-on

Prompts, memory, context, search, RAG, evals, reasoning, agents and multi-agent teams, measured on recorded runs.

A twenty-five-video, hands-on series on building with language models: prompt settings and tests, memory and context, embeddings and search, RAG over documents and databases, evals in CI, reasoning models, tool-calling agents, MCP, multi-agent systems, and fine-tuning. Each chapter uses recorded lab results and a runnable exercise.

▶ Start with episode 1Watch playlist ↗

LLM calls & promptingEp. 01–02

  1. Ep. 01What you control when you call an LLM
  2. Ep. 02Prompt engineering, measured

MemoryEp. 03–04

  1. Ep. 03Six ways to give an assistant memory
  2. Ep. 04Where AI memory lives, and how to delete it

Context engineeringEp. 05–06

  1. Ep. 05The prompt is a budget
  2. Ep. 06Long agent sessions and skills
Show all 25 episodes + 32 coming soonThen: Embeddings & search · RAG · Evals · Reasoning models · Agents & tools · Multi-agent systems · Fine-tuningComing soon: Multimodal & voice · Serving models · Resilience · Observability · Safety & security · Compliance · Governance · Self-improving systems

Embeddings & searchEp. 07–09

  1. Ep. 07Embeddings and semantic search
  2. Ep. 08Test search in customer words
  3. Ep. 09Chunk documents for search

RAGEp. 10–12

  1. Ep. 10RAG: checkable, not correct
  2. Ep. 11Better retrieval: keywords, hybrid, reranking
  3. Ep. 12Answer questions from a database

EvalsEp. 13–14

  1. Ep. 13A test gate you can trust
  2. Ep. 14Prompt tuning without fitting the test

Reasoning modelsEp. 15–16

  1. Ep. 15Reasoning models and the effort dial
  2. Ep. 16Routing vs. voting

Agents & toolsEp. 17–19

  1. Ep. 17Test what an agent did
  2. Ep. 18When a tool call times out
  3. Ep. 19When a tool's description gives orders

Multi-agent systemsEp. 20–22

  1. Ep. 20One agent or a team?
  2. Ep. 21Debate, voting, or a bigger model
  3. Ep. 22Orchestrators and remote agents

Fine-tuningEp. 23–25

  1. Ep. 23Fine-tune a small model with LoRA
  2. Ep. 24What fine-tuning breaks
  3. Ep. 25Distillation, DPO and RFT

Multimodal & voiceComing soon

  • Text extraction vs. vision models
  • RAG over scans and screenshots
  • Voice agent latency

Serving modelsComing soon

  • Quantization: a smaller model, measured
  • Batching many users at once
  • Self-hosting vs. an API

ResilienceComing soon

  • Retries and jitter
  • Timeouts, circuit breakers and fallbacks
  • Durable agents that survive a crash
  • Silent failures

ObservabilityComing soon

  • Tracing every model call
  • Dashboards and alerts
  • Finding causes in traces
  • From traces to evals and user feedback

Safety & securityComing soon

  • Removing personal data
  • Prompt injection detectors
  • Keeping secrets out of the prompt
  • Prompt injection in agents
  • Pinning MCP tools

ComplianceComing soon

  • Access control for RAG
  • Policy checks
  • Tamper-evident audit logs
  • Replays and explanations

GovernanceComing soon

  • An inventory of every AI system
  • Risk levels under the EU AI Act
  • A launch checklist
  • Incident response

Self-improving systemsComing soon

  • Auto-generated tests
  • Can a model check its own answers?
  • Regression tests and flaky alarms
  • Self-healing RAG
  • Self-improving loops and the wrong reward
AvailableIntermediate

MCP Servers: Build, Secure, Evaluate

Connect a real MCP server, prove tenant boundaries, and test tool use before release.

A seven-video, hands-on series: understand a multi-tenant MCP server, explore it with Inspector, test its authorization and failure modes, automate release checks, and watch an AI client use the tools safely.

▶ Start with episode 1Watch playlist ↗
  1. Ep. 01How a multi-tenant MCP server works
  2. Ep. 02Five layers of MCP server testing
  3. Ep. 03Explore a real server with MCP Inspector
  4. Ep. 04Test authorization and tenant isolation
  5. Ep. 05Break the server safely
  6. Ep. 06Automate MCP server tests
  7. Ep. 07Evaluate an AI client using the server
AvailableIntermediate

Voice AI Evaluation

Speech-to-text, text-to-speech, and voice agents.

The work of a model evaluation team: metrics you can trust, test sets that match real audio, telling real regressions from noise, quality gates in CI, and monitoring in production. A 10-episode hands-on YouTube series.

▶ Start with episode 1Watch playlist ↗
  1. Ep. 01Word error rate, explained
  2. Ep. 02Test sets that match real audio
  3. Ep. 03Catching speech-to-text hallucinations
  4. Ep. 04Real upgrades vs. noise
  5. Ep. 05Testing streaming speech-to-text
  6. Ep. 06Testing text-to-speech
Show all 10 episodes
  1. Ep. 07Quality gates for voice AI
  2. Ep. 08Monitoring voice AI in production
  3. Ep. 09Testing a voice agent end to end
  4. Ep. 10A model release pipeline
Coming soonBeginner → Intermediate

LLM Evals: Foundations

How many evals are enough, and how to trust an LLM judge.

The questions every AI evaluation interview asks. Part I covers eval dataset design and statistics. Part II applies them to LLM outputs with no single right answer, where the grader is often another model.

  1. 01What makes a good eval dataset?
  2. 02How many evals?
  3. 03The ±0.4% problem: sample size
  4. 04Why random sampling isn't enough
  5. 05Dataset saturation
  6. 06Repeated runs
  7. 07The eval decision engine
  8. 08Rubric design
  9. 09Metrics without a reference answer
  10. 10The judge as a measuring instrument
  11. 11Pairwise comparisons and rankings
  12. 12Is your synthetic data realistic?
  13. 13When the metric stops measuring quality
  14. 14A system-one judge vs. an LLM judge
Coming soonIntermediate

Agent Evaluation

Measure what agents do, not just what they say.

Build a small agent, break it on purpose, grade its steps as well as its answer, and gate every change in CI.

  1. 01The dumbest possible agent
  2. 02Your first agent eval
  3. 03Trajectory evaluation
  4. 04The agent eval scorecard
  5. 05Adversarial agents
  6. 06LLM-as-judge for trajectories
  7. 07The golden dataset
  8. 08AgentEval CI
  9. 09Production → eval
Coming soonAdvanced

Support Agent: Full Lifecycle

One real agent, taken from spec to production evals.

Specify an agent, instrument it, find its failures by reading traces, turn those failures into validated evaluators and a CI gate, attack it, then improve it on accuracy and cost.

  1. 01Building agents
  2. 02Error analysis
  3. 03CI/CD
  4. 04Security & governance
  5. 05Improving agents

New labs and episodes come out regularly. Follow along on YouTube.

Subscribe on YouTube