Week 4 of 5
Evals, security and observability
Know when the assistant works, stop it from being turned against you, and see what every request costs.
0%
Build this week
- 30 to 40 eval cases (retrieval, faithfulness, tool choice)
- An LLM-as-judge checked against 15 hand-labelled cases
- A CI job that fails PRs below a score threshold
- A red-team set with a poisoned document, plus tool allowlists, field minimisation and output checks
- Tracing across agent steps and MCP calls, token cost per request, and a response cache
- Day 221 h 30 minToday
- Day 231 h 30 minToday
Judges and metrics
- 45 min
- 45 min
Notes
- Day 241 h 30 minToday
Attacks
- 45 min
- 45 min
Notes
- Day 251 h 25 minToday
- Day 264 h 45 min · weekendToday
Eval set and judge
- 2 h
- 2 h
- 45 min
Notes
- Day 274 h 25 min · weekendToday
Defend and observe
- 2 h
- 1 h 40 min
- 45 min
Notes
- Day 281 h 20 minToday
Week review
- 50 min
- 30 min
Notes
If you have time
- 45 min
Checkpoint
Answer out loud, then rate yourself.