Skip to content
← All articles

What teaching 15 engineers at a time taught me about AI systems

Running live cohorts changed how I architect. When you watch people try to operate a system you designed, you find out very quickly which of your decisions were actually good.

Chetan Karnatakam· · 1 min read
TeachingMLOpsCareersHouse of Data

I run cohorts of fifteen. Small enough that I see every person’s code, big enough that every mistake shows up more than once — and a mistake that shows up more than once is a design problem, not a student problem.

Courses stop where the job starts

Almost every AI course ends at a working notebook. The job starts about four steps later: root cause analysis on something that broke in an environment you cannot reproduce, a sprint board, a ticket, a code review, a release. Newcomers who can build a model and cannot do any of that do not last.

That is why the mix in my training is roughly 75% practical and 25% theory, and why the projects have to be deployed rather than demonstrated.

The lesson that changed my architecture work

One cohort’s production app went down because both of the model providers it depended on had an outage on the same afternoon. Single provider, no fallback chain. Underneath that, two slower problems nobody had been watching: retrieval quality drifting and latency creeping up.

We built the fallback together in class. But the real lesson was about instrumentation — the outage was obvious, the drift was not, and the drift had been doing more damage for longer.

Since then, every architecture I sign off has three things before it ships: a provider fallback path, a retrieval quality metric that someone actually looks at, and a latency budget with an alert attached.

Teaching is a debugging tool

If you cannot explain why a design decision is right to someone who has never seen the system, it is worth asking whether you know. Fifteen people asking “but why” is the most reliable architecture review I have found.

Newsletter

The week in production AI

One email a week: what actually shipped, what broke, and what it means if you have an LLM system in production. No hype, no reposts, unsubscribe in one click.