Building an AI harness in music education - choosing the right model

At Singing Carrots, what we're building on the AI side is essentially a harness. It's a very specific and sophisticated harness that is not just mixing models and using them with the tools that are writing text or coding. In our case it's interacting with an external environment through audio. On one side, we have to generate sounds, write melodies and play notes. On the other, we have microphone input. We read telemetry from it and feed that into the coach. There's a whole pipeline for processing all of this, both from the mechanical and the educational point of view.

We use different models from different vendors for different pieces of the task. We have to make it understand the telemetry. We have to make it write music. We have to make it make pedagogical decisions. We have to make it chat. We have to make it plan. We have to make it retrospect. We have to make it work with the memory. And we need to measure and evaluate all of that.

When we started with the AI coach, the choice was pretty straightforward. Only a few models could handle the hardest part, where the coach is actually conducting the lesson and latency matters. We picked the one that seemed like the obvious choice and started doing excessive prompt engineering.

We'd come up with metrics, take the anonymized session data and replay the sessions to see how the prompt behaved, much like Pedro Tabacof described for Fin. Then we'd pick the most promising version and run a live A/B test. It was a two-stage rollout. First we'd do the offline replay. Then we'd check how the new prompt actually affected our product metrics before updating it.

As you can imagine, after a series of these A/B tests, the prompting system and the harness became tied to the model. Then a new model came out. We started the same way, replaying sessions, picking out critical moments and judging them. But what were we actually seeing? Was the new model better or worse? Or were we seeing how well our existing prompts worked on that model?

How do we compare apples to apples?

In the beginning, it was pretty obvious. The models varied widely on things we could measure. Some would write the wrong music. Sometimes the notes the coach played on the piano were different from the ones it was talking about in the chat. We built our own testing framework around cases like those, and we could catch them early.

Then newer models started handling those cases about equally well. Where they differed was in how they followed the pedagogical instructions. How do they plan a session? How do they give feedback? How do they adapt within the lesson? Those things are much harder to judge, and they have a lot to do with what the student gets out of a lesson. So how do we know what's actually going to work better?

The first thing we did was automate the test cases we already had. There are three things that come into play for us when we pick a model. What does it cost? How long does it take to reply? And how good is the output? At maximum effort, the top models from the top labs can produce pretty good output and pretty good judgments on the things we're working on. But a call might take a few minutes and cost tenfold or even a hundredfold what we pay for a call in a live lesson, where the coach needs to reply almost instantly.

What that means is we can use a slow, expensive model to judge a cheaper, faster one. We started using these judges both to read the replays and to read sessions from live A/B tests. We had them read blind. One interesting thing we noticed was that judges tended to favor models from their own vendor, even when we didn't tell them which model had produced the output. In one run, one judge flagged 14% of the answers from the other vendor's top model, while flagging only 2.5% from its own.

When the judges disagreed by a lot, we'd have our singing teachers or engineers read a handful of those anonymized sessions. Then we'd use their feedback to refine how we defined the metric in the judging prompt, so that the metric itself makes the judges align.

That helped us pick candidates for production A/B tests a bit faster. We still had to spend some money on replays, but we could focus on the critical cases and scenarios. The judges would score them, a human would weigh in where the judges disagreed, and then we'd decide what went into the A/B test.

Newer models win our benchmarks and lose our A/B tests

The funny thing we've noticed over the last six months or so is that some of the newer, cheaper, supposedly smarter models look better on the generic benchmarks. They even look better on our own replay benchmarks. Then we start a live A/B test and they lose to the previous generation on product metrics. And those are the only things that are showing us what's going on in the real product in real life. How are users rating the sessions? How are they engaging? Are they coming back? Are they achieving their goals in the product? And are they moving toward the thing that matters most to us, the educational outcome?

One example is GPT-5.6 Terra against GPT-5.1. One of our judges found bad turns in 0-2% of Terra's replies and in 15-26% of GPT-5.1's. And still, about 5 points fewer students came back within 72 hours after a session with Terra.

It's really early days in this whole thing, and we're trying to follow where the industry is going. But our learning so far is that generic benchmarks already diverged a lot from domain-specific benchmarks. Another example from OpenAI is GPT-5.6 Sol and GPT-5.6 Terra. Sol performed better across the benchmarks we looked at. But in a live A/B test for one of the things we use these models for, Terra just did better. It was the smaller, cheaper and supposedly stupider model. And that's within the same vendor, where you'd expect the next tier up to do better.

The judge quality ceiling

For the calls where latency matters, we're approaching something I call the judge quality ceiling. Imagine the best, most expensive model we can get right now, say Fable 5.2 or the latest Astra. Imagine it was cheap and could reply in a second or two, even at maximum effort. In reality, at maximum effort these models took a median of 66 and 87 seconds per answer, and a live lesson needs it in about 4.

We can try that in a local replay. We take the calls that need to be fast in a live lesson and give them to those slow, expensive models. Then we use the judges, which are essentially the same smartest models on the market, so they judge themselves without knowing that they've produced the output and without knowing the latency and the cost. Well, I mean, themselves is the set of high-tier models judging the set of high-tier models. That gives me a score I can use as a ceiling. I can take a cheaper model, say GPT-6.1 Terra, and see how close it gets on the same calls, with the same judges.

Over the last two generations of releases from several vendors, we've seen models one or two tiers below the top come really close to that ceiling on our offline benchmarks, even before we adapt the prompt to the model. Once you start twisting the prompt per model, you still get differences. But without twisting the prompts, we've basically already approached that self-judging ceiling.

Then we compare those same models in production and see completely different results, which means that our benchmarks are good at predicting the quality but bad at predicting the human behavior and psychology and interpretation of what's going on.

Part of that is a weakness in our own judging system, and we need to improve it. But there's also a limit to how fast we can learn. Think about how long you have to run an A/B test to see whether people come back. Sometimes you only have enough traffic to test one prompt variation. You can run a few in parallel, but you only have so much data and so much time. So you can only move so fast.

That happened to us this autumn. We'd started testing OpenAI's GPT-6.1 Sol. It was already taking up a share of our A/B testing traffic. Then Terra came out so soon after Sol that we still didn't have enough sessions to see the result of the Sol test, and we already had Terra waiting for its turn.

So, yeah, interesting times we're living in. I hope this is helpful to people building something similar. We'll keep innovating and updating everyone on how things are going.