Roll out a new model

This is the whole path from a new model to real users, and not a single code deploy is needed. Two things are worth having before you start: a Primary success metric to judge by (see Prove the impact), and a history of real calls to use as test inputs.

Testingscreen on real inputs Routingroute 10% Comparewatch, then judge Routingswitch to 100%

1. Screen the candidate in Testing

In Testing, keep the variant currently in production as your baseline and put the new model next to it, leaving the system prompt the same for now. Fill the input with “Load from history”, using requests real customers actually sent. Results from inputs you made up are hard to trust.

Press “Run all” and compare quality by eye, cost and latency by the numbers. Repeat over a handful of different inputs, and if quality visibly collapses, stop here. That candidate never meets a real user.

2. Save, then route just 10%

Once it passes the screening, turn the candidate into a new variant and give it 10% in Routing. From this moment, one user in ten gets the new model’s responses. The rule that keeps a given user on the same variant is covered in Routing.

3. For a few days, just watch for problems

Put both variants on Compare and check only that the new variant’s error rate is not spiking and its average latency is where you expect it. It is not time to judge outcomes yet.

If something looks off, filter Requests to the new model and find the cause, and if it is serious, drop the new variant to zero in Routing. That is the rollback, and it takes seconds.

4. Judge in Compare

Put both variants into Compare and the feature’s Primary success metric is already set as the default basis.

Not rushing matters here. Verdicts are on the Results tab in Compare. Until each variant has 100 users, 25 of whom sent a metric event or call, the Results tab shows Collecting data, and the numbers until then are not verdicts. Saving a different ratio starts a new ratio period, and verdicts build up again from that point.

Once there is a verdict, read the error rate, the success metric, then cost per request, in that order. Error rate and cost per request are on the Monitoring tab. If conversion is slightly lower but cost is much cheaper, a “revenue per AI dollar” metric folds both into one number.

5. If the results are good, expand

The stages for raising the ratio, and what to confirm at each, are in Routing. Drop the variant you are not keeping to zero and it drops out of the routing. There is no need to delete it.

Whether it is a prompt change or parameter tuning, whatever you change, the sequence is the same. Take the cost change before and after the switch from Overview’s 30-day trend and share it with your team.