Compare

Compare is where you go a few days after splitting traffic in Routing, to read the result. It is also the screen to watch in the first days after a new variant starts taking traffic, in case error rate or latency spikes. Per-variant numbers are read here, side by side.

Success metricsdefine the bar Testingtry candidates Routingroute traffic Comparejudge outcomes

Compare has two tabs: Monitoring and Results. The Monitoring tab shows per-variant numbers as they are, and the Results tab shows each variant’s change against the baseline variant and a verdict for every success metric.

Each variant keeps the same color in Routing and both Compare tabs. Variants from the same provider also receive distinct colors; solid and dashed lines provide an additional distinction. A variant’s color does not indicate improvement or decline.

The Monitoring tab

Area	What you see	How to read it
Basis selector	Choose an event’s conversion rate or a success metric as the basis, and pick up to five variants to compare	A Primary metric, if there is one, is selected from the start
Chart	The chosen basis over time, one line overlaid per variant	Read a difference only when the lines separate and hold their gap
Success-metric comparison	The value of each event that makes up the chosen success metric, and the composite value. Ratio metrics also show each event’s weight	Appears only when a success metric is chosen. Shows which event a difference in the composite comes from
Variant cards	Per variant: cost per request, average latency, error rate, and the value of the basis you chose	Four numbers, read side by side. The first three average one request across every node in the feature
Per-run comparison	Per variant: run count, error runs (share of runs), requests per run, cost per run, run duration (average, p50, p95, p99), and a chart of requests per run	A variant that is cheaper per request can still cost more per run when each run sends more requests. Read it next to the cost per request on the variant cards
Node details	Per node and variant: request count, cost per request, average latency, p50, p95, p99, and error rate	When node request counts differ between variants, the variant cards move with them. This table compares node with node

The per-run comparison counts one agent or workflow run as the unit. It counts only calls for which your server SDK sends a run ID, and the table does not appear when no call carries one. A variant with successful requests whose cost is unknown shows no cost per run. An error run is a run with at least one request that ended in a failure counted by the error rate below. Run duration is the time from when ABTO receives the run’s first request until the response to its last request finishes. It includes time your app spends running tools between requests, and leaves out time before the first request and after the last response. Run duration includes error runs. The requests-per-run chart shows, for each variant, the share of runs that ended after each number of requests. Runs with 100 or more requests share a single 100+ bar. To group several requests into one run, see the server SDK docs for JavaScript, Python, or JVM.

The variant cards and node details count one request your app sent as the unit.

• Request count: a request ABTO retried still counts once.
• Average latency: the time a successful request takes to get its response, including time spent on retries. Failed requests are left out.
• Cost per request: based on requests whose cost is known.
• Error rate: the share of requests that ended in a provider error (5xx), a timeout, a connection failure, or an ABTO internal error. Rate limits (429), malformed requests, API key or billing problems, and requests your app cancelled are not the variant’s fault, so they are not counted. You can find those failures on the Requests screen.

The Results tab

The Results tab compares each variant with the baseline variant on every success metric it can judge. It judges ratio metrics and value metrics that can be counted per user. A value metric must be the count, sum, or average of one event or call, or two counts or sums joined by division, addition, or subtraction. Call cost, tokens, latency, and request count are counted per user by the deviceId on requests that went through ABTO. Requests without a deviceId are left out of the verdict along with their exposures.

These metrics do not appear on the Results tab. Read them as numbers on the Monitoring tab.

• Median, p95, and p99 metrics
• Formulas of two values that include an average, and multiplication formulas
• Event conversion rates that are not saved as a success metric

Area	What you see
Ratio period	The window the statistics are computed over. Defaults to the current period
Baseline variant	The variant the others are compared with. Defaults to the current Origin variant
Cumulative users	The card at the top. It shows users who got a variant in the ratio period, days since the period started, a chart of cumulative users per variant, and a table of each variant’s routing weight against its actual share. The actual share divides among the users of the period’s routed variants
Checks	Data checks to review before trusting a verdict. Checks still gathering data show as pending. Expanding them shows each check
Metric table	Per variant: users, metric value, change against the baseline, confidence interval, verdict. The metric value is the conversion rate (conversions / users) for ratio metrics, a per-user value for counts, sums, and addition or subtraction formulas, and a per-event or per-request value for averages
Graph	Open it with “Show graph” under the metric table. The Change view plots each compared variant’s change against the baseline, and the Metric value view plots each variant’s metric value including the baseline, both cumulative from the start of the period. Read confidence intervals from the bars in the metric table

A new ratio period starts each time you save a different serving split in Routing. The split does not change within a period, so each variant is compared on users gathered under the same conditions.

The verdict follows the direction in which the confidence interval leaves zero.

• Improvement confirmed: The whole interval is above zero. For lower-is-better metrics, below zero.
• Decline confirmed: The whole interval is below zero. For lower-is-better metrics, above zero.
• Undecided: The interval spans zero.
• Collecting data: The baseline and the variant do not yet both have 100 users, 25 of whom sent a metric event or call.

The Results tab does not pick a single winner. It shows each variant’s verdict against the baseline separately.

• Sequential test: The confidence interval holds 95% confidence no matter when you open it during the period. It narrows as users accumulate.
• Bonferroni correction: When several variants are compared with the baseline, the bar for a confirmed verdict tightens with the number of variants. With three variants compared, each comparison uses a 5% significance level divided by 3.
• Per-user comparison: Value metrics are also compared user by user. Counts and sums are per-user values rather than variant totals, so variants stay comparable when the serving split differs. Several events or calls from one user count as that one user’s value.
• Capping extreme values: For value metrics, a user’s value above the top 0.1% of users with a value above zero is lowered to that boundary. Zero and negative values are not lowered. This keeps one user’s large value from deciding the verdict. The numbers on the Monitoring tab stay as they are.
• Assignment ratio check: When users per variant statistically disagree with the routing weights, the Results tab shows a warning and pauses verdicts.
• Users who saw more than one variant: Users who got more than one variant in a ratio period are excluded from every variant.
• Exposures without a deviceId: Exposures recorded without a deviceId are not counted toward any variant.

Judge in this order

Order	Metric	The call
1	Error rate	A variant with visibly higher errors is out, regardless of outcomes
2	Success metric or conversion	Is the new candidate level with or ahead of the incumbent. Read it from the verdict on the Results tab
3	Cost per request	At similar outcomes, pick the cheaper one

If conversion is slightly higher at several times the cost, whether that is worth it is a business decision. Once the choice is settled, go back to Routing and move the traffic.