Why Your A/B Test Shows a Big Win on RPV but AOV Looks Weak or Flat

You ran a test and saw something like this:

• Revenue Per Visitor (RPV): +8% with 97% probability to beat original
• Average Order Value (AOV): +2% with only 68% probability (or even “no winner”)

This happens all the time and is completely normal. Here’s why.

How the metrics are calculated

• AOV = Total Revenue ÷ Number of Orders (only counts people who actually bought something)
• RPV = Total Revenue ÷ Total Visitors (includes every visitor, even the 95–98% who bought nothing)

Because most visitors spend $0, the starting RPV number is very low (usually $2–$4). A small increase in total revenue creates a much larger percentage jump in RPV than in AOV.

Example

Same extra revenue, two very different percentage lifts:

That’s why RPV almost always reaches statistical significance faster and shows higher confidence than AOV, even when the actual business improvement is exactly the same.

What you should do

Trust RPV as your primary metric for deciding winners.

Companies like Amazon, Booking.com, and Shopify do exactly this for the same reasons:

• It directly measures total revenue impact
• It naturally includes both conversion rate and order value changes
• It’s far more reliable and sensitive than AOV or conversion rate alone

Quick rule of thumb

• RPV is clearly positive (95%+) → Ship the winner
• RPV is up but AOV looks weak or flat → Still ship it (this pattern is expected)
• Both RPV is flat → No real revenue impact

Bottom line: Strong RPV + weak-looking AOV is not a bug, it’s how the math works. Feel confident declaring the variant the winner.