Measuring whether your WhatsApp automation actually works

Measuring an automation means asking whether it changed an outcome you care about — support load, conversion, or repeat purchase — rather than counting how many messages it sent. Message volume is the easiest number to produce and the least informative one available.

This is the honest difficulty with WhatsApp as a channel. A customer asks a question, disappears for two days, then buys through your website. Email would show you a click. WhatsApp shows you a conversation and leaves the causal question open.

The four numbers worth tracking

Number	What it tells you	Where it comes from
Conversations resolved without a human	Whether the bot is actually absorbing load	Flows that completed without an Assign Chat step
Time to first human reply	Whether service improved	Analytics
Reply rate on transactional messages	Whether people still read you	Analytics
Cost per outcome	Whether it pays	Message charges ÷ orders confirmed, carts recovered, etc.

The third one is the leading indicator nobody watches. When reply rates on order messages start falling, you are being tuned out — which shows up as a quality rating downgrade weeks later.

The attribution trap

Any recovery flow will report impressive numbers, because a share of the people it messages would have come back regardless. A tool that claims every subsequent purchase is measuring your customers' intentions, not its own effect.

The honest way to find out is a holdout: for a period, deliberately exclude a random slice of eligible customers from the flow and compare. If the flow is working, the messaged group converts meaningfully better. If it is not, you have learned something expensive for free.

What the platform can and cannot tell you

• Delivered and read are available per message, though read counts undercount because recipients can disable read receipts.
• Link clicks are a raw count. There is no click-through rate, and you cannot build an audience of people who clicked.
• Automation activity shows every run and where it stopped — the best debugging tool you have.
• Revenue is not attributed automatically. WhatsApp traffic arriving at your website generally lands in "direct" unless you tag your links.

That last point is fixable and worth twenty minutes: add UTM parameters to the links your flows send, so your store analytics can separate WhatsApp-driven sessions from everything else.

Judging a flow by its job

Different flows deserve different questions.

1. Order confirmation: what share of orders get confirmed, and how many cancellations arrive before you paid to ship? That second number is the whole value in cash-on-delivery markets.
2. "Where is my order?": what share of order-status questions finished without a human? Compare against your inbox volume before you launched it.
3. Abandoned checkout: holdout test, or you are guessing.
4. Review requests: reviews received per hundred messages sent.
5. Ad lead qualification: cost per qualified lead, not per conversation started.

The cost side

Cost per outcome needs both halves of your bill: the subscription and Meta's per-message charges. Since messages sent inside an open conversation window are free under Meta's pricing rules, a well-designed flow can have a near-zero marginal cost — which makes "cost per outcome" a genuinely useful comparison between flows rather than a formality.

It also shows you where money is actually going. Usually one marketing flow accounts for most of the message spend while the utility flows run for nothing.

A monthly review that takes twenty minutes

1. Check quality rating. Green?
2. Check each automation's activity view. Any flow with zero runs is broken or unnecessary — both worth knowing.
3. Read the unanswered branch of your AI step. Every question there is a gap.
4. Check opt-out volume against list growth.
5. Pick the flow with the worst cost per outcome and either fix or retire it.

Point two catches more problems than anything else. A keyword flow that stopped matching because customers changed how they phrase things does not error — it simply stops running, silently, and keyword lists are the usual culprit.

The uncomfortable question

For each flow, ask what would happen if you switched it off for two weeks. If you cannot name the number that would move, you are not measuring it — and a flow nobody can defend is a flow adding to your frequency budget for nothing.

How to run a holdout without much effort

A holdout is the only honest way to know whether a recovery flow caused anything, and it does not need to be sophisticated.

1. Pick one flow — abandoned cart is the usual candidate, because it is the one with the largest claimed effect and the largest cost.
2. Exclude a slice of eligible customers for a fixed period. A tenth is enough if your volume is reasonable; a quarter if it is not.
3. Run it for long enough to matter — a full purchase cycle, not a week.
4. Compare conversion between the two groups, not the messaged group against itself.
5. Subtract the message cost from any uplift before deciding.

If the messaged group does not convert meaningfully better, you have learned something valuable for the price of not sending some messages. That is a cheap experiment, and almost nobody runs it.

Numbers to treat with suspicion

This category is full of figures that circulate without a source. We went looking for the origins of the most common ones and could not find them, so we do not publish them and you should not plan against them.

Claim you will see	Status
"98% open rate"	No locatable primary source
"90% read within 3 minutes"	No locatable primary source
"11.7% cart recovery vs 0.34% for email"	Vendor marketing
"WISMO is 30–50% of support tickets"	Vendor claim, repeated widely
"AI resolves 76–92% of interactions"	Vendor claim, implausibly precise

When comparing tools, the useful question is not what percentage a vendor claims but whether they will tell you how it was measured. A number with a method attached is worth more than five without.

Fixing attribution in your store analytics

WhatsApp traffic arriving at your website generally lands in "direct" unless the links you send are tagged, which makes the channel look like it contributes nothing. This is fixable in about twenty minutes and it is the highest-value measurement change most stores can make.

Add UTM parameters to every link a flow or broadcast sends, distinguishing at least the flow that sent it. Then your existing store analytics can separate WhatsApp-driven sessions from everything else, and you can compare against email and paid on the same basis.

Two caveats to keep expectations honest. Link tracking inside WhatsApp gives a raw click count only — there is no click-through rate, and you cannot build an audience of people who clicked. And read counts are structurally undercounted, because recipients can disable read receipts, so treat delivered as the number that means something.

What "free" does to the arithmetic

Cost per outcome is unusually favourable here, and it is worth understanding why before you compare against email. Messages sent inside an open 24-hour conversation window are free under Meta's pricing rules, and service conversations have been free and unlimited since 1 November 2024.

So a support-deflection flow or an order-status lookup has a marginal cost of essentially zero, while a marketing broadcast is charged on every delivered message. When you rank your flows by cost per outcome, expect the free ones to dominate — and expect one marketing flow to account for most of your spend. That concentration is usually where the first improvement is.

The one-page monthly review

1. Quality rating — still green?
2. Any automation with zero runs? Broken or unnecessary; both worth knowing.
3. The unanswered branch of your knowledge base — what is in it?
4. Opt-outs against list growth.
5. Cost per outcome for your most expensive flow.

Twenty minutes, once a month. Every item on that list is a leading indicator, so finding a problem here costs you a fix rather than a customer.

Comparing WhatsApp against email fairly

The two channels are not measured the same way, so a direct comparison usually flatters one of them by accident. Email gives you opens and clicks; WhatsApp gives you delivered, an undercounted read, and a raw click count.

The honest basis for comparison is revenue per recipient after costs, on the same audience, over the same period. Anything else is comparing a metric one channel reports well against a metric the other does not report at all.

Pick one number and watch it

If all of this feels like a lot, choose a single metric and track it monthly. For most stores that is the share of conversations resolved without a person: it captures whether automation is doing its job, it is hard to game, and it moves for real reasons rather than seasonal ones.