CRM Control Group Case Study for Incrementality
CRM Control Group Case Study for Incrementality
A CRM control group is not primarily about proving that players who received a campaign deposited, wagered or returned.
Most CRM teams can already show that.
The more important commercial question is:
Did the campaign actually cause that behaviour, or did it simply reach players who were likely to act anyway?
That distinction matters for iGaming operators because it affects:
Bonus cost
Net gaming revenue
Player value
Retention strategy
Campaign frequency
CRM budget allocation
Without a control group, campaign reporting can easily overstate the value created by CRM.
Players who receive a reactivation email may produce more deposits than the wider database, but those audiences are often selected specifically because they already show characteristics associated with higher return probability.
A properly designed CRM control group provides a more reliable baseline.
In short: CRM incrementality testing compares a randomly selected treatment group with an equivalent holdout group that does not receive the campaign. The difference in commercial outcomes gives operators a stronger estimate of the value genuinely created by CRM.
What is a CRM control group?
A CRM control group is a group of eligible players deliberately excluded from a campaign so their behaviour can be compared with players who receive the treatment.
The two groups should be as similar as possible before the campaign begins.
The main difference should be:
Treatment group
Receives the CRM campaign.
Control group
Does not receive the campaign being tested.
If the treatment group subsequently produces stronger commercial outcomes, the difference between the two groups provides evidence of incremental impact.
This is much stronger than simply comparing campaign recipients with the wider customer base.
Why CRM campaign reporting can overstate performance
Consider a casino reactivation campaign.
The operator targets players who:
Previously deposited
Were historically active
Have recently become inactive
Remain eligible for marketing
The campaign may then report:
Deposits
Revenue
Reactivation rate
Click-through rate
among recipients.
Those numbers may look strong.
However, some of those players would have returned without receiving the campaign.
They may have come back because of:
Payday
A major sporting event
Natural playing patterns
Product seasonality
A new game release
Brand familiarity
Counting all revenue from campaign recipients as campaign-generated value therefore overstates the impact.
CRM incrementality asks a different question
Traditional CRM reporting asks:
What did players do after receiving the campaign?
Incrementality asks:
What did players do because they received the campaign?
That distinction is fundamental.
Suppose:
18% of campaign recipients reactivate
13% of the control group reactivate
The campaign did not create an 18% reactivation rate from nothing.
The estimated incremental uplift is the difference between the treatment and control groups.
In this example:
5 percentage points
That provides a much more realistic view of campaign contribution.
The commercial problem in this CRM control group case study
The operator in this CRM control group case study ran a busy lifecycle calendar across casino and sportsbook products.
Campaigns included:
Deposit reminders
Free spin promotions
Odds-led sportsbook messaging
VIP reactivation
Lifecycle communications
Standard CRM reporting appeared positive.
Open rates were healthy.
Click-through rates performed well.
Campaign recipients also produced more deposits than the broader active-player population.
However, the CRM team recognised an important problem.
Returning-player activity was already naturally strong during certain periods.
This included:
Major sporting fixtures
The beginning of the month
Other predictable periods of increased engagement
Comparing campaign recipients with all eligible players therefore created a biased view.
The audiences were not equivalent.
The operator needed to measure true CRM incrementality
The commercial question was not:
Does CRM produce revenue?
It was:
Which CRM journeys create genuinely incremental value, for which player segments and at what cost?
Answering that required a proper test.
The operator therefore selected one existing reactivation journey and introduced a randomised holdout group.
Designing the CRM control group
The selected campaign targeted previously depositing casino players who had been inactive for between 21 and 45 days.
Eligible players also needed to:
Be contactable through the relevant channels
Remain eligible under promotional rules
Pass applicable safer gambling checks
Meet the campaign's existing targeting criteria
The eligible population was then randomly divided into treatment and control groups.
What did the treatment group receive?
The treatment group received the existing multi-channel reactivation journey.
This included:
A personalised email
A push notification where permission was available
A follow-up email
A controlled incentive
The journey remained unchanged for the first test so the operator could measure the effectiveness of the existing programme.
What happened to the control group?
The CRM holdout group did not receive the reactivation journey during the test period.
However, those players could still receive communications that were not part of the experiment, such as necessary account messaging.
This distinction matters.
A control group does not mean removing players from every business communication.
It means preventing the specific CRM activity being tested from affecting their behaviour.
Random allocation is critical
A valid control group cannot simply consist of people who were not messaged.
For example, it would be misleading to use:
Players who opted out of marketing
Lower-value customers
Players outside the target segment
Customers who failed eligibility checks
as the control population.
These players differ from the treatment audience before the campaign even begins.
That creates selection bias.
The operator therefore applied the same campaign eligibility criteria first and then randomly selected the holdout group from that eligible population.
Why randomisation improves CRM testing
Randomisation helps distribute player characteristics across both groups.
These may include:
Historic player value
Deposit behaviour
Product preference
Acquisition source
Engagement level
Natural propensity to return
No experiment will create perfectly identical groups.
However, random assignment significantly reduces the risk that one group is systematically stronger before the test begins.
This makes the resulting comparison more credible.
Define the measurement window before launch
The operator agreed the measurement period before sending the campaign.
The primary outcome was:
Incremental net gaming revenue over 28 days
Secondary measures included:
Reactivation rate
Number of depositors
Deposit value
Bonus cost
Withdrawals
Post-campaign churn
Safer gambling review indicators
Agreeing these metrics before launch prevented the team from selecting whichever number looked strongest afterwards.
Why the control group used a 28-day measurement window
A next-day report would have captured the immediate response to the reactivation campaign.
But it would not answer the full commercial question.
For example, a promotional incentive could cause a player to deposit today instead of next week.
The 24-hour result would look positive.
Over a longer window, however, treatment and control behaviour may converge.
The campaign may have shifted the timing of the deposit rather than created an additional one.
Short-term CRM response can hide longer-term economics
Immediate campaign performance can also overlook:
Bonus cost
Withdrawal behaviour
Repeat inactivity
Margin
Player retention
Subsequent deposits
A campaign producing a large day-one deposit uplift may therefore create limited incremental value over several weeks.
The 28-day window gave the operator more time to assess the actual commercial result.
The correct CRM test window depends on the campaign
There is no universal measurement window for every iGaming CRM control group.
For example:
Sportsbook event campaign
A campaign tied to a specific football match may only need a short evaluation period.
Casino reactivation
A dormant-player journey may require several weeks to assess whether returning players remain active.
Early-life onboarding
The operator may want to examine first deposit, second deposit and early retention over defined windows.
The key requirement is consistency.
Treatment and control players should be measured across the same period using the same rules.
What did the CRM incrementality test reveal?
The treatment group produced a higher reactivation rate than the control group.
This confirmed that the journey was successfully moving some inactive players back into activity.
However, the incremental effect was considerably smaller than the operator's standard campaign reporting suggested.
This was the most important finding.
The previous approach effectively treated much of the activity from campaign recipients as campaign-generated.
The control group revealed how much of that activity was likely to have occurred naturally.
Measure incremental revenue above the baseline
Once a control group exists, the operator can estimate the value created above normal player behaviour.
For example:
Treatment NGR: £120,000
Control-group-equivalent NGR: £100,000
The campaign's estimated incremental NGR is:
£20,000
rather than £120,000.
The exact analysis depends on population size and test design, but the principle is straightforward.
The control group establishes the baseline.
The incremental effect is what happens above that baseline.
Move from incremental revenue to incremental contribution
Revenue alone still does not provide the complete commercial result.
The operator therefore also considered costs associated with the campaign.
These included:
Bonus expense
Campaign delivery
Operational costs
This created a measure closer to:
Incremental contribution
rather than simply campaign turnover.
A campaign can increase gross revenue while still generating weak incremental contribution if incentives are expensive.
Segment-level CRM incrementality was even more valuable
The overall campaign result was useful.
The segment-level analysis was more actionable.
Different player groups responded very differently to the same reactivation journey.
Recently inactive, historically valuable players
Players with:
Strong previous deposit frequency
Recent inactivity
Higher historic engagement
produced positive incremental contribution.
Even after incentive costs, the reactivation journey generated additional value.
This suggested that the operator should continue targeting this cohort.
However, the test also raised another question:
Did these players need the full incentive?
Their natural likelihood of returning was already relatively high.
That created an opportunity to reduce promotional cost.
Long-inactive, low-value players
Another segment behaved very differently.
Players with:
Longer inactivity
Low historic player value
showed some response to the campaign.
They opened messages and some returned.
However, the incremental net value was insufficient to justify the promotional exposure.
Some players reactivated briefly and then became inactive again.
A standard CRM report could have counted those players as successful reactivations.
The control-group analysis showed that the economics were much weaker.
The operator changed its CRM reactivation strategy
Based on the control-group results, the operator made three main changes.
1. Reduced incentive use for higher-propensity players
Players already showing a strong natural likelihood of returning did not always require the full promotional offer.
The operator began testing simpler personalised reminders.
This reduced bonus exposure while preserving incremental return behaviour.
2. Stopped using the same offer for low-value dormant cohorts
Long-inactive, low-value players were no longer automatically targeted with the same incentive.
This reduced spend on cohorts that produced weak incremental contribution.
3. Created product-specific reactivation tests
Players with clear casino category preferences entered a separate experiment.
Instead of defaulting to a blanket bonus, the operator tested:
Relevant content
Product-specific messaging
Timing
This allowed CRM to determine whether relevance could create value with lower promotional cost.
The lesson was not simply to send fewer campaigns
The test did not prove that CRM activity should be reduced across the board.
It showed where additional promotional pressure created genuine value and where it did not.
That distinction is important.
A strong incrementality programme helps teams:
Invest more where CRM changes behaviour
Reduce cost where customers would return anyway
Stop activity where incremental value is weak
The objective is better allocation of CRM effort, not less CRM activity for its own sake.
Protect the CRM control group throughout the experiment
Creating the holdout group is only the beginning.
The control audience needs to remain protected for the entire experiment.
For example, control players should not accidentally enter:
Another reactivation campaign
A related promotional broadcast
An overlapping automated journey
A paid-media promotional audience
if doing so would contaminate the test.
Once control players begin receiving similar treatment, the behavioural difference between groups becomes harder to interpret.
Coordinate holdouts across teams
CRM activity often overlaps with:
Paid media
Customer service
Product
Promotional operations
Affiliate activity
Teams therefore need a shared understanding of the experiment.
The operator should document:
Which players are held out
Which campaign is being tested
What communications remain permitted
How long the holdout lasts
This protects test integrity.
Use sufficiently large CRM test groups
Small samples can produce volatile results.
A handful of high-value players can dramatically change:
Revenue
Deposit value
NGR
This is particularly relevant for VIP segments.
A tiny VIP test may be commercially interesting but statistically unreliable.
In these cases, operators may need:
Longer test periods
Repeated experiments
Larger combined cohorts
Carefully matched analysis
rather than drawing a definitive conclusion from one small campaign.
Manage channel overlap in CRM control groups
Multi-channel campaigns create another measurement issue.
If the treatment journey includes:
Email
Push
SMS
and the control receives none of them, the experiment measures the incremental effect of the overall journey.
That may be exactly what the operator wants to understand.
However, it does not reveal which individual channel caused the uplift.
Test individual CRM channels separately when needed
If the business wants to know whether SMS adds value beyond email, the test design needs to reflect that question.
For example:
Group A
Email only
Group B
Email + SMS
Comparing those groups can estimate the incremental contribution of SMS.
Similarly, operators can test:
Email versus push
Incentive versus no incentive
One message versus multiple messages
Control-group design should always match the commercial question.
Measure player profitability rather than deposits alone
Deposits are not the same as value.
A reactivation campaign may increase deposits while reducing profitability because of:
Bonus expense
Withdrawal patterns
Low margin
Short-lived activity
CRM measurement should therefore align with the commercial metrics the business actually cares about.
These might include:
Net gaming revenue
Incremental contribution
Repeat deposits
Retention
Player lifetime value
rather than simply deposit volume.
Turn CRM control groups into an operating discipline
The strongest outcome of this case study was not one successful experiment.
It was the creation of a repeatable measurement process.
The operator began introducing persistent holdouts into selected lifecycle programmes.
A small proportion of eligible players remained outside the campaign as an ongoing baseline.
This made it possible to measure cumulative CRM impact rather than running isolated one-off tests.
What is a persistent CRM holdout group?
A persistent holdout is a group of eligible players who remain excluded from selected CRM activity over a longer period.
This allows operators to compare:
Marketed players
Unmarketed eligible players
across several campaigns or lifecycle journeys.
The approach can provide a clearer view of total CRM incrementality.
Persistent holdouts have trade-offs
Holding customers out of potentially profitable marketing can feel uncomfortable.
The control group creates an opportunity cost if some excluded players would have responded positively to the campaign.
For that reason, the appropriate holdout size depends on:
Audience volume
Campaign value
Test duration
Player segment
Statistical requirements
There is no single percentage that every operator should use.
Large lifecycle programmes may support a modest persistent holdout.
Smaller or seasonal campaigns may be better suited to shorter experiments.
Incremental value per eligible player
The control-group framework gave different teams a useful common metric:
Incremental value per eligible player
This shifts CRM discussion away from vanity metrics such as:
Open rate
Click-through rate
Raw campaign revenue
and towards the commercial value created by the intervention.
That metric can help inform:
Incentive strategy
Contact frequency
Creative investment
CRM budget
Acquisition versus retention spend
It also reduces the risk that several channels claim credit for the same player revenue.
Control groups help account for external market factors
Player behaviour is affected by more than CRM.
Examples include:
Major football fixtures
New product releases
Competitor promotions
Payment issues
Regulation changes
Seasonal behaviour
A control group does not remove these external factors.
Instead, randomisation means treatment and control players are exposed to the same market environment during the test.
This makes it easier to isolate the incremental effect of the campaign.
How to run a CRM incrementality test
A practical process can follow several stages.
1. Choose a meaningful campaign
Start with a journey that has:
Sufficient audience volume
Clear targeting rules
Meaningful cost
A commercial decision attached to the result
Good candidates might include:
Reactivation promotions
Onboarding incentives
VIP retention
Bonus-led lifecycle journeys
2. Define eligibility
Establish exactly which players qualify for the campaign.
Apply the same eligibility criteria before treatment and control groups are created.
3. Randomise the holdout
Randomly split eligible players between treatment and control groups.
Avoid using opted-out or otherwise ineligible players as the control.
4. Define success before launch
Agree the primary and secondary outcomes before sending the campaign.
Possible measures include:
Incremental NGR
Incremental deposits
Reactivation
Bonus cost
Retention
Player value
5. Choose the measurement window
Allow enough time for the intended player behaviour and commercial outcome to develop.
6. Protect the control group
Prevent control players from entering overlapping activity that would contaminate the experiment.
7. Measure the difference
Compare treatment and control results using the same definitions and time period.
8. Analyse segments
Examine whether incremental impact varies according to:
Historic value
Inactivity
Product preference
Acquisition source
Player tenure
9. Change the CRM strategy
Use the findings to alter:
Incentives
Targeting
Message frequency
Channel mix
Audience exclusions
The objective is action, not another dashboard.
How Cognaix approaches CRM incrementality
Cognaix approaches CRM measurement as an execution and efficiency problem.
The purpose of a CRM control group is not simply to produce a statistically interesting report.
It is to help iGaming teams understand:
Which campaigns genuinely change player behaviour
Which segments create incremental value
Where bonus cost can be reduced
Which journeys deserve additional investment
Where CRM activity creates unnecessary contact or spend
The strongest tests connect CRM response with commercial outcomes.
That allows teams to move investment towards campaigns that change behaviour profitably rather than simply generating attractive recipient-level reporting.
Final thoughts
A strong CRM control group case study should answer a much more useful question than whether campaign recipients generated revenue.
It should determine how much additional value the CRM activity actually created.
Randomised holdout groups allow operators to establish a baseline for normal player behaviour.
That makes it possible to distinguish:
Campaign-attributed revenue
from:
Incremental campaign value
The difference can materially change decisions around:
Incentives
Audience targeting
Reactivation
Campaign frequency
CRM investment
The best place to begin is usually a high-cost lifecycle journey where the reported results look impressive but the incremental impact is still unknown.
Build a valid control group, agree the outcome before launch and allow enough time for player value to develop.
The result may show that the campaign works.
It may show that it works only for certain players.
Or it may reveal that much of the reported revenue would have occurred anyway.
All three outcomes are commercially useful.
Frequently asked questions
What is a CRM control group?
A CRM control group is a randomly selected group of eligible customers who do not receive the campaign being tested, allowing their behaviour to provide a baseline for comparison.
Why are control groups important in iGaming CRM?
They help operators separate campaign-driven behaviour from deposits, wagering or reactivation that would have happened naturally.
What is CRM incrementality?
CRM incrementality is the additional commercial outcome caused by a CRM intervention above the baseline behaviour observed without that intervention.
How do you create a CRM holdout group?
First define the eligible campaign population, then randomly assign a proportion of those players to a holdout group that does not receive the tested campaign.
Can opted-out customers be used as a CRM control group?
Generally, they are not an appropriate equivalent control for a marketing campaign because they may differ systematically from the eligible treatment population. Randomisation within the eligible audience creates a stronger comparison.
How long should a CRM control-group test run?
The correct duration depends on the campaign objective and product. Short event-led campaigns may need a brief window, while casino reactivation or retention tests may require several weeks.
What should an iGaming CRM control group measure?
Useful outcomes can include reactivation, deposits, net gaming revenue, bonus cost, retention, withdrawals and incremental contribution.
What is a persistent CRM holdout?
A persistent holdout keeps a small proportion of otherwise eligible players outside selected CRM activity over a longer period so the operator can estimate cumulative CRM incrementality.
Should CRM teams measure revenue or incrementality?
Both can be useful, but incrementality provides a stronger estimate of the value actually caused by the campaign rather than simply the revenue generated by recipients.