A/B Testing and Privacy: can it work?
Hi,
The first deep dive in the new format - yay!
A question I get a lot is can we run experimentation programs and personalize while being compliant?
Today we will focus on Part 1 of the question: experimentation, more specifically A/B testing.
The basics:
Experimentation programs rely heavily on A/B testing to understand what layout, changes, or additions work best for their users. It's one of the growth levers a business uses to increase it's revenues, AOV, or conversion rate of a specific metric. It is also utilised to run sanity tests that verify nothing is broken when releasing a new version, tool, or script on the site.
In short - it's essential to growth.
The way A/B tests are run is that you show a specific treatment (of whichever change you have made) to random users while measuring how the change affects the metrics you are testing for. To make sure any specific user sees the same version of the experiment for the whole duration of the test a cookie is set that stores all sorts of information and behaviours.
The issue:
...a cookie is set that stores all sorts of information and behaviours.
A big questions for business that are either in, targeting, or selling to the EU is if one can run an a/b test considering that it requires you to drops cookies (the CCPA - and other US regulations - do not require explicit consent but rather work with an opt out model so they are not as affected).
The ePrivacy Directive (this is not really a GDPR issue) says that you need to get consent for all not strictly necessary cookies. Yes, that is even for cookies that do not hold any personal data.
To make it all more fun the ePrivacy Directive is interpreted slightly different by all individual EU member states.
But most agree that strictly necessary means anything that is required to make the site function - nothing else.
Load-balancing is ok but analytics is not.
This would imply that you can not drop a cookie for you A/B test unless you have consent. And you can not drop a cookie pre-consent so you have to wait to get consent first.
This pretty much eliminates running an experiment on your home page or landing page. It also limits your sample size to the users who do consent (anywhere from 20-60% depending on device and country).
A Solution (or rather, my way of thinking on this)
(This is not legal advice and always check with your DPO or legal team first)
In short - run the experiment.
Or, consider a soft-opt in for A/B testing cookies.
Yes, I'm essentially saying that even thought you are dropping a cookie and you did not ask you users consent you should run the experiment. Especially if you are an eCommerce or SaaS.
Why?
First of all because countries are not on the same page regarding if A/B testing is an exemption or not.
The ePrivacy directive is applied by each country as it is not a regulation that has to be enforced in the same manner across the EU.
For example the CNIL - that is the supervisory authority in France - has an exemption for cookies used for A/B testing. Whereas the ICO, the Brits, say clearly that you can't use an exemption for A/B testing.
Some other countries have not been tested or have not voiced their opinion.
Second, you can evaluate the risk. It's essential, when making any privacy vs growth decisions, to evaluate the risks involved.
In most scenarios I would say the risk is quite low for A/B testing considering:
-
the test is only running for a limited time
-
that one usually collects aggregated data with no intention to understand users individual behaviours.
-
that testing is an expectation by the user as it is a technique utilised to improve their user experience.
So....How?
(Again, make sure you run this by your legal team or DPO - I am not a lawyer and this is not legal advice.)
When I try to evaluate if it is an experiment that could be run I keep it simple.
What is my consent rate?
If it's above 50-60% and that is enough traffic to reach a decent sample size then only run the experiment when you have consent.
Is soft-opt in an option?
Can I use soft opt in to gather consent with this? Is it possible to be clear and transparent so that the user knows what it's all about?
What are the risks or running the experiment?
Consider the risks to your users data and determine what the risks are.
Regardless of all of the above make sure to:
-
Be clear and transparent about what you are doing
-
State why you are doing it (the purpose)
-
Add experimentation to your Privacy Notice
-
Make it easy to opt out
In other news:
-
The "enshittification" of TikTok - not a privacy read but still interesting on "where platforms go to die"
-
You are probably doing privacy UX wrong - a great read by Luiza Jarovsky (she also has a great newsletter called the pRivacy Whisperer)
-
Replika get scolded for violating child safety - this is a precursor as to the risks involved with AI and how privacy is affected.
Privacy Hall of Shame:
There are a lot of consent banners out there that are not compliant - and most I won't pick on. But this one has out done itself.
They try - as in the Reject All and Accept All buttons appear at the top for a brief moment - and then they are gone. Instead they are asking us to scroll through every single cookie they use and select each individually.
(Quality is shit - I know. It's a gif of a screen recording i took on my phone.)
Closing Thought
It's about balance.
It's not about going to extremes.
A business needs to survive and grow. A user/customer needs to be respected. Laws need to be abided by.
But all of this make it seem nearly impossible - you just can't do all three really well.
Find balance.
Consider risks.
Most importantly - consider your user/customer.
We can respect our customer, be pretty damn compliant, and grow our business - it just won't work if you take everything to an extreme.
Don’t let privacy fuck with your growth, have it work for your growth instead.
Have a great week,
Siobhan