Personalization or A/B testing: what your traffic level actually supports
A test that needs four months to finish is not a decision tool. How to work out what your pages can support, and what to use when testing is too slow.
- Conversions decide, not preference. Whether a page can support A/B testing depends on how many conversions it records each month, because that sets how long a test needs to reach a reliable answer.
- The calculation is fixed. At a 2% baseline, detecting a 20% relative lift at 95% confidence and 80% power takes 21,109 visitors per variant: 4 weeks on a page with 50,000 monthly visitors, 19 weeks on one with 10,000.
- Personalization asks a different question. It responds to each visitor’s behavior in the session instead of splitting traffic to find one winner, so it does not wait on a queue of tests.
- It still has to be proven. Personalization needs a randomly assigned control group to show uplift, and below roughly 100 monthly conversions on one event neither method gives a fast, reliable read.
Whether to A/B test or personalize is decided by how many conversions a page records each month, not by which method you prefer. A page with around 1,000 monthly conversions can finish a well-sized test in about four weeks. A page with 200 needs more than four months for the same test, and at that pace intent-based personalization, measured against a control group, is the method the traffic supports.
Most sites with 50,000 to 500,000 monthly visits sit between those two cases, and the answer differs page by page. A home page may carry enough volume to test. A pricing page, a product category or a paid landing page often does not, even though that is where the ad budget lands.
This article shows the calculation, so you can run it on your own numbers, and then turns it into a decision table. It also says when A/B testing is plainly the better tool, and when neither method will tell you anything.
Count conversions per month, not visits
A test does not learn from visitors. It learns from the difference in conversions between two groups, and that difference has to be large enough to stand out from random variation. Two inputs set the bill: the page’s baseline conversion rate and the minimum detectable effect you choose. Put together, they come out as a number of conversions each variant has to collect, and that number is almost the same whether the page converts at 1% or at 5%.
This is why sites with respectable traffic still stall. A mid-traffic site rarely runs out of ideas to test; it runs out of conversions to test them with. Total visits are spread over dozens of pages, and the pages that matter commercially often carry a small share of them.
Count one event, on one page, for one audience. Which event depends on the business:
| Business | Conversion event to count | Where it is usually recorded |
|---|---|---|
| E-commerce | Purchase | Product page, cart and checkout |
| B2B SaaS | Demo request or trial start | Pricing, product and demo pages |
| Travel | Booking | Tour, itinerary and availability pages |
| Healthcare and clinics | Appointment request | Treatment and booking pages |
| Education | Enquiry or enrollment | Course and application pages |
Agencies should run the count on each client’s own event. Do not add events together to reach a bigger number: 300 purchases and 200 newsletter sign-ups are not 500 conversions, and a test that mixes them measures nothing a business can act on. For the wider diagnosis of why a page converts below expectation, start with the conversion rate optimization guide.
A mid-traffic site rarely runs out of ideas to test; it runs out of conversions to test them with.
The sample size calculation, with the numbers
The standard calculation for comparing two conversion rates gives the visitors needed in each variant:
n = ( za × √(2 × p × (1 – p)) + zb × √(p1 × (1 – p1) + p2 × (1 – p2)) )2 ÷ (p2 – p1)2
p1 is the baseline rate, p2 the rate you want to be able to detect, and p their average. za is 1.96 for 95% confidence (two-sided) and zb is 0.84 for 80% statistical power.
Worked through for a page converting at 2% and a 20% relative lift (2% to 2.4%), the answer is 21,109 visitors per variant, or 42,218 for a two-variant test. The control records about 422 conversions on the way. The first table gives the same calculation for other baselines and lifts.
| Baseline conversion rate | 10% relative lift | 20% relative lift | 30% relative lift |
|---|---|---|---|
| 1% | 163,095 | 42,693 | 19,827 |
| 2% | 80,682 | 21,109 | 9,798 |
| 3% | 53,211 | 13,914 | 6,455 |
| 5% | 31,234 | 8,158 | 3,780 |
Visitors needed per variant. Two-proportion z-test, two-sided, 95% confidence, 80% power, equal split between two variants. Calculated for this article.
Visitors per variant become weeks once you know how many people reach the page. The second table holds the baseline at 2% and divides the total sample by weekly visitors (monthly visitors × 12 ÷ 52), rounded up to whole weeks.
| Monthly visitors to the page | Conversions a month at 2% | 10% lift | 20% lift | 30% lift |
|---|---|---|---|---|
| 5,000 | 100 | 140 weeks | 37 weeks | 17 weeks |
| 10,000 | 200 | 70 weeks | 19 weeks | 9 weeks |
| 20,000 | 400 | 35 weeks | 10 weeks | 5 weeks |
| 50,000 | 1,000 | 14 weeks | 4 weeks | 2 weeks |
| 100,000 | 2,000 | 7 weeks | 2 weeks | 1 week |
| 200,000 | 4,000 | 4 weeks | 1 week | 1 week |
Weeks to complete one two-variant test at a 2% baseline. Assumes every visitor to the page enters the test and traffic is steady. A result of one week should still run for two, to cover full weekly cycles.
Three things follow. A 10% lift is out of reach within a quarter for any page under about 55,000 monthly visitors at this baseline. A 30% lift is detectable quickly, but few single changes are that large. And every extra variant adds another full sample to the total, so a three-variant test takes half as long again as the figures above.
Where A/B testing works and where it stalls
Those figures come from Georgi Georgiev’s meta-analysis of 1,001 tests (Analytics Toolkit, 18 October 2022), where the median minimum detectable effect was a 6% lift. They describe well-run programs on sites with enough volume to plan for small effects, and even there two tests in three did not produce a winner.
A/B testing is the right choice when the page records enough conversions to finish in weeks and the question has one answer for everyone. A checkout step, a price presentation, a form length, a plan layout: each visitor should end up with the same version, and you want to know which. On a page with 1,000 or more conversions a month, a test sized for a 20% lift completes in four weeks or less, and a measured answer at that speed beats any prediction.
It stalls for three reasons on mid-traffic sites. The first is duration: at 200 to 400 conversions a month the same test takes 10 to 19 weeks, long enough for campaigns, seasons and the traffic mix to change underneath it. The second is yield: most tests do not find a winner, so a slow queue produces very few changes a year. The third is the answer itself. A winner is the version that performs best on average, and on a page that receives first-time visitors from paid social next to returning visitors ready to act, the average describes neither.
- The page records 1,000 or more conversions a month
- The question has one answer for every visitor
- The sample size is fixed before the start
- One test takes 10 weeks or more
- Visitors arrive with different intent
- The expected lift is under 10%
- Stop at the first significant reading
- Add different conversion events together
Common mistake: stopping a slow test the first time the dashboard shows significance. Evan Miller’s analysis of repeated significance testing (18 April 2010) shows that checking after every observation and stopping at the first significant result can raise the false positive rate from the intended 5% to 26.1%. Fix the sample size before the test starts and run full weeks: Optimizely’s documentation (updated 20 June 2025) sets a minimum of one business cycle of seven days, and Peep Laja at CXL (updated 20 December 2022) recommends “1 or better yet 2 business cycles”.
What intent-based personalization does differently
Intent-based personalization does not split traffic to find one winner. It responds to what each visitor does during the session. The unit of decision moves from the page to the visit. Two people on the same product page can be at opposite ends of a decision, and their behavior shows it: how far they scroll, which pages they open, how long they stay. A visitor still comparing options and one checking delivery terms do not need the same prompt.
That changes what low conversion volume costs you. A test needs the full sample before anything on the page changes. A system that reads behavior starts responding with the first sessions, and it can treat the visitors who are not ready to convert yet differently from the ones who are, instead of choosing one version for both.
How Pathmonk applies it
Pathmonk runs on first-party, cookieless behavioral data from your own website. It reads in-session behavior, predicts the visitor’s buying stage, and serves a microexperience matched to that stage: an explanation for someone still learning, proof for someone comparing, a clear path to the purchase, demo request, booking, appointment or enquiry for someone ready. It is installed with one snippet and leaves the page’s own content as it is. The product page covers the detail.
Personalization does not answer single-answer questions. If you need to know whether a shorter form or a different price presentation is better for everybody, that is a test, and no amount of intent reading replaces it.
Why personalization still needs a control group
Responding in the session is a mechanism, not a result. The only way to know that personalization raised conversions is to keep a randomly assigned control group that never sees it, and compare the two on one conversion event. A before-and-after chart cannot do this, because a new campaign, a seasonal peak or a change in traffic mix moves the line just as easily.
The comparison obeys the same arithmetic as the tables above. A 50/50 split between personalized visitors and control needs the same sample as any two-variant test to detect the same lift. Three things make it workable where a testing queue is not: it is one standing comparison instead of a series; it runs across every page where personalization is active, so it draws on the whole site’s conversions for that event; and visitors are being served while the evidence accumulates.
Pathmonk is built this way. Its help center (updated 10 November 2025) describes the assignment: each visitor is placed at random in a group that can see microexperiences or in the control group before anything is shown, and conversions are recorded for both. The split starts at 50/50 until the result reaches 95% confidence, and a control of 5% stays in place afterward. The same page advises running at least one to two months before drawing conclusions, longer on lower-traffic sites.
A published example of the method: in the Forrest Technical Coatings case study, Pathmonk ran on a 50/50 traffic split and the sales request conversion rate was 2.43% for the personalized group against 1.49% for the control.
- Random assignment on arrival
- A control that sees the unmodified site
- One conversion event
- A sample size fixed in advance
- Before-and-after comparisons
- Conversions counted only among visitors who engaged
One caution on the small control. A 5% group collects conversions slowly, so it is a way to keep checking a result that was established at 50/50, not a way to establish it. If the site changes substantially, go back to an even split for a period.
A decision table by traffic and conversion volume
Read the table by the conversions the page records in a month for one event. The visits column of your analytics matters only through that number: 400 conversions can come from 20,000 visitors at 2% or from 8,000 at 5%, and the test takes about the same time either way.
| Monthly conversions on the page | One test for a 20% lift | A/B testing | Intent-based personalization |
|---|---|---|---|
| Under 100 | More than 37 weeks | Not practical | Control group cannot give a reliable read within a quarter either |
| 100 to 400 | 10 to 37 weeks | Stalls: one to five tests a year | Fits, measured at 50/50 on the sitewide event over months |
| 400 to 1,000 | 4 to 10 weeks | For large, single-answer changes, one at a time | Fits for visitors with different intent |
| 1,000 to 4,000 | 1 to 4 weeks | Right choice for single-answer decisions | Useful alongside, with its own control group |
| Over 4,000 | 1 week (run 2) | Right choice, including 10% lifts in about 4 weeks | Useful alongside, with its own control group |
Durations use the calculation above (95% confidence, 80% power, two variants) at a 2% baseline. Other baselines between 1% and 5% change them by a few percent.
The two methods are not rivals at the top of the table. A site with several thousand conversions a month can test the single-answer questions on its busiest pages and personalize for intent across the rest, as long as each has its own random assignment so the effects can be read separately.
Below roughly 100 conversions a month on one event, neither method produces a reliable read in a useful time, and choosing between them is the wrong question. Check the conversion tracking, the offer and the quality of the traffic first. CRO for low-traffic sites covers what to do at that stage. The same applies when most paid traffic lands on pages that cannot convert it: that is an acquisition problem, and no on-site method fixes it.
More on website personalization
This article is the starting point for Pathmonk’s website personalization content. Each topic below goes deeper into one part of the mechanism.
- AI website personalization. How a model decides what to show each visitor, and what it needs to learn from: how AI can help decide which website content to show each visitor.
- Intent-based personalization for anonymous visitors. What can be read from a session when you know nothing else about the person.
- Popup and exit intent alternatives. Whether interrupting at the moment of leaving still works: are exit-intent popups still effective or outdated?
- Cookieless personalization. Personalizing from first-party, in-session behavior without third-party cookies: real-time personalization in a cookieless environment.
- Personalization by industry. E-commerce purchases, B2B SaaS demo and trial requests, travel bookings, clinic appointments and education enrollments, each treated on its own terms.
- Website personalization as an agency service. How agencies deliver and report it for client websites.
Key takeaways
- The choice between A/B testing and personalization depends on monthly conversions per page, not on visits or preference.
- At a 2% baseline, detecting a 20% relative lift takes 21,109 visitors per variant at 95% confidence and 80% power.
- Conversions needed per variant are nearly constant across baselines: about 420 for a 20% lift and about 1,600 for a 10% lift.
- A/B testing is the right choice for single-answer questions on pages with 1,000 or more conversions a month.
- On pages with 100 to 400 conversions a month, one test takes 10 to 37 weeks and most tests do not find a winner.
- Intent-based personalization responds to each visitor’s behavior in the session instead of looking for one average winner.
- Personalization uplift is only proven against a randomly assigned control group, on one conversion event.
- Below roughly 100 conversions a month on one event, neither method applies: fix tracking, offer and traffic first.
FAQs on personalization and A/B testing
Is personalization better than A/B testing?
Neither is better in general. A/B testing is the right choice when a page records enough conversions to finish a test in a few weeks and the question has one answer for every visitor. Intent-based personalization fits when visitors arrive with different needs and the page does not record enough conversions to work through a queue of tests.
How much traffic do I need to run an A/B test?
It depends on conversions more than visits. At a 2% baseline conversion rate, detecting a 20% relative lift at 95% confidence and 80% power takes 21,109 visitors per variant, about 420 conversions in the control. A page with 50,000 monthly visitors gets there in 4 weeks. A page with 10,000 needs 19 weeks.
How long should an A/B test run?
Until it reaches the sample size fixed before it started, and never less than full weeks. Optimizely’s documentation sets a minimum of one business cycle of seven days, and CXL recommends one or better two business cycles. If the calculation says more than about eight weeks, the test is too slow to be a practical way of deciding.
Can I run A/B tests and personalization at the same time?
Yes, if visitors are assigned to each at random and independently, or if the two run on different pages. Use testing for single-answer decisions on the pages with the most conversions, and personalization for visitors whose needs differ. Keep the personalization control group in place so each effect can be read separately.
Does personalization need a control group?
Yes. Without a randomly assigned group that sees the unmodified website, a rise in conversions cannot be separated from seasonality, campaign changes or a shift in traffic mix. The comparison follows the same sample size rules as an A/B test, so it also takes weeks or months to become reliable.
What if my site does not have enough conversions for either?
Below roughly 100 conversions a month on one event, a test sized for a 20% relative lift takes about 37 weeks, and a personalization control group takes just as long to read. Work on the offer, the tracking and the quality of the traffic first, and count a conversion event that happens more often only if it reliably leads to the one you care about.
- What can be learned from 1,001 A/B tests? Georgi Georgiev, Analytics Toolkit, 18 October 2022 (blog.analytics-toolkit.com)
- How not to run an A/B test, Evan Miller, 18 April 2010 (evanmiller.org)
- How long to run an experiment, Optimizely documentation, updated 20 June 2025 (support.optimizely.com)
- Stopping A/B tests: how many conversions do I need? Peep Laja, CXL, updated 20 December 2022 (cxl.com)
- What is a Pathmonk A/B test? Pathmonk help center, updated 10 November 2025 (pathmonk.freshdesk.com)