A/B Testing

• A. To permanently change a product without measuring results
• B. To compare two versions of something to determine which performs better on a defined metric ✓
• C. To collect demographic information about website visitors
• D. To fix bugs in a software application

Why: A/B testing compares a control version (A) against a variant (B) by randomly splitting users between them and measuring which performs better on a predefined metric, such as conversion rate.

• A. The group of users who receive the new variant
• B. The group of users who are excluded from the test entirely
• C. The group of users who experience the existing version, serving as the baseline for comparison ✓
• D. The group of users selected by the test administrator based on their behavior

Why: The control group sees the current or original version. Its results serve as the baseline against which the variant's performance is compared.

• A. It makes the test run faster
• B. It ensures that differences in outcomes are likely caused by the change being tested rather than by pre-existing differences between groups ✓
• C. It allows testers to choose which users see the best version
• D. It reduces the total number of users needed for the test

Why: Random assignment balances known and unknown characteristics across groups, so that any measured difference in results can be attributed to the variant rather than to differences in the users themselves.

• A. The variant is guaranteed to be better than the control
• B. The observed difference between groups is unlikely to have occurred by random chance alone ✓
• C. The test has been running for the maximum allowed time
• D. The sample size is larger than the control group

Why: Statistical significance means the observed difference is unlikely under the assumption of no real effect, typically judged against a threshold such as p < 0.05. It does not guarantee the result is true or important.

• A. Defining the success metric before the test begins
• B. Stopping the test early as soon as the results look favorable ✓
• C. Calculating the required sample size in advance
• D. Testing only one change at a time

Why: Stopping early when results look good, known as peeking, inflates the chance of false positives. Tests should run until the predetermined sample size or duration is reached.

• A. To determine the exact color scheme used in the experiment
• B. To know which metric will determine whether the variant is considered successful ✓
• C. To ensure the test runs for as long as possible
• D. To eliminate the need for a control group

Why: A clear goal specifies the success metric in advance, so you can objectively judge whether the variant performed better than the control and avoid choosing metrics after seeing the results.

• A. The new design looks nicer.
• B. Let's see what happens if we change the button.
• C. Changing the checkout button from gray to green will increase the purchase completion rate because it stands out more. ✓
• D. Users will like the new homepage better.

Why: A strong hypothesis names the specific change, the expected outcome, the metric being measured, and a reason for the expected effect. Options A, B, and D are vague and cannot be tested or measured.

• A. A primary metric is measured only after the test ends, while a guardrail metric is measured during the test.
• B. A primary metric is the main success measure for the test, while a guardrail metric monitors for unintended harm to other important business areas. ✓
• C. A primary metric is always revenue, while a guardrail metric is always user satisfaction.
• D. There is no difference; the terms are interchangeable.

Why: The primary metric directly reflects the test's goal, while guardrail metrics ensure the change does not negatively affect things like page load time, churn, or customer support contacts.

• A. It determines the color of the test variant.
• B. It sets the smallest change in the metric that you care about detecting, which is used to calculate the required sample size. ✓
• C. It guarantees that the variant will win.
• D. It removes the need to measure a baseline conversion rate.

Why: The MDE is the smallest improvement worth detecting. Combined with the baseline rate, significance level, and power, it determines how many users you need in each group to reliably detect that effect.

• A. Make onboarding better for everyone.
• B. Increase the percentage of new users who complete account setup within 7 days of signing up. ✓
• C. Test whether users like the new flow.
• D. Find out if the design team's idea works.

Why: A good goal is specific, measurable, and time-bound. Option B names a clear metric (account setup completion), a population (new users), and a timeframe (7 days), making success unambiguous.

• A. To track every possible user behavior during the experiment
• B. To measure the main outcome the experiment is designed to improve ✓
• C. To serve as a backup if the experiment fails
• D. To compare the design teams' performance

Why: The primary metric (also called the Overall Evaluation Criterion) is the single main outcome the test is designed to move, such as conversion rate. Having one clear primary metric keeps the decision focused and avoids cherry-picking results.

• A. Click-through rate on the new call-to-action button
• B. Average revenue per user
• C. Page load time, monitored to ensure the change does not slow the site down ✓
• D. Number of button clicks on the homepage

Why: Guardrail metrics protect against unintended harm. Page load time is a classic guardrail because a change that improves conversions but badly degrades performance would be undesirable. The other options are typically primary or secondary success metrics.

• A. Raw counts are always inaccurate
• B. Conversion rate normalizes for different numbers of users in each group, enabling a fair comparison ✓
• C. Conversion rates are easier to hide from stakeholders
• D. Raw counts cannot be calculated in modern analytics tools

Why: If the control group has 10,000 users and the treatment has 5,000, comparing raw conversions would be misleading. Dividing by the number of users in each group gives a rate that can be compared fairly across unequal group sizes.

• A. It is sensitive to the change being tested and can move within the test period ✓
• B. It is the most complicated metric available
• C. It is guaranteed to increase whenever any change is made
• D. It is measured only after the test has ended

Why: A good metric is sensitive enough to detect a real effect from the change and measurable within the experiment's duration. Metrics that rarely move or take months to materialize make tests inconclusive or impractical.

• A. Number of times the site logo is viewed
• B. Average time spent on the About Us page
• C. Number of social media followers gained
• D. Purchase completion rate among users who start checkout ✓

Why: The checkout flow directly affects whether users complete purchases, so purchase completion rate among those who begin checkout is a direct, relevant measure of the change's impact. The other options are unrelated to the checkout experience.

• A. To make sure each group has exactly the same number of users
• B. To ensure that differences between groups are attributable to the treatment rather than to pre-existing differences ✓
• C. To speed up data collection by running both versions at the same time
• D. To guarantee that the test results will be statistically significant

Why: Random assignment balances both known and unknown confounding variables across groups in expectation, so any measured difference in outcomes can be causally attributed to the treatment. It does not guarantee equal group sizes or significance.

• A. Baseline conversion rate
• B. Minimum detectable effect
• C. Statistical power
• D. The average revenue per user after the test ends ✓

Why: Sample size calculations use the baseline rate, the minimum detectable effect, the significance level (alpha), and the desired power. Outcomes measured after the test ends are not needed in advance.

• A. It decreases, because smaller effects are easier to find
• B. It stays the same, because the effect size does not affect sample size
• C. It increases, because detecting smaller differences requires more data ✓
• D. It becomes zero if the baseline rate is high enough

Why: Required sample size is inversely related to the square of the effect size. Detecting a smaller difference relative to the noise requires substantially more observations to separate signal from random variation.

• A. The probability of rejecting the null hypothesis when a true effect exists ✓
• B. The probability that the control group performs better than the treatment
• C. The chance that the test results will replicate in a future experiment
• D. The proportion of users who are exposed to the treatment

Why: Power is the probability of detecting a real effect of at least the specified size. A common target is 80%, meaning a 20% chance of a false negative (Type II error).

• A. The test will have too little traffic to be valid
• B. Repeatedly checking results inflates the false positive rate, so the significance reported is misleading ✓
• C. The treatment group will lose users who prefer the control
• D. The sample size calculation will become invalid only if the baseline rate changes

Why: Peeking at results and stopping at the first significant p-value inflates the Type I error rate well above the nominal alpha, because random fluctuations are likely to cross the threshold at some point. Fixed-horizon tests or sequential methods are needed to control this.

• A. To make the test run faster
• B. To ensure that users assigned to each variant are comparable, so differences in outcomes can be attributed to the change being tested ✓
• C. To guarantee that the winning variant will always have a higher conversion rate
• D. To reduce the total number of users needed for the test

Why: Randomization balances known and unknown user characteristics across groups, so any observed difference in outcomes is more likely caused by the variant rather than pre-existing differences between groups.

• A. Individual page views
• B. Geographic region of the country
• C. Individual users (or user IDs), so each user consistently sees the same variant ✓
• D. Day of the week

Why: Randomizing by user keeps the experience consistent for each person and avoids contamination, where one user sees both variants and their behavior is influenced by both.

• A. A situation where the test has more than two variants, indicating the test is too complex
• B. A significant deviation between the intended and observed traffic split across variants, suggesting a problem with randomization or data collection ✓
• C. A case where the sample size is too small to reach statistical significance
• D. A mismatch between the metric being measured and the business goal

Why: If a 50/50 split produces an unexpectedly uneven number of users per group, the assignment mechanism or logging may be broken, which can bias results and makes the test results unreliable.

• A. Because the test cannot be stopped early under any circumstances
• B. To avoid stopping the test as soon as a favorable result appears, which inflates the chance of a false positive ✓
• C. Because the sample size determines which variant wins
• D. To ensure the test runs during peak traffic hours only

Why: Repeatedly checking results and stopping at the first significant p-value (peeking) increases false positives. Predefining sample size and duration based on power calculations protects the validity of the conclusion.

• A. Changing the traffic allocation mid-test to favor the better-performing variant
• B. Running the control and treatment at different times of the year to compare seasonal effects
• C. Using a consistent, hash-based assignment method so each user is reliably placed in the same group across sessions ✓
• D. Allowing users to choose which variant they want to see

Why: Consistent, deterministic assignment ensures users don't switch groups between visits, which would contaminate results. Running variants at different times or letting users self-select introduces confounding and bias.

• A. There is a 97% chance that variant B is better than variant A.
• B. If there were truly no difference between variants, a result this extreme would occur about 3% of the time. ✓
• C. The effect size is 3% larger in variant B.
• D. The test has a 3% chance of being run again with the same result.

Why: A p-value measures how likely the observed data (or more extreme data) would be if the null hypothesis of no difference were true. It does not give the probability that a variant is better.

• A. 0.01
• B. 0.05 ✓
• C. 0.50
• D. 0.95

Why: An alpha of 0.05 is the conventional threshold, meaning the team accepts a 5% risk of a false positive (Type I error) before declaring a result statistically significant.

• A. The probability of detecting a real effect when one exists. ✓
• B. The total revenue generated by the winning variant.
• C. The number of users who see each variant.
• D. The probability that the null hypothesis is true.

Why: Statistical power is the probability of correctly rejecting the null hypothesis when a true difference exists. Common targets are 80% or higher, and it depends on sample size, effect size, and alpha.

• A. Early results are always reversed by the test platform.
• B. Repeatedly checking results inflates the false positive rate. ✓
• C. Significance can only be measured after exactly 30 days.
• D. Early stopping reduces the total number of users in the test.

Why: Each peek at the data is another chance for random noise to cross the significance threshold. Continuous monitoring without correction (the peeking problem) substantially increases the chance of a false positive.

• A. Variant B is definitely better and should be shipped immediately.
• B. The result is statistically significant but the effect size is uncertain, so the business impact may be small or unclear. ✓
• C. The test failed because the confidence interval includes positive values.
• D. The p-value is meaningless when a confidence interval is present.

Why: Statistical significance does not guarantee a large or reliable practical effect. A wide confidence interval signals imprecision about the true lift, so decisions should weigh practical significance and the range of plausible outcomes.