2×2 Designs: Simplifying Complex Business Problems

Book: Data Science: The Hard Parts
Author: Daniel Vaughan
ISBN: 978-1-098-14647-4
Publisher: O’Reilly Media (2024)

Previous: Growth Decompositions and Tailwinds | Next: Building Business Cases for Data Science

Vaughan opens Chapter 4 with a confession. Early in his career, a consulting firm came in and sketched simplified 2×2 views of the business. He dismissed them as sales tricks.

Now he embraces them.

The chapter is about smart simplification. Not dumbing things down. Making things as simple as possible, but not simpler. Einstein’s line, basically.

Why simplify at all?

More data doesn’t mean less complexity. You can project high-dimensional data onto lower dimensions. That doesn’t mean you understand how things work.

Simplification helps you:

• Frame what’s important when starting a project
• Communicate technical results to stakeholders
• Apply Occam’s razor when choosing models

Feynman said it: if you can’t explain something in simple terms, you don’t understand it.

What’s a 2×2 design?

Pick two factors you believe matter. Put them on horizontal and vertical axes. Set threshold levels to create four quadrants (A, B, C, D).

In experiments, factors are treatments (banner color, message type, price level). Randomization keeps everything else constant on average. That’s the ANOVA setup.

In non-experimental use, you lose that guarantee. But you still focus on just two features. Vaughan commonly uses this with multiplicative decompositions from Chapter 2. Unit price on one axis, transactionality on the other. Quadrant A is high price, high transactions. Your best customers.

Use case 1: test a model and a lever

This is my favorite example.

You trained a cross-selling model. You also want to test a new communication campaign vs. the baseline. A standard A/B test only checks the lever. A 2×2 design checks both.

Horizontal axis: model probability score (high vs. low). Vertical axis: lever on or off.

• Groups A and B get the treatment (new campaign)
• Groups C and D are control (baseline campaign)

You can test two hypotheses:

Monotonicity: higher scores should mean higher conversion. CR(A) > CR(B) and CR(D) > CR(C).

Effectiveness: the new campaign should beat baseline for high-score users. CR(A) > CR(D). Low-score users shouldn’t respond much. CR(B) ≈ CR(C).

Sample size is the catch. You might have enough power for the lever test but only casual evidence for model performance. Vaughan says that’s usually fine. But go all in if you can.

Use case 2: understand user behavior

Vaughan used a 2×2 to find product-market fit. Engagement on one axis, unit price on the other. Quadrant A users (high on both) got labeled as 1. Everyone else as 0.

He trained a classifier to understand who these users are. Then opened the black box to learn what makes them special. He calls this combination “profitable fit.”

Same logic works for customer lifetime value (LTV). LTV is a discounted sum of revenue and survival probability streams. Collapse those streams to one dimension each (average, median, or pick a critical time window like six months). Plot survival proxy vs. revenue. Label quadrant A. Train a model. Score the full user base. Estimate opportunity size.

None of the aggregation methods are perfect. Younger cohorts look worse on cumulative revenue. Vaughan acknowledges the tradeoffs.

Use case 3: correlated outcomes (credit)

Loan origination has a nasty problem called adverse selection. Riskier borrowers are more likely to accept expensive offers. Acceptance probability and default risk are positively correlated.

The 2×2 grid maps acceptance likelihood vs. default risk. Quadrant B is the sweet spot: likely to accept AND likely to repay. Target those customers.

You can also move thresholds to trade risk for volume. Credit originators do this calibration constantly based on risk appetite.

Use case 4: prioritize your workflow

The classic consulting grid. Value on one axis, effort on the other.

High value, low effort (top left) is where projects should live. Project X and Y have similar value, but X takes less effort. Easy choice.

The limitation: how do you compare project X (top left) with project Z (top right)? Both are high value but very different effort levels. Vaughan points to Chapter 5’s business case framework for that.

What I took away

I used to think 2×2 grids were oversimplified consultant nonsense. Vaughan convinced me they’re a legitimate tool when used correctly.

The model-plus-lever experiment design alone is worth knowing. Testing whether your model actually ranks users correctly in production, not just on a holdout set, is something most teams skip.

And the LTV segmentation approach is practical. Label your best quadrant, train a classifier, interpret the features. It’s not fancy. It works.

The chapter’s honest about limits. Non-experimental 2×2s can’t control for confounders. Aggregation choices for time-series metrics are messy. The value-vs-effort grid can’t rank everything.

But as a starting point for thinking and communicating? Hard to beat a four-quadrant picture.