Reading Causal Inference in Python — Part 5: Alternative Experimental Designs

I'm reading Causal Inference in Python by Matheus Facure and writing about its key takeaways I find most useful for applied work. I'm aiming to solidify my existing understanding and help others avoid common pitfalls. You'll want to read my prior posts covering Part 1, Part 2, Part 3, and Part 4 before reading this last part!

Through my work in operations research, I learned to recognize optimization problems hidden as ordinary business constraints. I'm noticing a similar pattern exists in recognizing causal inference opportunities. When a standard A/B test is unavailable, the surrounding business process may still contain a usable causal design. The challenge is to recognize the pattern and check the underlying assumptions.

When You Can Randomize Regions, Not People

Suppose you want to estimate the effect of rolling out a treatment across a population of units such as cities, and you have pre-treatment panel data. Part 4 assumed post-treatment panel data exists, but we can create it ourselves through an experiment. If enough units are available, random assignment may be enough. With only a few units, randomization may leave the experiment underpowered, so choosing the units becomes part of the design.

One approach is to choose two disjoint groups whose population-weighted pre-treatment outcomes each approximate the full population. Assign one group to treatment and use the other as control. This repurposes Synthetic Control from an analysis method into an experiment-design method. Additional constraints can limit the size of the treatment group and prevent the groups from overlapping. The goal is to estimate what would happen if the treatment were rolled out across the full population, rather than only across the selected units.

Finding the optimal best groups requires checking many combinations or solving a nonconvex optimization problem. In cases where optimization is overkill, a random search over candidate treatment sets is a simpler alternative.

Randomize which selected group receives treatment where possible. If operational constraints determine the treatment assignment, the analysis must rely more heavily on the assumption that the groups would have followed similar trends without treatment.

When You Can Toggle a Treatment

A Synthetic Control experiment design works when you have a small number of units to choose from. Even with only one unit, a switchback experiment can still support a causal analysis. Switchback experiments randomize whether treatment is on or off across repeated time blocks, like a 20% off discount code for ride share. You can use the repeated treatment sequences to estimate the effect of keeping treatment "always on" rather than "always off" over the carryover window.

The carryover order $m$ is the number of previous treatment periods that can affect the current outcome. Switchback designs work best when $m$ is small. A larger $m$ leaves fewer usable sequences of consecutive treatment or control periods, which increases the variance of the estimate. You can estimate the carryover order with a regression on current and lagged treatments, provided you are willing to assume that their effects are additive. You can then use the estimated order to space the randomization decisions.

This design also assumes that future treatments must not affect current outcomes. Announcing a new program in advance, for example, could change behaviour before treatment begins and violate this assumption.

When You Can Randomize an Offer, But Not Uptake

Sometimes customers can choose whether to take up a treatment. Suppose an electricity provider wants to know whether a home-energy audit reduces winter electricity use. It randomly offers a free audit to some customers, but not everyone accepts. Customers who proceed may differ from those who decline in ways that also affect electricity use. A direct comparison between the two groups would therefore mix the treatment effect with pre-existing differences.

Let $Z$ indicate whether the offer was made, $T$ indicate whether the customer received treatment, and $Y$ denote the outcome. Randomizing $Z$ identifies the effect of making the offer available on $Y$, known as the intention-to-treat effect (ITTE). This is useful, but it measures the offer's effect rather than the effect of receiving the treatment. In cases, like our example, where there's control over who receives an offer, the ITTE understates the effect of receiving the treatment. This is because some customers who receive the offer do not take it.

To use the offer as an instrument and identify that treatment effect, four assumptions are required:

1. The instrument must be assigned independently of an individual's potential treatment and potential outcomes. Randomization is the usual way to support this assumption.
2. The instrument only affects the outcome through treatment.
3. The instrument changes the probability of receiving treatment.
4. The instrument affects uptake in only one direction.

Under these assumptions, dividing the ITTE by the increase in uptake caused by the offer identifies the Local Average Treatment Effect (LATE). This is the average effect for the compliers: customers who always take up any treatment offer made to them and do not receive the treatment otherwise. LATE is not the ATE for every customer, but it's appropriate for a cost-benefit analysis if both the costs and benefits apply mainly to customers who opt in.

You can estimate these quantities with two regressions. The coefficient on $Z$ in Y ~ Z estimates the ITTE. The coefficient on $Z$ in T ~ Z estimates the increase in uptake caused by the offer. Just apply division to calculate the LATE.

Low compliance makes this estimate noisy. Controls that strongly predict the outcome can improve precision. A DAG can display the assumptions and help analysts distinguish useful controls from possible instruments.

When a Business Rule Changes Treatment Probability

Another useful pattern appears when treatment eligibility or treatment probability changes at a known threshold, such as a minimum income or spending level. Individuals just above and below the cutoff may be comparable apart from that change. If their potential outcomes would otherwise vary smoothly through the threshold, the jump in outcomes, adjusted for the jump in treated probability, identifies a local treatment effect. This pattern is called Regression Discontinuity Design (RDD). It uses the same instrumental-variable logic as noncompliance, where you divide the jump in the outcome by the jump in treatment probability. The result is a LATE for compliers near the threshold, rather than an estimate for the whole population.

Bunching can undermine the analysis if individuals can manipulate the running variable to qualify for treatment and sort themselves around the cutoff. An unusual spike or "bunch" near the threshold in a density plot of the running variable can indicate this kind of sorting.

Final Thoughts

A strong causal inference practitioner learns to spot when an ordinary business process creates a fair causal comparison. Geography, time, imperfect uptake, and thresholds can each provide one when a standard A/B test is unavailable.

The model is rarely the hardest part. It is easy to calculate a number, but difficult to know whether that number is trustworthy. Practitioners can trust the result only when they use a good design and state clear assumptions.