Ten Years That Changed Evaluation · Part 3 of 12

Causal inference beyond the RCT: theory-based methods come of age

Andrej Salner · researched and written with AI assistance · September 2026 · links checked September 2026

The causal-inference repertoire in 2026Experimental / counterfactualTheory-based / case-basedRCTs (2019 Nobelprize)Synthetic controls,causal forestsQuIP (blindedqualitativeattribution)QCAContributionanalysis &contribution rubricsProcess tracing,Bayesian updatingRealist evaluation(RAMESES II),outcome harvestingPosition indicates the dominant logic of inference, not a ranking of rigour.
Methods that became mainstream alongside RCTs over the decade. Source: Deaton & Cartwright 2018 (pre-print). Figure: evaluacia.sk, based on the sources cited in the article.

What changed

Two things happened at once. RCTs reached their apogee — the 2019 Nobel prize to Banerjee, Duflo and Kremer, the scale-up of J-PAL and IPA, 3ie's evidence gap maps — and theory-based causal inference professionalised into a credible alternative for the large majority of interventions where randomisation is impossible, unethical or beside the point. Realist evaluation acquired reporting standards (RAMESES II, 2016–17).

Process tracing was hybridised with Bayesian confidence updating by Barbara Befani and colleagues ("contribution tracing", from an Oxfam Ghana case). Qualitative Comparative Analysis (QCA) was road-tested in development cooperation. Outcome harvesting matured from a technique into a book (Wilson-Grau, 2018) and a donor-endorsed approach. The Qualitative Impact Protocol (QuIP) from Bath SDR offered a low-cost, "blindfolded" qualitative attribution method.

Rubrics — including "contribution rubrics" and quality-of-evidence rubrics — became a way to make expert judgement transparent rather than hidden. John Mayne's Revisiting Contribution Analysis (2019) kept CA at the centre but connected it to these newer tools. On the quantitative side, synthetic control methods and machine-learning estimation of heterogeneous treatment effects (causal forests) migrated from econometrics into applied evaluation.

Frameworks and tools

  • Realist evaluation and CMO (context–mechanism–outcome) configurations
  • process tracing with hoop, smoking-gun and doubly-decisive tests
  • Bayesian confidence updating and contribution tracing
  • QCA
  • outcome harvesting
  • QuIP
  • causal link monitoring
  • contribution rubrics
  • evidence gap maps
  • pre-analysis plans and pre-registration
  • synthetic controls
  • causal forests

Vocabulary to recognise

  • Mechanism
  • generative versus successionist causality
  • small-n
  • causal pathway
  • hoop test
  • smoking gun
  • confidence updating
  • evidence gap map
  • pre-registration
  • heterogeneous treatment effects
  • external validity
  • equipoise
  • "randomista"

Live controversies

Angus Deaton and Nancy Cartwright's 2018 paper remains the reference critique of RCT primacy: randomisation delivers unbiased estimates of an average effect in one place and time, and says little on its own about mechanisms or transfer. Lant Pritchett's provocations ("the debate is over — we won, they lost") and Martin Ravallion's "Should the Randomistas (Continue to) Rule?" bracket the post-Nobel argument; World Development ran a symposium in 2020.

The 2015 "Worm Wars" — a public re-analysis of the Miguel–Kremer deworming trial — became the case study for replication, open data and the limits of a single study. Within the theory-based camp, sceptics ask whether Bayesian updating in process tracing is rigour or rhetorical veneer, and whether outcome harvesting's reliance on informants is adequately triangulated.

On evidence synthesis, the growth of gap maps and systematic reviews has been criticised for privileging the kinds of interventions that lend themselves to trials.

Further reading

About the author. Andrej Salner is an economist and evaluator, co-founder of the Slovak Governance Institute and of evaluacia.sk. He has designed and led evaluations for Slovak ministries, EU and EEA programmes and NGOs since 2004. More about the team · Impact measurement & evaluation services.
A note on authorship and method. This series was researched and written by Andrej Salner with the assistance of AI tools (large language models used for literature search, summarising and drafting). The framing, the selection of sources, the judgements and the final text are the author's, and the author takes responsibility for them; links were checked in September 2026. The Slovak version is an AI-assisted translation of the English original, reviewed by the author. Corrections and additions are welcome at info@evaluacia.sk.
← PreviousNext →