Understanding Politics

Reading studies

This guide explains, in plain language, how to read the studies cited on this site. You do not need to know statistics. Each term is explained where it first appears.

The core question: what would have happened without the policy?

Saying a policy "worked" means comparing what happened with what would have happened without it. But nobody can observe the "without it". Each type of study below is a different way to estimate that comparison, and each can show some things but not others.

Types of study

Descriptive

Shows how something changed or how groups differ, such as average income by year. It can show what happened. It cannot show why: other things changed at the same time. In the GDP per capita guide, the study of inequality and poverty in Brazil between 1981 and 2004 is descriptive.

Decomposition

Splits a change into accounting parts, for example how much of a fall in inequality came from each income source. It is arithmetic, not an experiment. It tells you how much each part weighed in the sum, not what would have happened if that part had not existed. An example is in the Progresa/Oportunidades policy brief, in the income inequality section.

Observational

Compares people or places without the researcher controlling who gets the policy. It shows associations. Because the groups may already have differed beforehand, it does not prove cause. The GDP per capita guide has two examples, on income and reported well-being, that seem to point in opposite directions.

Randomized experiment

Draws lots for who receives the policy and who stays in the comparison group. Because the draw balances the groups, a later difference can be attributed to the policy. It is the strongest design for showing cause. Limits: drawing lots is not always possible or ethical, and the result may not apply elsewhere.

Difference-in-differences

Compares the change over time in a group reached by the policy with the change over the same period in a group not reached. It shows cause if the two groups would have changed in a similar way without the policy. That assumption cannot be fully tested. The GDP per capita guide has a similar study, comparing Brazilian regions with larger and smaller import tariff cuts.

Regression discontinuity

Uses a cutoff rule, such as a maximum income to enter a program. It compares people just below the cutoff with people just above it, who are very similar. It shows cause, but only for people near the cutoff.

Synthetic control

Builds a "twin" of a country or region by combining others that did not get the policy, then compares the two over time. It is useful when only one unit was affected. It depends on finding a good twin. The GDP per capita guide has examples, such as the Manaus Free Trade Zone and German reunification.

Instrumental variables

Uses an outside factor that pushes people toward the policy but does not affect the outcome any other way. It shows cause if that factor is well chosen, which is hard to prove.

Systematic review and meta-analysis

A systematic review searches for and gathers, using rules written in advance, all the studies on a question, instead of picking favorites. A meta-analysis numerically combines the studies' results; many systematic reviews include one, but not all. It is usually more reliable than a single study, but it is only as good as the studies it gathers and may mix different programs and countries. The Progresa/Oportunidades policy brief uses a systematic review of cash transfers.

How to read the numbers

Effect size

How much the outcome changed because of the policy, in the outcome's own units or compared with its natural variation. An effect can be statistically clear and still too small to matter in real life. Always ask "how much?", not only "was there an effect?". In the GDP per capita guide, the estimate of how much German reunification reduced West German GDP per capita is an effect size.

Confidence interval

Every study uses a sample, not the whole population, so its estimate carries uncertainty. The confidence interval is the range of plausible values. For a 95% interval, if we repeated the study many times, about 95 intervals out of 100 would contain the true value. A wide interval means an imprecise estimate. If the interval includes zero, the study cannot rule out "no effect".

P-value and "significant"

The p-value answers: if the policy had no effect at all, how often would we see a result at least this large by pure chance? By convention, below 0.05 (5%) the result is called "statistically significant". That does not mean the effect is large or important, and a p-value above 0.05 does not prove there is no effect: the study may have been too small to detect it. In the GDP per capita guide, one study finds no significant impact of more recent economic liberalizations; that does not prove the effect was zero.

External validity

Whether the result holds outside where it was measured. A program evaluated in poor rural communities may work differently in a big city or another country. That is why the site's policy briefs have an external validity section. The Progresa/Oportunidades policy brief shows one case: the program was evaluated in Mexico, and the text explains why its results do not transfer automatically to Brazil.

Where the study comes from

Peer reviewed versus working paper

In a peer-reviewed article, independent experts assess the work before it is published in a scientific journal and can require corrections or reject it. A working paper circulates before that assessment. It can be good, but it has not yet passed the filter, and its final version sometimes changes.

The site's rule: indicator guides use only peer-reviewed studies. This does not guarantee that a study is right, but it does mean it went through an outside check. The site also requires a DOI, the article's permanent identifier, which lets anyone look the article up. DOIs are checked against Crossref (ADR 0007).

Conflict of interest

This is when whoever did or paid for the study could gain from a certain result, for example an agency evaluating its own program. It does not make the study wrong, but it is a reason to look more closely. Good journals ask authors to disclose funding and ties.

Level of evidence in policy briefs and guides

Each policy brief gets a level based on the type of study available (ADR 0007):

The level describes the design, not the result: a policy can have high-level evidence that it does not work. And causal claims appear only where there is a causal design.