Investing

Finance Research Replicates In Sample, but Effect Sizes Fade in New Data

Oct 12, 2026 • 5 min read • views

The central finding in Dmitriy Muravyev’s 2026 working paper is a split verdict: empirical finance is often reproducible in the data researchers originally studied, but its reported effects are much less durable in later periods. That distinction matters when published estimates are used to frame expectations about returns, risk premiums, or factor exposures. Recovering a result from its original sample confirms that the analysis can be reconstructed; it does not establish that the estimated relationship will persist.

Muravyev uses an autonomous large language model pipeline to rebuild results from 1,328 papers published between 2000 and 2020 in the Journal of Finance, Journal of Financial Economics, and Review of Financial Studies. The pipeline reconstructs each paper’s main result using the paper and available databases, with independent audits of each replication. For 1,005 results, there was enough data after the original sample to conduct an out-of-sample test.

Reconstruction works better than extension

In the original samples, about 75% of the key results retained both the published sign and statistical significance. The median replicated coefficient reached 93% of the estimate reported in the paper. Sign reversals that remained statistically significant were rare: they occurred in 1.3% of key estimates. Most results that failed to match the original instead lost statistical significance.

That is a comparatively strong showing for replication. It suggests that many published findings can be recovered when researchers revisit the same underlying period and analysis. But the out-of-sample results shift the assessment. In later data, only 42% of key findings retained their original sign and statistical significance. Adjusting for the lower precision associated with shorter post-sample periods lifts that share to 48%, still well below the in-sample rate.

Effect sizes tell a similar story. The median replicated coefficient was 93% of the published estimate within the original sample, but only 45% in later data. The difference is more informative than a binary count of significant and insignificant results: even where a relationship remains detectable, its estimated magnitude may be substantially smaller than the figure that entered the literature.

The sample boundary is where the decline appears

The timing of the change offers a clue about why results weaken. Muravyev tracks estimates year by year relative to the end of each paper’s original sample. Coefficients remain relatively stable during that sample, fall sharply around its endpoint, and then show little further decline. The estimated coefficient ratio drops by about 0.28 at the boundary. The analysis finds no clear additional fall when a paper later circulates or reaches publication.

That pattern is more consistent with selection in the original sample, including data snooping, than with a gradual change caused by investors learning about a published result or by markets slowly evolving afterward. It does not identify one definitive cause. The boundary may capture several forces tied to how results are developed and selected, and the study leaves room for changes in economic relationships, model misspecification, and other explanations.

Conventional publication selection models account for only part of the gap. Two fitted models predict that about 90% of a published coefficient should remain in later data. The observed median retention is 45%. On those assumptions, publication selection explains roughly one-fifth of the measured decay; most of the difference remains unexplained by the models.

Stronger evidence tends to last longer

Persistence varies with the strength of the original statistical evidence. Among findings in the weakest quartile, around 31% retain their sign and significance out of sample after adjustment for precision, and roughly one-third of the original coefficient remains. In the strongest quartile, approximately 72% retain their original category, while the median coefficient ratio is 0.62.

The gap suggests that a threshold result and a strongly supported result should not be treated as equally persuasive evidence of a durable relationship. Yet even the strongest quartile experiences meaningful attenuation. A large original t-statistic improves the odds that a finding will survive later testing; it does not guarantee that its original magnitude will carry forward.

What the results mean for research users

For professionals translating academic findings into investment assumptions, the practical issue is calibration. Historical coefficients can provide useful evidence about a relationship in a particular period, but the study cautions against treating those estimates as fixed inputs for future conditions. That applies to expected returns, risk premiums, factor exposures, and other assumptions informed by empirical finance.

Replication and validation answer different questions. A successful reconstruction shows that a result can be recovered from the original sample and analysis. A test on later data asks whether the relationship continues beyond the period that produced the published estimate. The study’s results make that second question essential whenever a historical finding is used to support a forward-looking assumption.

The evidence also favors attention to the full strength of a result rather than its passage across a conventional significance cutoff alone. A finding barely over the threshold has a different record of out-of-sample persistence from one supported by much stronger original evidence. In either case, later tests add information that the original estimate cannot supply by itself.

Evidence that holds across multiple samples and market periods therefore carries a different weight from a striking estimate drawn from one historical window. This paper does not imply that academic finance has little value: its in-sample replication results are broadly reassuring. Its sharper message is that reproducibility should not be mistaken for durability, and that the size of an effect deserves scrutiny alongside whether the effect can be reproduced.

Reading the chart

The chart shows the median ratio of replicated to published coefficients by year, measured against the end of each paper’s original sample. The line is roughly flat within the sample, drops around the sample boundary, and remains comparatively flat afterward, including after circulation and publication. Its shape reinforces the paper’s main interpretation: the decline is concentrated at the point where the original data stop.

The results are hypothetical results and are NOT an indicator of future results and do NOT represent returns that any investor actually attained. Indexes are unmanaged and do not reflect management or trading fees, and one cannot invest directly in an index.

Study details

Dmitriy Muravyev’s working paper, “Does Empirical Finance Replicate?”, reports 1,328 in-sample replications and 1,005 out-of-sample tests. Its headline comparison is 75% replication of published sign and significance in sample, versus 42% in later data, with median coefficient ratios of 0.93 and 0.45, respectively. The author concludes that published finance is largely reproducible within its original samples but substantially less durable beyond them.

  • Dmitriy Muravyev
  • Working Paper, 2026
  • A version of this paper can be found here
  • Want to read our summaries of academic finance papers? Check out our Academic Research Insight category

Does Empirical Finance Replicate? AI answers the question. was originally published at Alpha Architect. Please read the Alpha Architect disclosures at your convenience.

Source: Elisabetta Basilico, PhD, CFA · alphaarchitect.com