Reproduce and validate a result

Reproduction becomes useful when the result is small enough to specify exactly and the scientific claim is narrow enough to fail. This lesson uses one normalized heavy-scalar exchange channel to separate three questions: does the original procedure run, can the stated result be obtained again, and does a check with different failure modes support the physical interpretation?

Required background. Bring the bounded claim and source trace from Literature, sources, and claim tracing together with the assumptions, convention translation, evidence basis, and cutoff from the scope and conventions sheet. The heavy-field calculation comes from effective field theory and matching. If tolerances, precision, or provenance are unfamiliar, first use the numerical and reproducibility review.

Separate execution, reproduction, and validation

Terminology varies across communities, so state the operation rather than relying on a label. This page uses the following distinctions, consistent with the operational discussion in National Academies 2019, ch. 3:

Operation	What is held fixed or changed	Question answered	What it does not establish
Rerun an implementation	Same source, procedure, and inputs in the intended environment	Can this implementation execute again and recover its output?	Independence from its formulas, design, or defects
Reproduce a specified result	A new calculation follows the same mathematical specification, preferably with separately written analysis	Can the declared numbers be obtained without relying on the original execution path?	That the specification represents the intended physics
Validate the scientific claim	A limiting case, identity, scaling law, alternative formulation, or other check can fail for a different reason	Does evidence beyond output agreement support the interpretation in its stated domain?	A stronger claim outside the tested observable, assumptions, or regime

A clean-environment rerun is valuable evidence about instructions and dependencies. It is not, by itself, independent scientific evidence. Two implementations can agree because they share the same incorrect equation; two environments can disagree while the analytic claim remains sound.

Fix a bounded heavy-exchange claim

Use the same two-scalar model as in the preceding claim trace: a light real field $\phi$, a heavy real field $H$ of mass $M$, and interaction $\mathcal L_{\mathrm{int}}=-gH\phi^2/2$. Eliminating $H$ at tree level gives

$$\Delta\mathcal L_{\mathrm{tree}} = \frac{g^2}{8}\phi^2\frac{1}{\Box+M^2}\phi^2.$$

In one exchange channel, isolate the momentum-dependent factor

$$\mathcal A(q^2) = \frac{g^2}{M^2-q^2}.$$

Normalize away the low-energy coefficient $g^2/M^2$ and define

$$x=\frac{q^2}{M^2}, \qquad F(x) = \frac{\mathcal A(q^2)}{g^2/M^2} = \frac{1}{1-x}, \qquad |x|<1.$$

In four spacetime dimensions $[g]=1$ in mass units, so both $\mathcal A$ and $F$ are dimensionless; $x$ is a unit-independent expansion parameter.

The result to reproduce is deliberately limited:

For the real inputs $x=0.05,0.10,0.20$, compute $F(x)$, its leading approximation $F_{\mathrm{LO}}=1$, and its next approximation $F_{\mathrm{NLO}}=1+x$. Verify the exact relative errors and their scaling.

This normalized channel is the momentum-space image of the local expansion

$$\frac{1}{\Box+M^2} = \frac{1}{M^2} \left( 1-\frac{\Box}{M^2}+\cdots \right),$$

because $\Box\to-q^2$ on the exchanged momentum. It checks the sign, expansion parameter, and first two local terms in this channel. Manohar develops this integration and propagator expansion in Manohar 2018, §§3.5, 4.1–4.2, and 5.4, preprint pp. 17–20 and 37, PDF.

It does not prove the full decoupling theorem. It omits other exchange channels, field and vertex renormalization, loop matching, scheme dependence, operator mixing, and known situations in which naive decoupling assumptions fail. The general low-momentum theorem has additional hypotheses and a much broader argument Appelquist and Carazzone 1975, pp. 2856–2861.

Derive the exact residuals before computing

Define signed residuals by approximation minus exact result:

$$\begin{aligned} r_{\mathrm{LO}}(x) &= F_{\mathrm{LO}}-F = -\frac{x}{1-x}, \\ r_{\mathrm{NLO}}(x) &= F_{\mathrm{NLO}}-F = -\frac{x^2}{1-x}. \end{aligned}$$

Therefore the relative errors are

$$\epsilon_{\mathrm{LO}} = \frac{|r_{\mathrm{LO}}|}{|F|} =|x|, \qquad \epsilon_{\mathrm{NLO}} = \frac{|r_{\mathrm{NLO}}|}{|F|} =|x|^2.$$

The expected values are:

$x$	Exact $F(x)$	$F_{\mathrm{NLO}}$	$(\epsilon_{\mathrm{LO}},\epsilon_{\mathrm{NLO}})$
$0.05$	$1.052631578947$	$1.05$	$(5\%,0.25\%)$
$0.10$	$1.111111111111$	$1.10$	$(10\%,1\%)$
$0.20$	$1.25$	$1.20$	$(20\%,4\%)$

For these positive inputs, both approximations lie below the exact result and NLO is closer than LO. That sign check is capable of exposing a swapped denominator or a wrong derivative-operator sign.

Specify inputs, outputs, and tolerance exactly

Use this calculation specification before either route below.

Inputs. The ordered binary64 values are [0.05, 0.10, 0.20]. They represent the dimensionless ratio $q^2/M^2$ in a single real channel. The prescribed run contains no random input. The broader series condition is $|x|<1$; a program must not report the series as convergent at or beyond that boundary.

Outputs. For each input, return the columns x, exact, lo, nlo, residual_lo, residual_nlo, relative_lo, and relative_nlo in input order. Compute with unrounded values; rounding belongs only to presentation.

Tolerance. Require the echoed x value to have the input bit pattern. Compare exact, lo, nlo, both residuals, and both relative errors with the analytic values using an absolute tolerance of $5\times10^{-13}$. Also require the algebraic identity $|(1-x)F-1|\leq5\times10^{-15}$ for each row. These are deterministic binary64 tolerances for this well-conditioned range, not general EFT uncertainties.

Provenance. Record the mathematical specification, source revision or content hash, script hash, exact command, operating system and architecture, runtime and patch version, numeric type, input bytes and checksum, standard output and checksum, warnings, and every modification made after the first clean attempt. There are no dependencies beyond the language runtime in the short route below; if a notebook, dataframe library, or plotting package is added, record its exact version and execution order.

Use an analytic route or a short code route

Analytic route

Treat the inputs as the exact rationals $1/20$, $1/10$, and $1/5$. Evaluate $1/(1-x)$, $1$, and $1+x$ as rational numbers, then compute the signed and relative residuals before converting to decimals. This route has no floating- point tolerance; the equalities are exact. It is the reference against which the binary64 calculation is compared.

Short code route

The following standard-library-only Python is sufficient:

xs = (0.05, 0.10, 0.20)

for x in xs:
    exact = 1.0 / (1.0 - x)
    lo = 1.0
    nlo = 1.0 + x
    residual_lo = lo - exact
    residual_nlo = nlo - exact
    relative_lo = abs(residual_lo / exact)
    relative_nlo = abs(residual_nlo / exact)
    print(x, exact, lo, nlo, residual_lo, residual_nlo,
          relative_lo, relative_nlo)

Run it from an otherwise empty working directory with the recorded runtime. Redirect or copy the unrounded output only after the command and script hash have been captured. A spreadsheet or another language is equally suitable if it implements the same specification and records its calculation order.

Rerun, reproduce, then validate

Perform the three operations separately and preserve each outcome.

1. Rerun. Execute the recorded script with the recorded binary64 runtime, inputs, command, and clean working directory. Compare every output column with the predeclared values and tolerances. This tests the execution instructions and environment.
2. Reproduce. Recalculate the table by the exact rational route or by a separately written spreadsheet or program. Do not copy the original residual calculation. This tests whether the specified result survives a different execution path.
3. Validate. Derive the geometric-series remainder, test its predicted scaling, and connect the momentum expansion to the local operator. These checks can reject an implementation that reproduces a few rounded numbers for the wrong reason.

Report the evidence in bounded language:

Outcome	Conclusion supported
Clean rerun agrees	The recorded implementation and environment recover the declared output
Independent reproduction agrees	A separate calculation implements the same numerical specification
Remainder, scaling, and local-operator checks agree	The bounded one-channel expansion is supported in the tested range
All three agree	The one-channel result has reproducible output and independent checks; the omitted channels, loops, and theorem hypotheses remain untested

If the rerun fails but both analytic routes agree, report an environment or execution problem rather than a refutation of the series. If the outputs agree but the independent identity fails, report that the numbers were recovered without validating the stated calculation.

Use a check with different failure modes

For any nonnegative integer $N$ and $|x|<1$, the finite geometric series obeys

$$\frac{1}{1-x} = \sum_{n=0}^{N}x^n +\frac{x^{N+1}}{1-x}.$$

The relative remainder after order $N$ is therefore $|x|^{N+1}$. Reducing $x$ by a factor of two predicts

$$\frac{\epsilon_N(x)}{\epsilon_N(x/2)} = 2^{N+1}.$$

LO must improve by a factor of $2$ and NLO by a factor of $4$. The pairs $x=0.20,0.10$ in the table test both predictions without using the decimal value of $F$ as the reference. Also verify $(1-x)F=1$ and the negative signed residuals for positive $x$.

The operator check fails differently again. Solving the heavy-field equation and expanding $(\Box+M^2)^{-1}$ tests the sign and normalization of the local terms before evaluating any chosen $x$. It still shares the same tree-level model and one-channel reduction, which is why the stronger decoupling claim remains outside the evidence.

Triage a failure before changing anything

Preserve the first failed output, then locate the earliest layer that can explain it.

Symptom	First checks	Appropriate response
Recorded command does not start	Script hash, path, runtime, permissions, and working directory	Repair the execution description without changing the scientific input
Same script produces different rows	Input bytes, numeric type, locale, hidden state, and source revision	Isolate the execution difference and retain both outputs
Rerun agrees but independent route differs	Mathematical specification, sign convention, copied formula, and rounding order	Resolve the specification before interpreting physics
Table agrees but $(1-x)F=1$ fails internally	Output may be hard-coded, rounded too early, or computed from inconsistent columns	Reject the calculation and expose the dependency between columns
Identity passes but error scaling fails	Approximation order, residual definition, input ratio, or domain is wrong	Narrow the numerical claim and repair the convergence test
Every check passes only near $x=0$	The asymptotic claim may be sound but the requested range may be too wide	State the tested range; do not enlarge it without new evidence
A result is quoted for $\lvert x\rvert\geq1$	The geometric expansion is outside its convergence disk	Reject the EFT-series interpretation even if the rational function is finite

Do not widen a tolerance merely to accept the desired result. If a source or environment modification is required, retain the original failed attempt, state the change, and repeat every affected check.

Copyable reproduction bundle

This compact bundle is sufficient for the present deterministic calculation and can be extended for data or stochastic work.

source claim and exact locator:
bounded scientific claim:
explicit nonclaim:
scope, assumptions, and convention translation:

mathematical specification:
ordered inputs, units, and input checksum:
required output columns and order:
predeclared tolerances and justification:

source revision and script checksum:
runtime, operating system, architecture, and numeric type:
exact command and working-directory assumptions:
standard output, warnings, and output checksum:
changes after the first clean attempt:

independent reproduction route:
identity, remainder, and scaling checks:
shared assumptions between the two routes:

failure classification and retained evidence:
strongest conclusion supported:
next unanswered question:

For a stochastic calculation, add the random generator, stream construction, seed policy, raw samples, autocorrelation treatment, estimator, and covariance. For external data, add acquisition date, release identity, checksum, units, selection, calibration, and license.

Check your understanding

1. Recover both exact relative errors

Starting from $F=1/(1-x)$, derive the signed LO and NLO residuals using the definition approximation minus exact result. Then divide by $|F|$.

Solution

At LO,

$$r_{\mathrm{LO}} = 1-\frac{1}{1-x} = -\frac{x}{1-x}.$$

At NLO,

$$\begin{aligned} r_{\mathrm{NLO}} &= 1+x-\frac{1}{1-x} \\ &= \frac{(1+x)(1-x)-1}{1-x} = -\frac{x^2}{1-x}. \end{aligned}$$

Because $|F|=1/|1-x|$,

$$\epsilon_{\mathrm{LO}}=|x|, \qquad \epsilon_{\mathrm{NLO}}=|x|^2.$$

For positive $x<1$, both signed residuals are negative: the two truncated sums underestimate the exact rational function.

2. Measure the observed order without using the exact decimals

Use the relative errors at $x=0.20$ and $x=0.10$ to compute

$$p = \frac{\log[\epsilon(x)/\epsilon(x/2)]}{\log2}$$

for LO and NLO.

Solution

At LO, the errors are $0.20$ and $0.10$, so

$$p_{\mathrm{LO}} = \frac{\log(0.20/0.10)}{\log2} =1.$$

At NLO, the errors are $0.04$ and $0.01$, so

$$p_{\mathrm{NLO}} = \frac{\log(0.04/0.01)}{\log2} =2.$$

The refinement test recovers the predicted first- and second-order relative remainders. It tests scaling, not the broader heavy-field theorem.

3. Classify three successful calculations

Classify each result and state its evidence boundary:

1. the recorded script runs in a new empty environment and reproduces its old output;
2. a second person evaluates the exact rationals in a spreadsheet built from the mathematical specification; and
3. a derivation proves the finite-series remainder and its factor-of-two scaling.

Solution

The first is a rerun. It supports the sufficiency of the environment and execution instructions, but it shares the implementation and its defects.

The second is an independent reproduction of the specified result to the extent that the spreadsheet was written independently. It can expose defects in the original script, but both routes still share the mathematical specification.

The third is a validation check for the bounded series claim. It can expose a wrong approximation order or remainder even if selected table values were copied correctly. It does not test other exchange channels, loops, or the hypotheses of a general decoupling theorem.

4. Detect a wrong denominator sign

Suppose an implementation computes $F_{\mathrm{wrong}}=1/(1+x)$. At $x=0.10$, compare it with the expected value and test $(1-x)F=1$. Which checks fail?

Solution

The wrong result is

$$F_{\mathrm{wrong}}(0.10) = \frac{1}{1.10} \approx0.909090909091,$$

whereas the expected value is

$$F(0.10) = \frac{1}{0.90} \approx1.111111111111.$$

Their difference is about $0.2020$, far above the declared $5\times10^{-13}$ tolerance. The independent identity also gives

$$(1-0.10)F_{\mathrm{wrong}} = \frac{0.90}{1.10} \approx0.81818 \ne1.$$

The numeric comparison, algebraic identity, sign of the residual, and expected low-$x$ series all fail. Enlarging the tolerance would conceal a model error, not accommodate ordinary roundoff.

Continue to uncertainty and disagreement

Keep the source trace, scope sheet, first clean run, independent reproduction, and every failed check together. They distinguish an environment disagreement from a numerical mismatch or a challenge to the scientific claim.

Next, use Uncertainty, disagreement, and negative results to compare results whose inputs, approximations, or evidence do not align. For a larger computational target, the computational field theory pathway supplies convergence, correlated uncertainty, and provenance requirements.

References

• Thomas Appelquist and J. Carazzone, “Infrared Singularities and Massive Fields,” Physical Review D 11 (1975), 2856–2861, doi:10.1103/PhysRevD.11.2856.
• Aneesh V. Manohar, “Introduction to Effective Field Theories,” in Effective Field Theory in Particle Physics and Cosmology: Lecture Notes of the Les Houches Summer School, Volume 108, edited by Sacha Davidson, Paolo Gambino, Mikko Laine, Matthias Neubert, and Christophe Salomon (Oxford: Oxford University Press, 2020), 47–136, doi:10.1093/oso/9780198855743.003.0002; author manuscript, arXiv:1804.05863v1 (2018).
• National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science, Washington, DC: National Academies Press, 2019, doi:10.17226/25303.