calc_ftest

sherpa.utils.calc_ftest(dof1, stat1, dof2, stat2)[source] [edit on github]

Compare two models using the F test.

The F-test is a model comparison test; that is, it is a test used to select from two competing models which best describes a particular data set. A model comparison test statistic, T, is created from the best-fit statistics of each fit; as with all statistics, it is sampled from a probability distribution p(T). The test significance is defined as the integral of p(T) from the observed value of T to infinity. The significance quantifies the probability that one would select the more complex model when in fact the null hypothesis is correct. See also calc_mlr.

Parameters
  • dof1 (int or sequence of int) – degrees of freedom of the simple model

  • stat1 (number or sequence of number) – best-fit chi-square statistic value of the simple model

  • dof2 (int or sequence of int) – degrees of freedom of the complex model

  • stat2 (number or sequence of number) – best-fit chi-square statistic value of the complex model

Returns

sig – The significance, or p-value. A standard threshold for selecting the more complex model is significance < 0.05 (the ‘95% criterion’ of statistics).

Return type

number or ndarray

See also

calc_mlr, incbet

Notes

The F test uses the ratio of the reduced chi2, which follows the F-distribution, (stat1/dof1) / (stat2/dof2). The incomplete Beta function is used to calculate the integral of the tail of the F-distribution.

The F test should only be used when:

  • the simpler of the two models is nested within the other; that is, one can obtain the simpler model by setting the extra parameters of the more complex model (often to zero or one);

  • the extra parameters have values sampled from normal distributions under the null hypothesis (i.e., if one samples many datasets given the null hypothesis and fits these data with the more complex model, the distributions of values for the extra parameters must be Gaussian);

  • those normal distributions are not truncated by parameter space boundaries;

  • the best-fit statistics are sampled from the chi-square distribution.

See Protassov et al. 2002 1 for more discussion.

References

1

Protassov et al., Statistics, Handle with Care: Detecting Multiple Model Components with the Likelihood Ratio Test, Astrophysical Journal, vol 571, pages 545-559, 2002, http://adsabs.harvard.edu/abs/2002ApJ…571..545P

Examples

>>> calc_ftest(11, 16.3, 10, 10.2)
0.03452352914891555
>>> calc_ftest([11, 11], [16.3, 16.3], [10, 9], [10.2, 10.5])
array([0.03452353, 0.13819987])