An alternative test has been suggested by . Research has shown that the Binomial test, or the associated confidence interval, can have large variation, depending on \(n\) and \(\pi_0\). An alternative is Jeffrey’s prior test, which is a Bayesian test.

It is similar to the Binomial test in that it also assumes a Binomial distribution for the independent defaults, compares forecast PDs with observed defaults. The null hypothesis is that the PD estimate for a rating grade is greater than the true one (one-sided hypothesis test), i.e. \(H_0: \ \pi \leq \pi_0\). The alternative hypothesis is that the PD is greater than \(\pi_0\).

However, the distribution of defaults is modeled from a Bayesian perspective. It assumes that the true PD is not a fixed parameter but rather a random variable with a prior distribution, which has to be specified. If the prior is a Beta distribution, then the posterior distribution of defaults is also a Beta distribution because the Beta distribution is the conjugate prior for the Binomial distribution.

Let the number of defaults be binomially distributed, i.e., \(D\sim Bin(n,\pi)\) and let the true PD \(\pi\) have prior \(Beta(\alpha, \beta)\). Then the posterior of \(\pi\) is \(Beta(d+\alpha, n-d+\beta)\) for a given default number \(d\). A special prior Beta distribution, is the Jeffrey’s prior \(Beta(0.5, 0.5)\). Hence, the posterior of \(\pi\) is \(Beta(d+0.5, n-d+0.5)\). An analogy to the p-value is computed as the value of the CDF of this posterior Beta-distribution, evaluated at the predicted PD \(\pi_0\). As before, high default rates yield rejection of \(H_0\).

Let’s have a look at the above example with \(n=6\), \(d=2\) and a default rate \(d/n=0.3333\). We plot \(\pi_0=0.0367\), the observed default rate, Jeffrey’s prior and the posterior distribution. We also compute the value of the CDF of this posterior distribution which yields 0.0044. Hence, a predicted PD of 0.0367 is very unlikely given the observed number of defaults.

Note the difference in interpretation: in the Binomial test, we conclude that the occurrence of a number of defaults or less is likely or unlikely, given the PD prediction is correct. In the Jeffrey’s test, we conclude that a true PD being equal or lower than the predicted PD is unlikely, given the observed number of defaults. This is due to the Bayesian perspective of the latter.

n = len(PD_pred)
pi_0 = np.mean(PD_pred)
d= 2
defaultrate = d/n
x = np.linspace(0, 1, 50)
prior = beta.pdf(x, 0.5, 0.5)
posterior = beta.pdf(x, d+0.5, n-d+0.5)

print("Jeffrey's Prior Bayesian 'p-value':", beta.cdf(pi_0, d+0.5, n-d+0.5).round(decimals=4))

plt.plot([pi_0, pi_0], [0, 2.5], label='PD under H0')
plt.plot([d/n, d/n], [0, 2.5], color='red', linestyle='dashed', label='Observed rate')
plt.plot(x, prior, color='green', linestyle='dotted', label='Prior' )
plt.plot(x, posterior, color = 'black', linestyle='dashdot', label='Posterior')
plt.ylabel('PDF')
plt.xlabel('Default rate')
plt.legend(loc="best")
plt.show()

Jeffrey's Prior Bayesian 'p-value': 0.0044