Emerging challenges for IRB PD calibration

Banks in the UK which have been involved in the wave of mortgage Internal Ratings Based (IRB) model applications to the regulator will be familiar with the depth of internal and external scrutiny relating to probability of default (PD) calibration. They will also be aware of the growing focus on the development of other rating systems across non-retail portfolios (corporates, banks, project finance, etc.) and non-mortgage retail portfolios (personal loans, motor finance, etc.).

The purpose of this article is to explore, and offer suggestions for, two areas where challenge for PD calibration of non-retail portfolios is anticipated, both from internal validation teams and from regulatory review:

• Establishing the calibration period; and
• Setting the level at which the calibration is performed. 

While some of the considerations are also applicable to unsecured retail portfolios, such portfolios are subject to specific considerations, including the current widespread application of the ‘point-in-time plus buffer’ approach that will be addressed in a future article. 

In the context of the IRB approach, PD calibration is the process of transforming the output of models that rank order customers by likelihood of default into estimates that can be used for determining risk weights and capital requirements for a bank’s credit exposures. It is common for these estimates to be used also in non-regulatory settings such as pricing and credit decisioning.

Table 1: Illustrative example of PD calibration results

While both the rank ordering model development and PD calibration must be aligned with applicable requirements, those for the latter allow less flexibility in terms of methods, processes, and selection of the development time-period. The existing requirements[1] have been aggregated and refined as part of the PRA’s draft proposals for Basel 3.1 implementation as set out in Consultation Paper (CP) 16/22.

There are many factors to consider when undertaking long-run PD calibration for non-retail portfolios. Two requiring particular attention are:

• Setting the calibration period: The historical time period over which the calibration target (average observed default rates) is determined; and
• The level of calibration: The level at which the calibration target is determined (grade level average observed default rates versus calibration segment level average observed default rates). 

These two aspects are explored below, with some of the key considerations described and suggestions offered.

As specified in the PRA’s Basel 3.1 proposals, IRB PDs should be calibrated to a target which represents the ‘long run average of one-year default rates over a representative mix of good and bad economic periods’. Although mortgage portfolios are required to include the 1990s, regulations do not set out explicit time periods for other portfolios. However, the following aspects must be considered when setting the period:

• Variability of observed default rates;
• The relative frequency of ‘good’ and ‘bad’ years as reflected by relevant economic indicators; and
• Significant economic, legal, and business environment changes across time. 

In practice, banks generally identify the 2008-2010 recessionary years as the ‘bad’ periods, which are then extended to the most recent period available at the time of model development. They have also taken various approaches with response to exclusion of the periods impacted by Covid.

While banks may provide justification for this ‘all date’ approach in various quantitative and qualitative terms, such an approach is susceptible to challenge, primarily because it necessarily links the calibration period selected with the point at which the model was developed. To illustrate, if data was available until 2022, calibration with Covid exclusions would be performed on the 2008-2019 period. However, if data was only available until 2017, then calibration would have been performed based on the 2008-2017 period. As the period extends into more recent periods, more benign economic years are contained within the calibration, without justifying that this results in a more representative mix. Overall, this seems to contradict the PRA’s Basel 3.1 proposal that it ‘does not expect firms to automatically update an estimate to incorporate the experience of additional years, as these periods may not be representative’.

As well as avoiding the automatic incorporation of the most recent data points available into the calibration period:

• Firms should perform detailed assessments on various macroeconomic indicators and observed default rates. This may uncover macroeconomic drivers beyond the usual suspect, GDP, that can provide insights into the relationship between the economy and observed default rates. An example may be energy prices for project finance exposures. Ultimately, while IRB regulations refer to macroeconomic indicators when setting the representative mix, if the period selected does not contain relatively high default rates that have been observed internally, then it will be difficult to argue that it is appropriate; and
• Model developers should consider assessing the sensitivity of the calibrated PDs to different economic periods. Observed default rates across the economic period can be averaged to set the calibration target and it is a relatively simple exercise to assess how this target changes with the period selected (2008-2014 versus 2008-2015 etc.). Greater sensitivity of the default target to the time window indicates greater need for analysis to justify the period adopted. 

Having specified the time-period for calibration, the level at which the calibration is performed must be determined. Two approaches are permitted to derive the grade-level PD estimates required for use in regulatory capital calculations:

• Grade level calibration: This targets PDs such that the average PD within each grade is equal to the average observed default rate over the calibration period in each grade; and
• Segment level calibration: This targets PDs such that the average PD across the segment is equal to the segment’s observed default rate over the calibration period[2]. Grade level PDs can then be inferred. Segmentation of the portfolio is not always necessary; in which case calibration targets the portfolio level average observed default rate.

A common challenge across all IRB PD models is the ability to assign grades for accounts historically as a result of limited availability of historical data. For instance, if a rank ordering model is developed using data from 2020 to 2023, it is possible that the bank does not hold data for all risk drivers featuring in the model over the 2008 to 2012 period and accounts for this period cannot be linked to each grade.

A reliance on segment level calibration is therefore to be expected. The following simplified example shows a potential limitation of this approach: 

Infallible analysis has determined 2008-2016 as the calibration period, but the rank ordering model which assigns accounts to grades can only be applied from 2018 onwards i.e., after the calibration period. With only four grades shown for simplicity, annual observed default rates may be as follows:

Table 2: Illustrative observed default rates over time

A calibration target of 1.6% is calculated as the average of segment observed default rates over the period. This observed default rate target could be achieved through various PD assignments to accounts. One example (that has been calculated with simulated data and performing an intercept shift in the log-odds space) may be:

Table 3: Illustrative PD estimates following segment level calibration

While the PD target is met at segment level, there is no guarantee that the inferred grade PDs correspond to actual default rates across grades. For instance, in the simulated data, grade 4 experiences an average 23% observed default rate. Such a grade may represent watchlist accounts in a non-retail setting.

While this example is oversimplified, it is intended to show the importance of considering grade level default rates when undertaking segment level calibration, and the potential for segment level calibration methods to fail to reflect true grade level default rates.

While in such cases data is not available to assign grades through the calibration period, to help mitigate the risk of misassignment of grade-level PDs, the following steps could be considered. For example:

• Indicators such as those relating to arrears history and watchlist assignment are likely to have longer data histories than other rank ordering model drivers; overrides resulting from application of watchlist criteria should be applied prior to performing calibration in any case. These indicators could be used to identify groups of accounts with particularly elevated risk and thereby assess the suitability of PDs assigned; and
• Alternative data could be considered to proxy the rank ordering score and assignment to grades. For instance, if a non-retail rank ordering model relies on a score derived on several financial metrics, a potential relationship could be derived between the score and the subset of financial metrics for which there is greater historical availability. This would allow another dimension of calibration testing at grade level despite the overall segment level approach. 

As banks work through the next waves of IRB submissions, two of the areas requiring special attention in relation to the challenges around long-run PD calibration are the specification of the calibration time-period and the level at which calibration is performed. Finalisation of the PRA’s Basel 3.1 proposals, expected in Q2 2024, should provide further clarification, particularly on whether the challenges will apply across all portfolios, including those currently modelled on a point-in-time basis.

KPMG in the UK has deep expertise across credit risk model development, validation, and internal audit. Our credit risk specialists can support you on a wide range of related activities from large-scale IRB model development programmes to advice on Basel 3.1 regulatory interpretation. Please contact us to discuss your requirements.

Footnotes:

[1] UK Capital Requirements Regulation (CRR), PRA Supervisory Statement (SS) 11/13, and EBA guidelines on PD and LGD estimation.

[2] Under the PRA’s Basel 3.1 proposals, PD models should always assign customers grades, rather than a continuous PD estimate. Notwithstanding this, the overall calibration target may not be set at grade level, instead using the less granular segment level. ‘Segment’ may commonly be referred to as the ‘portfolio’ view in model developments, but the former term has been used for consistency with regulatory texts.