Model Risk With Estimates of Probabilities of Default

Model Risk With Estimates of Probabilities of Default Dirk Tasche Imperial College, London

June 2015

The views expressed in the following material are the author’s and do not necessarily represent the views of the Global Association of Risk Professionals (GARP),

its Membership or its Management.

2

Outline Two forecasting problems A taxonomy of dataset shift Estimation under prior probability shift assumption Estimation under ’invariant density ratio’ assumption An application to the mitigation of model risk for PD estimation Concluding remarks References

Dirk Tasche (Imperial College)

Model Risk With Estimates of Probabilities of Default

2 / 18

Two forecasting problems

Single borrower’s probability of default (PD)

I

Moody’s corporate issuer and default counts in 20082 . Grade Issuers Defaults

Caa-C 417 63

B 1151 25

Ba 528 6

Baa 1021 5

A 966 5

Aa 582 4

Aaa 140 0

All 4805 108

I

January 1, 2009: What is a Baa-rated borrower’s probability to default in 2009?

I

Natural (?) estimate:

2

5 1021

≈ 0.49%.

Source: Moody’s (2015)



3 / 18


Rating profile known Moody’s corporate issuer proportions and default rates in 2008 and issuer proportions in 20093 . All numbers in %. Grade Caa-C B Ba Baa A Aa Aaa All

Issuers 8.7 24.0 11.0 21.2 20.1 12.1 2.9 100.0

2008 Default rate 15.1 2.2 1.1 0.5 0.5 0.7 0.0 2.2

Issuers 11.4 20.9 11.0 21.9 20.7 11.2 2.8 100.0

2009 Default rate ? ? ? ? ? ? ? ?

How to take account of the additional data? 3

Source: Moody’s (2015)



4 / 18


Some thoughts I

Compared to 2008, the rating profile in January 2009 has changed.

I

Why should the grade-level or total default rates remain the same? Invariant grade-level default rates would imply almost invariant discriminatory power:

I

I I

Observed accuracy ratio in 2008: 63.4% Forecast accuracy ratio for 2009: 66.6%

I

What other ways are there to reflect ’almost invariant’ discriminatory power?

I

Assuming an invariant accuracy ratio is not sufficient for inferring PDs if only the rating profile is known.



5 / 18


Alternatives to ’invariant grade-level default rates’ I

Geometric interpretations of accuracy ratio (for continuous score): I I

Based on area under Receiver Operating Characteristic (ROC) Based on area between Cumulative Accuracy Profile (CAP) and diagonal

I

Derivative of CAP is (essentially) the PD curve of the scores.

I

Derivative of ROC is (essentially) the density ratio of the scores.

I

Is ’invariant density ratio’ a viable alternative?

I

We also look at ’invariant rating profiles of defaulters and non-defaulters’ as an alternative assumption on ’almost invariant’ discriminatory power.



6 / 18

A taxonomy of dataset shift

The Machine Learning perspective I

Classification on datasets with changed distributions is a problem well-known in Machine Learning.

I

Moreno-Torres et al. (2012) proposed a taxonomy for dataset shifts. Setting:

I

I I

I

I

Each item in a dataset has a class y and a covariates vector x. ptest (x, y ) and ptrai (x, y ) are the joint distributions of (x, y) on the test and training sets respectively. ptrai (x, y ) is known from observation but for ptest (x, y) only the marginal distribution ptest (x) is observable now. How to determine unconditional class probabilities ptest (y = c) and conditional class probabilities ptest (y = c | x)?

I

Definition: Dataset shift occurs if ptest (x, y) 6= ptrai (x, y).

I

On Slide 4, y = default status, x = rating grade.



7 / 18

A taxonomy of dataset shift

The Moreno-Torres et al. taxonomy I

Four types of dataset shift ptest (x, y) 6= ptrai (x, y ): I

Covariate shift: ptest (x) 6= ptrai (x), but ptest (y | x) = ptrai (y | x).

I

Prior probability shift: ptest (y) 6= ptrai (y ), but ptest (x | y ) = ptrai (x | y ).

I

Concept shift: ptest (x) = ptrai (x), but ptest (y | x) 6= ptrai (y | x),

or

ptest (y ) = ptrai (y), but ptest (x | y ) 6= ptrai (x | y). I

I

Other shifts.

Assuming ’invariant grade-level default rates’ on Slide 4 is equivalent to an assumption of covariate shift.



8 / 18

Estimation under prior probability shift assumption

Moody’s corporate rating profiles 2008

0.3 0.0

0.1

0.2

Frequency

0.4

0.5

Defaulters All Non−defaulters

Caa−C Dirk Tasche (Imperial College)

B

Ba

Baa

A


Aa

Aaa 9 / 18


The least-squares estimator I

Setting as on Slide 4: I

I I

I

y = default status (classes D default and N non-default), x = rating grade ptest (x) known, but ptest (x) 6= ptrai (x) Want to determine ptest = ptest (y = D) and ptest (y = D | x)

Prior probability shift assumption: ptest (x | y = c) = ptrai (x | y = c) for c = D, N.

I

Hence, for all x, the class probability ptest should satisfy ptest (x) = ptest ptrai (x | y = D) + (1 − ptest ) ptrai (x | y = N).

I

This is unlikely to be achievable. Therefore least squares approximation R ptest (x)−ptrai (x | y=N) ptrai (x | y=D)−ptrai (x | y=N) dx btest = . p 2 R ptrai (x | y=D)−ptrai (x | y=N)



(1)

dx

10 / 18


Fitted and observed Moody’s corporate rating profiles

0.10 0.00

0.05

Frequency

0.15

0.20

Fitted profile 2009 Observed profile 2009


B

Ba

Baa

A


Aa

Aaa 11 / 18

Estimation under ’invariant density ratio’ assumption

The ’invariant density ratio’ estimator I

Setting as on Slide 4: I

I I

I

y = default status (classes D default and N non-default), x = rating grade ptest (x) known, but ptest (x) 6= ptrai (x) Want to determine ptest = ptest (y = D) and ptest (y = D | x)

Invariant density ratio assumption: ptest (x | y = D) p (x | y = D) def = λ(x). = trai ptest (x | y = N) ptrai (x | y = N)

I

I

Then the class probability ptest must satisfy Z λ(x)−1 0= 1+(λ(x)−1) ptest d ptest (x).

(2a)

There is a unique solution to (2a) if and only if Z Z λ(x) d ptest (x) > 1 and λ(x)−1 d ptest (x) > 1.

(2b)



12 / 18


Properties I

I

If condition (2b) is not satisfied then the profiles ptest (x) and ptrai (x) are so different that any ’inheritance’ of discriminatory power seems questionable. etest come Exact fit: With the ’invariant density ratio’ estimate p etest (x | y = c), c = D, N, estimates of the conditional densities p such that λ(x) =

I

e ptest (x | y=D) , e ptest (x | y=N)

and

etest p etest (x | y = D) + (1 − p etest ) p etest (x | y = N). ptest (x) = p R Call ptest = ptrai (y = D | x) d ptest (x) the ’covariate shift’ estimator of ptest . Then it holds that etest . ptest = (1 − π) ptrai (y = D) + π p

(3)

Where 0 ≤ π ≤ 1 and π is the closer to 1 the more discriminatory power the scores x have on the training set. Dirk Tasche (Imperial College)


13 / 18


Conditional profiles 2009: Invariant from 2008 vs. fitted

0.2

0.4

Observed 2008 Fitted 2009 Observed 2009

0.0

Frequency

0.6

Defaulters

Caa−C

B

Ba

Baa

A

Aa

Aaa

0.10

Observed 2008 Fitted 2009 Observed 2009

0.00

Frequency

0.20

Non−defaulters


B

Ba

Baa

A

Aa


Aaa 14 / 18

An application to the mitigation of model risk for PD estimation

Different forecast methods I

Three methods of forecasting portfolio-wide default rate ptest : I I I

I

I

Covariate shift estimator ptest (Slide 13) btest , (1) Prior probability shift estimator p etest , (2a) Invariant density ratio estimator p

btest and p etest give the same estimate under a prior probability p shift. btest and p etest of ptest provide estimates of the Estimates p conditional default rates ptest (y = D | x) by ptest (y = D | x) =

I

ptest λ(x) . 1 + (λ(x) − 1) ptest

etest ) could be an upper bound for (3) suggests that max(ptest , p next year’s portfolio-wide default rate.



15 / 18

An application to the mitigation of model risk for PD estimation

Observed vs. forecast corporate default rates4

4 2

Per cent

6

8

Observed Covariate shift forecast Invariant Density Ratio forecast

1990

1995

2000

2005

2010

2015

Year 4

Source of observed rates: Moody’s (2015).



16 / 18

Concluding remarks

I

Straightforward ’covariate shift’ (or ’invariant conditional default rate’) PD estimates sometimes may seriously underestimate future default rates.

I

The ’invariant density ratio’ approach often provides very different estimates that may be used for model risk mitigation.

I

The ’invariant density ratio’ approach can also be applied for the estimation of loss rates.

I

Rating agency data like Moody’s (2015) possibly are ’subjective’, making results of the approach conservative.

I

Further reading: I I

Background and more details: Tasche (2013), Tasche (2014) More sophisticated approaches: Hofer and Krempl (2013), Hofer (2015)



17 / 18

References

V. Hofer. Adapting a classification rule to local and global shift when only unlabelled data are available. European Journal of Operational Research, 243(1):177–189, 2015. V. Hofer and G. Krempl. Drift mining in data: A framework for addressing drift in classification. Computational Statistics & Data Analysis, 57(1):377–391, 2013. Moody’s. Annual Default Study: Corporate Default and Recovery Rates, 1920-2014. Special comment, Moody’s Investors Service, March 2015. J.G. Moreno-Torres, T. Raeder, R. Alaiz-Rodriguez, N.V. Chawla, and F. Herrera. A unifying view on dataset shift in classification. Pattern Recognition, 45(1):521–530, 2012. D. Tasche. The art of probability-of-default curve calibration. Journal of Credit Risk, 9(4):63–103, 2013. D. Tasche. Exact fit of simple finite mixture models. Journal of Risk and Financial Management, 7(4):150–164, 2014. Dirk Tasche (Imperial College)


18 / 18

Model Risk With Estimates of Probabilities of Default

Recommend Documents