In our ever increasing efforts to educate and inform, our marketing department here at team Plansmith has been on me to contribute to our Blog. Quite frankly, I’m not really a "blog" guy, but for those of you that know me, I’m not short on opinions either. So, given that I sit here stuck on a plane for a few hours, this seems like a good time to take a shot at it.
In my more than 20 years at the FDIC and now three years here at Plansmith, I can’t count the number of times I’ve heard the terms "Independent Review," "Model Validation," and "Backtesting" misused. Many times, they’re even used synonymously, but are they the same? If so, I guess I could end this blog post now and dig out the Skymall Catalog tucked neatly in the seat back pouch in front of me. But, of course, they’re not the same, and so my shopping for self-cleaning litter boxes and speakers that look like garden gnomes will have to wait.
First, let’s examine why we even care about this subject in the first place. According to the 1996 Joint Agency Policy Statement on IRR, "Effective control of the interest rate risk management process includes independent review…", and according to the Policy Statement, one of the five areas of the review should be "The validity of the risk measurement calculations" (i.e., model validation), which is "often tested by comparing actual versus forecasted results" (i.e., backtesting). Essentially, Independent Review is an audit function, the scope of which should cover five separate areas (all five areas are detailed in the Policy Statement). One of those five areas is Model Validation, and one method of Model Validation is backtesting (as an aside, the other common method referenced in the policy statement is benchmarking). So, for those of you that like pictures more than words, the simple diagram below illustrates this point. That is, Backtesting is part of Model Validation, and Model Validation is part of Independent Review.
Now that we have that straight, you may be asking yourself "what does this all mean to me, and does my compliance program meet the Regulatory standard?" Well, maybe yes; maybe no. What I can tell you is that we rarely see financial institutions get away with just giving examiners a copy of the third-party model validation that we give you for free. That document is only a small part of the "Validation" picture above. It simply affirms that the model’s math, formulas, code, etc. are valid and the model functions properly. It, by no means, is a full Independent Review as required by the Policy Statement. It also does not include the backtesting of your interest rate risk modeling results or assumptions. At a minimum, you must ensure that your interest rate risk management process includes a full scope Independent Review addressing the following five areas as fully detailed in the Policy Statement:
- Adequacy of the internal control system
- Appropriateness of risk measurement system
- Accuracy and completeness of data inputs
- Reasonableness and validity of scenarios
- Validation of risk measurement calculations
And, don’t forget the Backtesting…or maybe we should forget backtesting for now since that opens a whole other set of questions such as who should do it, how often should it be done, what period should be covered, do you need to backtest model results and assumptions, and why even bother if market rates really aren't changing. Hmmm, all good questions. Looks like I’ll have to write another blog to follow this one up. But for now…I need to ask the flight attendant for a copy of the Skymall; mine seems to be missing. In the meantime, give me a call or shoot me an email if you’re interested in any of our Advisory Services, including Independent Review for Plansmith and non-Plansmith model users, as well as backtesting.