So if you’re reading this, my second ever blog post, you’ve probably already seen the first one entitled "Independent Review, Model Validation, and Backtesting: Same Thing, Only Different." In that piece, we looked at the interrelationship of these three items and brought up a few questions on backtesting. Specifically, we questioned who should do it, how often should it be done, what period should be covered, do you need to backtest model results and assumptions, and why even bother if market rates really aren’t changing.

Let’s take that last question first. For those of you that have heard me speak on the subject in the past, you know that I’m not a big believer in backtesting in periods when market rates aren’t materially changing. I argue that the goal of backtesting is to gauge model performance in times of changing market rates and thereby identify possible areas to adjust the model (change assumptions, inputs, scenarios, etc.) to improve its predictability. Backtesting in periods where market rates haven’t materially changed represents really just a budget/forecast analysis comparing actual results to what the model projected for the base/no change scenario. In such a "no-change" rate environment, NIM differences (between actual performance and the base case) generally only result from changes in volumes (growth, runoff, shift in balance sheet composition, etc.), not rate offerings. If they are due to changes in rate offerings, those still would not be attributed to changes in market rates (since there was no change in market rates). As such, the practice of backtesting the IRR model in periods where market rates did not materially change does not bear merit.

While I may have you convinced at this point, unfortunately my former employer (the FDIC) and their fellow Regulatory bodies have been taking a different stance lately. Perhaps it’s due to the fluctuations in long-term rates in recent years, maybe it’s a result of an increased emphasis on the entire IRR management process, or it might be because they continue to find that financial institutions all too frequently still don’t have well documented, institution-specific model assumptions. Regardless, it appears that backtesting is back on the front burner again (how do you like that play on words?).

So, let’s get back to the rest of that list of questions; one by painful one…

Who Should do the Backtesting?

The Regulatory guidance does not specify if backtesting should be done by institution management or a third-party. It only states that it should be done as part of the model validation process. I would argue that it could be done by either management or a third party, but regardless, the backtesting procedures/results should at least be reviewed by and addressed in the annual independent review. That is, the independent review should certify that the scope and appropriateness of the backtesting program is sufficient.

How Often Should Backtesting be Done?

Again, the Regulatory guidance does not specifically address backtesting frequency, but I would argue that annual backtesting should be sufficient, particularly in times when market rates haven’t experienced a material change.

What Period Should Backtesting Cover?

The period subject to backtesting should be of sufficient length to encompass any potential "lags" in actual changes in rates after the subject change in market rates occurred. To be sure such "lags" are accounted for, generally, the backtesting exercise should cover a period of 12 months, as a shorter period, say 3-months, would likely not long enough to fully account for any "lags."

Do You Need to Backtest Model Results and Assumptions?

Ideally, yes. In fact, the FFIEC’s 2010 Advisory on Interest Rate Risk specifically states that the model validation process should include "the backtesting of assumptions and results." While the Advisory does not state which assumptions should be covered, it would seem that the most obvious assumptions to backtest include those that generally have the most significant impact on IRR modeling results. That is, asset prepayments, deposit repricing, and possibly even non-maturity deposit decay rates.

Well, I believe that answers all the questions. However, if you have more, or would like to explore how our Advisory Department can help you with backtesting, give me a call or shoot me an email.

Want more? Review Dave's webinar where he discusses these questions in more detail.