Backtesting can be a painful topic for bankers. In this post, I'll answer the top 5 most common questions I hear about backtesting. I'll reference my first post, Independent Review, Model Validation, and Backtesting: Same Thing, Only Different, so you might want to revisit it before reading on. In that blog, we looked at the interrelationship of these three items and brought up a few questions on backtesting.
Specifically, we questioned 5 things: who should do it, how often should it be done, what period should be covered, do you need to backtest model results and assumptions, and why even bother if market rates really aren’t changing.
5 Most Painful Questions about Backtesting
- Why even bother if market rates haven’t materially changed?
Let’s take that last question first. For those of you that have heard me speak on the subject in the past, you know that I wasn’t a big believer in backtesting in periods when market didn’t materially change since such a test Such tests were really just a budget/forecast analysis comparing actual results to what the model projected for the base/no change scenario. In such "no-change" rate environments, NIM differences (between actual performance and the base case) generally only resulted from changes in volumes (growth, runoff, shift in balance sheet composition, etc.), not rate offerings.
While that argument may have worked in the past, it likely won’t work anymore now that rates have started, and are expected to continue to move. More significantly, my former employer (the FDIC) and their fellow Regulatory agencies have a renewed interest in backtesting and are routinely asking for it at exams.
- Who should do the backtesting?
The Regulatory guidance does not specify if backtesting should be done by institution management or a third-party. It only states that it should be done as part of the model validation process. I would argue that it could be done by either management or a third party, but regardless, the backtesting procedures/results should at least be reviewed by, and addressed in the annual independent review. That is, the independent review should certify that the scope and appropriateness of the backtesting program is sufficient.
- How often should backtesting be done?
Again, the Regulatory guidance does not specifically address backtesting frequency, but I would argue that annual backtesting should be sufficient, particularly in times when market rates haven’t experienced a material change.
- What period should backtesting cover?
The period subject to backtesting should be of sufficient length to encompass any potential "lags" in actual changes in rates after the subject change in market rates occurred. To be sure such "lags" are accounted for, generally, the backtesting exercise should cover a period of 12 months, as a shorter period, say 3 months, would likely not be long enough to fully account for any "lags."
- Do you need to backtest model results and assumptions?
Ideally, yes. In fact, the FFIEC’s 2010 Advisory on Interest Rate Risk specifically states that the model validation process should include "the backtesting of assumptions and results." While the Advisory does not state which assumptions should be covered, it would seem that the most obvious assumptions to backtest include those that generally have the most significant impact on IRR modeling results. That is, asset prepayments, deposit repricing, loan pricing, and possibly even non-maturity deposit decay rates.
Well, I believe that covers it. If you have questions or want to explore how our Advisory Department can help you with backtesting, click here to book time with me.
