As we move into a new year, you may still be working on a few of those items you didn’t quite get to in 2019. And for a lot of our clients, one of those items is often backtesting. Given all the confusion surrounding backtesting, it can be pretty easy to keep pushing it to the bottom of the “to-do” list.

So, we thought it might be a good idea to dust off a blog I wrote back in 2015 to jump start your 2020 so you can get one more thing crossed off your list. In that blog, we looked at a few of the most common questions we get on backtesting. Specifically, we discussed who should do it; how often it should be done; what period should be covered; and if you need to backtest just model results, or also key model assumptions.

Before we dive into each of these questions, let’s first examine why you should even be doing backtesting. For those of you who have heard me speak on the subject in the past, you know that I’m not a big believer in backtesting in periods when market rates aren’t materially changing. I argue that the goal of backtesting is to gauge model performance in times of changing market rates, and thereby identify possible areas to adjust the model (change assumptions, inputs, scenarios, etc.) to improve its predictability.

Backtesting in periods where market rates haven’t materially changed represents really just a budget/forecast analysis comparing actual results to what the model projected for the base/no change scenario. In such a "no-change" rate environment, NIM differences (between actual performance and the base case) generally only result from changes in volumes (growth, runoff, shift in balance sheet composition, etc.), not rate offerings. If they are due to changes in rate offerings, those still would not be attributed to changes in market rates (since there was no change in market rates).

As such, the practice of backtesting the IRR model in periods where market rates did not materially change does not bear merit. With that said, we’ve seen some pretty significant changes in market rates in the past several years, so backtesting is certainly now a regulatory expectation for virtually all financial intuitions. 

So, let’s get back to that list of backtesting questions; one by painful one…

Who Should do the Backtesting?

The Regulatory guidance does not specify if backtesting should be done by institution management or a third-party. It only states that it should be done as part of the model validation process. I would argue that it could be done by either management or a third party, but regardless, the backtesting procedures/results should at least be reviewed by and addressed in the annual independent review. That is, the independent review should certify that the scope and appropriateness of the backtesting program is sufficient.

How Often Should Backtesting be Done?

Again, the Regulatory guidance does not specifically address backtesting frequency, but I would argue that annual backtesting should be sufficient. With that said, there is no reason that backtesting has to be done for a calendar year; it can be done as of any quarter end, or even month end. 

What Period Should Backtesting Cover?

The period subject to backtesting should be of sufficient length to encompass any potential "lags" in actual changes in rates after the subject change in market rates occurred. To be sure such "lags" are accounted for, generally, the backtesting exercise should cover a period of 12 months, as a shorter period, say three months, would likely not be long enough to fully account for any "lags."

Do You Need to Backtest Model Results and Assumptions?

Ideally, yes. In fact, the FFIEC’s 2010 Advisory on Interest Rate Risk specifically states that the model validation process should include "the backtesting of assumptions and results." While the Advisory does not state which assumptions should be covered, it would seem that the most obvious assumptions to backtest include those that generally have the most significant impact on IRR modeling results. That is, asset prepayments, deposit repricing, and possibly even non-maturity deposit decay rates.

Well, I believe that answers all the questions. However, if you have more, or would like to explore how our Advisory Department can help you with backtesting, give me a call at (800) 323-3281 or shoot me an email.

And if you'd like to learn more about how our other budgeting, risk management, CECL, and strategy offerings can help your organization succeed in the new year, click here to schedule a discovery call.