Market Risk VaR Backtesting
VaR backtesting compares a forecast made before a period with the subsequent loss on a consistent basis. Count exceptions, inspect their timing and investigate their causes. An expected exception count is not a pass/fail cutoff.
The method at a glance
Illustrative comparison · exceptions · bars start at zero
- 95% confidence12.50
- 97.5% confidence6.25
- 99% confidence2.50
250 × (1 - confidence). These are model expectations, not pass/fail thresholds or observed counts.
| Concept | How it works | Watch for |
|---|---|---|
| Frequency | Expected exceptions = observations × (1 - confidence). | Demanding exactly the expected number of exceptions. |
| Independence | Look for clustering as well as the total count. | Treating clustered losses as harmless because the annual count looks acceptable. |
| P&L basis | Align the horizon, positions and actual/hypothetical P&L definition. | Comparing a static-portfolio forecast with unrelated intraday trading effects. |
01Frequency
How it works: Expected exceptions = observations × (1 - confidence).
Watch for: Demanding exactly the expected number of exceptions.
02Independence
How it works: Look for clustering as well as the total count.
Watch for: Treating clustered losses as harmless because the annual count looks acceptable.
03P&L basis
How it works: Align the horizon, positions and actual/hypothetical P&L definition.
Watch for: Comparing a static-portfolio forecast with unrelated intraday trading effects.
Original worked example
What does 2.5 exceptions mean?
Backtest 99% one-day VaR over 250 observations. Assume correctly calibrated, independent exceptions for the benchmark calculation.
- 1
Exception probability = 1 - 0.99 = 0.01.
- 2
Expected count = 250 × 0.01 = 2.5.
- 3
A realized count must be an integer; six exceptions require investigation and a defined statistical test.
2.5 is an expectation, not a permissible fractional count or automatic acceptance threshold.
Illustrative assumptions and calculations for learning. Not observed market data or an official exam item.
Apply the method and check its limits
Before you start: One-day VaR, binomial probabilities and actual versus hypothetical P&L.
Compare a count with a sampling distribution
X ~ Binomial(n, p), with p = 1 - confidence, under the independent calibrated benchmark
At 99% confidence over 250 independent observations, the expected count is 2.5. The probability of zero exceptions is 0.99²⁵⁰, about 8.11%. A zero count is therefore possible even under the benchmark. A formal test specifies its statistic and significance level; an expectation alone is not an acceptance region.
Assess independence and P&L alignment separately
Frequency asks whether the unconditional exception rate fits the target. Independence asks whether exceptions cluster through time. A model can satisfy one and fail the other. Reconcile the portfolio, forecast horizon and loss series before interpreting either result. Regulatory backtesting conventions also specify the P&L basis and should not be replaced with a homemade threshold.
A count can miss a regime problem
Three exceptions in consecutive days can be more informative about time dependence than three widely separated exceptions. Investigate whether volatility rose faster than the model adapted, whether factors were missing, or whether the loss series was inconsistent with the forecast definition. Document the diagnosis instead of tuning parameters solely to erase exceptions.
Work through the topic
Construct a consistent VaR backtest, evaluate exception frequency and timing, and distinguish evidence of a problem from a complete diagnosis of its cause.
Align the forecast with the outcome it was meant to predict
A backtest compares a risk forecast made before an outcome with the subsequent outcome under a defined measurement convention. Record the forecast timestamp, confidence level, horizon, position set and profit-and-loss definition. If a one-day VaR is compared with a two-day loss, or a fixed-position forecast with a materially changed trading book, the exception count may reflect a mismatch rather than model accuracy.
Under a positive-loss convention, an exception occurs when the observed loss exceeds the corresponding VaR threshold. If VaR is 1 million and the loss is 1.2 million, the indicator is one. If the position gains 0.3 million, its loss is -0.3 million and the indicator is zero. Keep the sign convention consistent throughout the data preparation and reporting.
Actual profit and loss can include new trades, fees and other effects that a particular forecast was not designed to model. A hypothetical fixed-position series can help isolate market movements. The appropriate comparison depends on the purpose and applicable framework. Reconcile differences rather than silently switching series to obtain a more favorable test.
Interpret exception counts as uncertain observations
For 250 independent days with a correctly calibrated 99% VaR and a constant 1% exception probability, the expected number of exceptions is 250 × 0.01 = 2.5. The count is an integer even though its expectation need not be. Its variance under the binomial assumptions is 250 × 0.01 × 0.99 = 2.475, giving a standard deviation of approximately 1.57.
Observing four exceptions rather than exactly 2.5 is not, by itself, a complete rejection of the model. A statistical test compares the observed count with its distribution under a specified null hypothesis and significance level. The assumptions matter: dependence between days or changing forecast coverage can invalidate a simple binomial interpretation. Regulatory exception bands also have their own definitions and consequences; do not invent them from the expected count.
The sample length affects interpretation. Four exceptions over 250 days and four over 40 days have different observed frequencies. A short window responds quickly to recent conditions but provides less information about rare events. Report both the number of observations and the dates covered, and avoid treating a quiet short sample as proof of robust tail calibration.
| Quantity | Calculation | Interpretation |
|---|---|---|
| Expected exception count | 250 × 1% = 2.5 | A model expectation, not a required observed count |
| Count standard deviation | √(250 × 0.01 × 0.99) ≈ 1.57 | Sampling variability under the null |
| Four observed exceptions | 4 / 250 = 1.6% | Observed frequency requiring statistical context |
Original 250-day, 99% VaR backtest under independent-trial assumptions
- Expected exception count
- Calculation: 250 × 1% = 2.5
- Interpretation: A model expectation, not a required observed count
- Count standard deviation
- Calculation: √(250 × 0.01 × 0.99) ≈ 1.57
- Interpretation: Sampling variability under the null
- Four observed exceptions
- Calculation: 4 / 250 = 1.6%
- Interpretation: Observed frequency requiring statistical context
Inspect exception timing, severity and the repair
Two models can have the same exception count and different weaknesses. Five exceptions spread across the year differ from five on consecutive days during a volatility change. Clustering can indicate that the model adapts too slowly or misses conditional dynamics. A coverage test examines frequency; an independence or conditional-coverage assessment examines additional properties. Passing one does not automatically imply passing the others.
Investigate each exception using position reconciliation, market data, valuation and event context. A stale price, omitted basis factor and unexpected market jump require different responses. Exception severity also matters to risk management: a loss just beyond VaR and a loss many times larger both count as one in a basic frequency test, but their implications differ.
After changing the model, distinguish development evidence from fresh validation. Repeatedly adjusting parameters until the same historical window passes can overfit the test. Preserve the original results, explain the mechanism behind the repair and evaluate it on suitable additional or out-of-sample evidence. Stress tests remain useful for risks that the backtest sample has barely observed.
Does zero exceptions prove the model is excellent?
No. It may reflect a short or quiet sample, an overly conservative threshold or a measurement mismatch. Assess calibration, data quality and the intended use. A model can avoid exceptions by producing uninformatively large estimates.
Does one exception mean the VaR forecast failed?
A calibrated quantile is expected to be exceeded occasionally. An exception is an observation to investigate, not automatic proof that the whole model is invalid. The frequency, clustering, severity and underlying cause determine the next assessment.
Try a changed assumption
Two models each have three exceptions. One has all three on consecutive days. Are their backtests equivalent?
Show answer and reasoning
Their counts match, but their timing differs. The clustered record raises a distinct question about dependence or adaptation to changing conditions.
Original learning check. No email or account needed.
Does no exception prove a good model?
No. The model might be conservative, the sample short or the period unusually calm. Evaluate the distribution and model purpose as well as the exception record.
Sources and scope
Links checked 2026-10-10. GARP establishes exam scope; the technical references support the methods identified below. These are selected study notes and original examples, not complete curriculum coverage or official exam questions.
- GARP FRM study materials and 2026 learning objectives
Exam-domain scope and the official route to the annual learning objectives. This page does not certify coverage of every reading.
- Basel Committee: minimum capital requirements for market risk (2019)
Regulatory context for expected shortfall and market-risk model assessment. Local implementation can differ.
- MIT: Value at Risk Models, lecture 7
Loss quantiles, estimation methods and the limitations of VaR.
- NIST: binomial distribution
Binomial probabilities and expected counts for a fixed number of independent trials with a common success probability.
Prepared by MiloPrep editorial team. MiloPrep is independent of GARP. References do not imply endorsement or independent review of these notes.