--- title: "Model validation" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Model validation} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ## Introduction Model validation is a common part of actuarial pricing work. After model estimation and coefficient interpretation, it is necessary to assess whether the model performs adequately and behaves in a stable and interpretable way. In practice, model validation typically considers several dimensions: - comparative model performance - coefficient structure - predictive stability - distributional diagnostics - portfolio-level behaviour `insurancerating` provides tools for several of these validation tasks. The purpose of validation is not only to assess statistical fit, but also to determine whether a model is suitable for use in a pricing context. ## Example setup The examples below use a simple frequency modelling setup based on `MTPL2`. ```{r} library(insurancerating) library(dplyr) df <- MTPL2 |> mutate(across(c(area), as.factor)) |> mutate(across(c(area), ~ set_reference_level(., exposure))) mod1 <- glm( nclaims ~ area, offset = log(exposure), family = poisson(), data = df ) mod2 <- glm( nclaims ~ area + premium, offset = log(exposure), family = poisson(), data = df ) ``` ## Step 1 — Comparative model performance A first validation step is to compare alternative model specifications. ```{r} model_performance(mod1, mod2) ``` This provides commonly used summary measures of model fit, such as RMSE. The purpose of this step is to assess whether the addition or removal of model terms leads to a materially different fit. In practice, this comparison is often used to support modelling choices before moving to tariff interpretation. ## Step 2 — Coefficient inspection Model validation is not limited to summary fit statistics. The coefficient structure also needs to be reviewed. ```{r} rating_table(mod1, mod2, model_data = df, exposure = "exposure") |> autoplot() ``` This is used to assess: - the relative size of fitted effects - the ordering of factor levels - the exposure behind each level - whether differences are plausible and stable In pricing practice, this is often part of validation, because a model with slightly better fit may still be less suitable if its coefficient structure is difficult to interpret or unstable in low-exposure segments. ## Step 3 — Predictive stability Single performance measures provide only a point estimate. In many pricing contexts, it is also relevant to assess how stable that performance is under small variations in the data. ```{r} bootstrap_performance(mod1, df, n_resamples = 100, show_progress = FALSE) |> autoplot() ``` This evaluates predictive stability by repeatedly refitting the model on bootstrap samples and storing the resulting RMSE values. The output is used to assess: - the variability of model performance - whether the fitted model behaves consistently - whether the model is highly sensitive to changes in the underlying sample This is particularly relevant when portfolios contain sparse segments or large claim volatility. ## Step 4 — Dispersion checks For Poisson models, it is common practice to check whether the variance assumption is broadly appropriate. ```{r} check_overdispersion(mod1) ``` A dispersion ratio above 1 indicates that the observed variance exceeds the variance implied by the Poisson model. This does not automatically invalidate the model, but it does provide an important diagnostic signal. In pricing practice, overdispersion may indicate: - omitted heterogeneity - model misspecification - clustering in the data - or unmodelled portfolio structure ## Step 5 — Residual diagnostics Residual diagnostics provide an additional view of model adequacy. ```{r} check_residuals(mod1, n_simulations = 600) |> autoplot() ``` This step is used to assess whether the residual behaviour is broadly consistent with the fitted model assumptions. In GLM settings, simulation-based residual diagnostics are often more useful than classical residual plots, because they allow the fitted model to be evaluated relative to its own implied distribution. The purpose of this step is not to search for perfect residual behaviour, but to identify material deviations that may be relevant for pricing. ## Step 6 — Portfolio-level structure Validation is also performed at portfolio or model-point level. ```{r} grid <- rating_grid(mod1) head(grid) ``` `rating_grid()` aggregates the fitted model to observed model-point combinations. This is useful when validation requires a more structured view of: - observed portfolio composition - combinations of rating factors - model-point level summaries - compact portfolio input for further review This step is particularly relevant when moving from model validation to tariff review or implementation support. ## Validation in context In practice, model validation is rarely based on a single statistic. A validation exercise often combines: - comparative performance measures - coefficient inspection - predictive stability - residual and dispersion diagnostics - portfolio-level review These steps serve different purposes: - performance measures assess fit - coefficient inspection assesses interpretability - bootstrap analysis assesses stability - diagnostics assess model adequacy - portfolio review assesses practical usability Taken together, they provide a structured basis for deciding whether a model is suitable for pricing use. ## Summary One possible validation sequence in `insurancerating` is: ```{r, eval = FALSE} model_performance(...) # compare fitted models rating_table(...) |> autoplot() # inspect coefficient structure bootstrap_performance(...) # assess predictive stability check_overdispersion(...) # assess dispersion check_residuals(...) # inspect residual behaviour rating_grid(...) # review model-point structure ``` The purpose of validation is not only to assess model fit, but to determine whether the fitted model is: - statistically adequate - stable under data variation - interpretable in tariff terms - suitable for practical pricing use ## Next steps For a modelling example, see: - [Getting started](getting-started.html) For the refinement step after validation, see: - [Refinement building blocks](refinement-workflow.html). For the conceptual background to exposure, risk premium, and tariff structure, see: - [Pricing workflow building blocks](pricing-workflow-building-blocks.html)