Simulation based calibration for OncoBayes2

Mon Apr 7 15:19:23 2025

This report documents the results of a simulation based calibration (SBC) run for OncoBayes2. TODO

The calibration data presented here has been generated at and with the OncoBayes git version as:

## Created:  2025-03-14 18:24:03 UTC
## git hash: 93931c92e5ab1b5d4e91a56633979bda3a3a6036
## MD5:      3b0d717520a54320f4f7b35a0dbdeb3d

The MD5 hash of the calibration data file presented here must match the above listed MD5:

## /Users/weberse2/rwork/OncoBayes2/inst/sbc/calibration.rds 
##                        "3b0d717520a54320f4f7b35a0dbdeb3d"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Model description TODO

The fake data simulation function returns … TODO. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

data_scenario N total_divergent min_ess_bulk min_ess_tail max_Rhat total_large_Rhat min_lp_ess_bulk min_lp_ess_tail
combo2_EX 10000 0 137.142 41.504 1.019 0 396.903 388.595
combo2_EXNEX 10000 0 44.494 34.822 1.058 0 345.659 284.837
combo3_EXNEX 10000 0 10.145 36.984 1.153 1 300.542 396.677
log2bayes_EX 10000 0 901.675 705.767 1.019 0 325.927 463.457
log2bayes_EXNEX 10000 0 255.437 385.463 1.016 0 410.029 571.638

Large Rhat is defined as exceeding \(1.1\).

Sampler Adaptation & Performance Overview

ESS speed is in units of ESS per second.

\(\chi^2\) Statistic: Single-agent logistic regression, stratified

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 25.811 31 0.730
beta_group[A,I(log(drug_A/1)),log_slope] 41.958 31 0.091
beta_group[B,I(log(drug_A/1)),intercept] 21.178 31 0.907
beta_group[B,I(log(drug_A/1)),log_slope] 29.222 31 0.558
beta_group[C,I(log(drug_A/1)),intercept] 17.754 31 0.973
beta_group[C,I(log(drug_A/1)),log_slope] 28.614 31 0.589
beta_group[D,I(log(drug_A/1)),intercept] 28.269 31 0.607
beta_group[D,I(log(drug_A/1)),log_slope] 36.307 31 0.235
beta_group[E,I(log(drug_A/1)),intercept] 41.325 31 0.102
beta_group[E,I(log(drug_A/1)),log_slope] 24.128 31 0.805
beta_group[F,I(log(drug_A/1)),intercept] 26.643 31 0.690
beta_group[F,I(log(drug_A/1)),log_slope] 35.533 31 0.263
mu_log_beta[I(log(drug_A/1)),intercept] 44.339 31 0.057
mu_log_beta[I(log(drug_A/1)),log_slope] 27.782 31 0.632
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 25.504 31 0.745
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 24.755 31 0.779
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_intercept] 23.731 31 0.821
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_log_slope] 23.718 31 0.822

\(\chi^2\) Statistic: Single-agent logistic regression, EXchangeable/NonEXchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 23.744 31 0.821
beta_group[A,I(log(drug_A/1)),log_slope] 30.227 31 0.506
beta_group[B,I(log(drug_A/1)),intercept] 23.635 31 0.825
beta_group[B,I(log(drug_A/1)),log_slope] 25.869 31 0.728
beta_group[C,I(log(drug_A/1)),intercept] 18.944 31 0.956
beta_group[C,I(log(drug_A/1)),log_slope] 31.386 31 0.447
mu_log_beta[I(log(drug_A/1)),intercept] 28.083 31 0.617
mu_log_beta[I(log(drug_A/1)),log_slope] 26.874 31 0.679
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 38.496 31 0.167
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 25.210 31 0.758

\(\chi^2\) Statistic: Double combination, fully exchangeable, stratified

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 37.011 31 0.211
beta_group[A,I(log(drug_A/1)),log_slope] 21.811 31 0.889
beta_group[A,I(log(drug_B/2)),intercept] 39.789 31 0.134
beta_group[A,I(log(drug_B/2)),log_slope] 23.456 31 0.832
beta_group[B,I(log(drug_A/1)),intercept] 22.317 31 0.873
beta_group[B,I(log(drug_A/1)),log_slope] 27.066 31 0.669
beta_group[B,I(log(drug_B/2)),intercept] 43.053 31 0.073
beta_group[B,I(log(drug_B/2)),log_slope] 32.736 31 0.382
beta_group[C,I(log(drug_A/1)),intercept] 27.034 31 0.670
beta_group[C,I(log(drug_A/1)),log_slope] 20.493 31 0.925
beta_group[C,I(log(drug_B/2)),intercept] 33.933 31 0.328
beta_group[C,I(log(drug_B/2)),log_slope] 23.206 31 0.842
beta_group[D,I(log(drug_A/1)),intercept] 28.915 31 0.574
beta_group[D,I(log(drug_A/1)),log_slope] 16.461 31 0.985
beta_group[D,I(log(drug_B/2)),intercept] 42.400 31 0.083
beta_group[D,I(log(drug_B/2)),log_slope] 44.774 31 0.052
eta_group[A,I(drug_A/1 * drug_B/2)] 36.128 31 0.241
eta_group[B,I(drug_A/1 * drug_B/2)] 28.154 31 0.613
eta_group[C,I(drug_A/1 * drug_B/2)] 35.904 31 0.249
eta_group[D,I(drug_A/1 * drug_B/2)] 33.549 31 0.345
mu_eta[m[1]] 32.954 31 0.372
mu_log_beta[I(log(drug_A/1)),intercept] 23.642 31 0.825
mu_log_beta[I(log(drug_A/1)),log_slope] 20.877 31 0.915
mu_log_beta[I(log(drug_B/2)),intercept] 52.454 31 0.009
mu_log_beta[I(log(drug_B/2)),log_slope] 35.859 31 0.251
tau_eta[stratum_1,m[1]] 31.539 31 0.439
tau_eta[stratum_2,m[1]] 26.470 31 0.698
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 24.141 31 0.805
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 27.366 31 0.654
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] 30.080 31 0.513
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] 36.685 31 0.222
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_intercept] 26.515 31 0.696
tau_log_beta[stratum_2,I(log(drug_A/1)),tau_log_slope] 25.638 31 0.738
tau_log_beta[stratum_2,I(log(drug_B/2)),tau_intercept] 30.310 31 0.501
tau_log_beta[stratum_2,I(log(drug_B/2)),tau_log_slope] 38.931 31 0.155

\(\chi^2\) Statistic: Double combination, EXchangeable/NonEXchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 30.010 31 0.517
beta_group[A,I(log(drug_A/1)),log_slope] 28.704 31 0.585
beta_group[A,I(log(drug_B/2)),intercept] 55.987 31 0.004
beta_group[A,I(log(drug_B/2)),log_slope] 35.859 31 0.251
beta_group[B,I(log(drug_A/1)),intercept] 28.032 31 0.620
beta_group[B,I(log(drug_A/1)),log_slope] 29.773 31 0.529
beta_group[B,I(log(drug_B/2)),intercept] 31.514 31 0.441
beta_group[B,I(log(drug_B/2)),log_slope] 22.880 31 0.853
beta_group[C,I(log(drug_A/1)),intercept] 34.157 31 0.318
beta_group[C,I(log(drug_A/1)),log_slope] 32.730 31 0.382
beta_group[C,I(log(drug_B/2)),intercept] 36.954 31 0.213
beta_group[C,I(log(drug_B/2)),log_slope] 22.662 31 0.861
eta_group[A,I(drug_A/1 * drug_B/2)] 30.803 31 0.476
eta_group[B,I(drug_A/1 * drug_B/2)] 16.941 31 0.981
eta_group[C,I(drug_A/1 * drug_B/2)] 40.141 31 0.126
mu_eta[m[1]] 30.374 31 0.498
mu_log_beta[I(log(drug_A/1)),intercept] 31.200 31 0.456
mu_log_beta[I(log(drug_A/1)),log_slope] 38.246 31 0.174
mu_log_beta[I(log(drug_B/2)),intercept] 35.872 31 0.251
mu_log_beta[I(log(drug_B/2)),log_slope] 22.042 31 0.882
tau_eta[stratum_1,m[1]] 23.347 31 0.836
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 26.074 31 0.718
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 31.277 31 0.452
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] 35.789 31 0.254
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] 27.834 31 0.630

\(\chi^2\) Statistic: Triple combination, EXchangeable/NonEXchangeable

param statistic df p.value
beta_group[A,I(log(drug_A/1)),intercept] 32.678 31 0.384
beta_group[A,I(log(drug_A/1)),log_slope] 19.565 31 0.945
beta_group[A,I(log(drug_B/2)),intercept] 34.298 31 0.312
beta_group[A,I(log(drug_B/2)),log_slope] 27.168 31 0.664
beta_group[A,I(log(drug_C/4)),intercept] 35.750 31 0.255
beta_group[A,I(log(drug_C/4)),log_slope] 21.235 31 0.906
beta_group[B,I(log(drug_A/1)),intercept] 30.573 31 0.488
beta_group[B,I(log(drug_A/1)),log_slope] 32.429 31 0.396
beta_group[B,I(log(drug_B/2)),intercept] 40.211 31 0.124
beta_group[B,I(log(drug_B/2)),log_slope] 27.136 31 0.665
beta_group[B,I(log(drug_C/4)),intercept] 26.752 31 0.685
beta_group[B,I(log(drug_C/4)),log_slope] 32.486 31 0.393
beta_group[C,I(log(drug_A/1)),intercept] 39.846 31 0.133
beta_group[C,I(log(drug_A/1)),log_slope] 41.907 31 0.091
beta_group[C,I(log(drug_B/2)),intercept] 47.046 31 0.032
beta_group[C,I(log(drug_B/2)),log_slope] 16.160 31 0.987
beta_group[C,I(log(drug_C/4)),intercept] 29.664 31 0.535
beta_group[C,I(log(drug_C/4)),log_slope] 25.882 31 0.727
eta_group[A,I(drug_A/1 * drug_B/2 * drug_C/4)] 15.398 31 0.991
eta_group[A,I(drug_A/1 * drug_B/2)] 27.686 31 0.637
eta_group[A,I(drug_A/1 * drug_C/4)] 22.899 31 0.853
eta_group[A,I(drug_B/2 * drug_C/4)] 38.829 31 0.158
eta_group[B,I(drug_A/1 * drug_B/2 * drug_C/4)] 27.264 31 0.659
eta_group[B,I(drug_A/1 * drug_B/2)] 36.986 31 0.212
eta_group[B,I(drug_A/1 * drug_C/4)] 28.256 31 0.608
eta_group[B,I(drug_B/2 * drug_C/4)] 32.838 31 0.377
eta_group[C,I(drug_A/1 * drug_B/2 * drug_C/4)] 21.888 31 0.886
eta_group[C,I(drug_A/1 * drug_B/2)] 39.770 31 0.134
eta_group[C,I(drug_A/1 * drug_C/4)] 40.499 31 0.118
eta_group[C,I(drug_B/2 * drug_C/4)] 21.338 31 0.903
mu_eta[m[1]] 33.357 31 0.353
mu_eta[m[2]] 31.917 31 0.421
mu_eta[m[3]] 29.562 31 0.540
mu_eta[m[4]] 47.590 31 0.029
mu_log_beta[I(log(drug_A/1)),intercept] 32.288 31 0.403
mu_log_beta[I(log(drug_A/1)),log_slope] 38.899 31 0.156
mu_log_beta[I(log(drug_B/2)),intercept] 22.259 31 0.875
mu_log_beta[I(log(drug_B/2)),log_slope] 29.869 31 0.524
mu_log_beta[I(log(drug_C/4)),intercept] 50.182 31 0.016
mu_log_beta[I(log(drug_C/4)),log_slope] 32.858 31 0.376
tau_eta[stratum_1,m[1]] 34.381 31 0.309
tau_eta[stratum_1,m[2]] 29.786 31 0.528
tau_eta[stratum_1,m[3]] 29.030 31 0.568
tau_eta[stratum_1,m[4]] 25.248 31 0.757
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_intercept] 25.990 31 0.722
tau_log_beta[stratum_1,I(log(drug_A/1)),tau_log_slope] 34.573 31 0.301
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_intercept] 37.050 31 0.210
tau_log_beta[stratum_1,I(log(drug_B/2)),tau_log_slope] 44.774 31 0.052
tau_log_beta[stratum_1,I(log(drug_C/4)),tau_intercept] 43.821 31 0.063
tau_log_beta[stratum_1,I(log(drug_C/4)),tau_log_slope] 31.763 31 0.428

Session Info

## R version 4.4.2 (2024-10-31)
## Platform: aarch64-apple-darwin20
## Running under: macOS Sequoia 15.3.2
## 
## Matrix products: default
## BLAS:   /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRblas.0.dylib 
## LAPACK: /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.0
## 
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
## 
## time zone: Europe/Zurich
## tzcode source: internal
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
## [1] ggplot2_3.5.1    broom_1.0.6      tidyr_1.3.1      dplyr_1.1.4     
## [5] assertthat_0.2.1 knitr_1.47       here_1.0.1      
## 
## loaded via a namespace (and not attached):
##  [1] gtable_0.3.5      jsonlite_1.8.8    compiler_4.4.2    tidyselect_1.2.1 
##  [5] jquerylib_0.1.4   scales_1.3.0      yaml_2.3.8        fastmap_1.2.0    
##  [9] R6_2.5.1          generics_0.1.3    backports_1.5.0   tibble_3.2.1     
## [13] munsell_0.5.1     rprojroot_2.0.4   bslib_0.7.0       pillar_1.9.0     
## [17] rlang_1.1.4       utf8_1.2.4        cachem_1.1.0      xfun_0.45        
## [21] sass_0.4.9        cli_3.6.2         withr_3.0.0       magrittr_2.0.3   
## [25] digest_0.6.35     grid_4.4.2        lifecycle_1.0.4   vctrs_0.6.5      
## [29] evaluate_0.24.0   glue_1.7.0        colorspace_2.1-0  fansi_1.0.6      
## [33] rmarkdown_2.29    purrr_1.0.2       pkgconfig_2.0.3   htmltools_0.5.8.1