ABPI Code Bench

Does the model know what it doesn't know? Models predict PMCPA clause (§) rulings and say how sure they are. That confidence is scored as a probability: high when right, low when wrong.

10,497 items 1,515 cases 2001–2024 Code years

T1 verdict with the defence · T2 without it · T3 appeal survival

P1 stated · P2 repeat agreement · P3 pooled · P4 deferral

Leaderboard
Calibration

accuracy (the %) confidence (the 0–1) over-confident under-confident perfectly calibrated 95% CI P4 confidence range

T1
P1 · N=100
gemini-3.7-flash
grok-4.6
claude-opus-5
gpt-5.6-sol
claude-sonnet-5
deepseek-v4-pro
kimi-k3
gpt-5.6-terra
glm-5.2
gpt-5.6-luna
claude-haiku-4-5-20251001
P2 · N=100
gemini-3.7-flash
grok-4.6
claude-sonnet-5
gpt-5.6-sol
glm-5.2
deepseek-v4-pro
claude-opus-5
gpt-5.6-terra
gpt-5.6-luna
claude-haiku-4-5-20251001
P3 · N=100
gemini-3.7-flash
gpt-5.6-sol
grok-4.6
glm-5.2
claude-opus-5
gpt-5.6-terra
deepseek-v4-pro
claude-sonnet-5
gpt-5.6-luna
claude-haiku-4-5-20251001
P1 · N=100
P2 · N=100
P3 · N=100
P4 · N=100
T2
P1 · N=100
claude-opus-5
gpt-5.6-sol
gemini-3.7-flash
kimi-k3
grok-4.6
claude-sonnet-5
gpt-5.6-terra
glm-5.2
deepseek-v4-pro
claude-haiku-4-5-20251001
gpt-5.6-luna
P2 · N=100
claude-opus-5
gpt-5.6-sol
gpt-5.6-terra
gemini-3.7-flash
grok-4.6
claude-sonnet-5
glm-5.2
deepseek-v4-pro
gpt-5.6-luna
claude-haiku-4-5-20251001
P3 · N=100
gpt-5.6-sol
claude-opus-5
gemini-3.7-flash
gpt-5.6-terra
grok-4.6
claude-sonnet-5
glm-5.2
gpt-5.6-luna
deepseek-v4-pro
claude-haiku-4-5-20251001
P1 · N=100
P2 · N=100
P3 · N=100
P4 · N=100
T3
P1 · N=200
grok-4.6
claude-sonnet-5
claude-opus-5
gpt-5.6-sol
gpt-5.6-terra
kimi-k3
gemini-3.7-flash
gpt-5.6-luna
claude-haiku-4-5-20251001
glm-5.2
deepseek-v4-pro
P2 · N=200
glm-5.2
gpt-5.6-terra
grok-4.6
gpt-5.6-sol
gemini-3.7-flash
deepseek-v4-pro
gpt-5.6-luna
claude-opus-5
claude-haiku-4-5-20251001
claude-sonnet-5
P3 · N=200
gpt-5.6-terra
glm-5.2
grok-4.6
gemini-3.7-flash
gpt-5.6-sol
claude-sonnet-5
claude-opus-5
claude-haiku-4-5-20251001
gpt-5.6-luna
deepseek-v4-pro
P1 · N=200
P2 · N=200
P3 · N=200
P4 · N=100
claude-opus-5 claude-sonnet-5 claude-haiku-4-5-20251001 gpt-5.6-sol gpt-5.6-terra gpt-5.6-luna gemini-3.7-flash grok-4.6 deepseek-v4-pro kimi-k3 glm-5.2
Cases 264 of 1,515 shown
CASE/0835/12/25Gilead84✗/231
CASE/0837/12/25Amgen118✗/232
CASE/0615/06/25Gossamer Bio17✗/116
CASE/0558/04/25Boehringer Ingelheim95✗/232
CASE/0513/03/25Organon0✗/116
CASE/0409/12/24Alnylam28✗/116
CASE/0392/12/24SOBI2✗/116
CASE/0381/11/24Chiesi77✗/116
CASE/0378/11/24GlaxoSmithKline53✗/81
CASE/0363/11/24Sanofi54✗/81
CASE/0277/08/24AstraZeneca25✗/197
CASE/0246/07/24Ferring53✗/232
CASE/0228/07/24GlaxoSmithKline89✗/232
CASE/0222/07/24Accord-UK76✗/232
AUTH/3924/6/24Roche37✗/232
CASE/0216/06/24Bayer91✗/197
AUTH/3911/5/24Theramex22✗/232
AUTH/3909/5/24GlaxoSmithKline19✗/116
AUTH/3890/4/24Novo Nordisk38✗/197
AUTH/3880/2/24GlaxoSmithKline90✗/116
AUTH/3873/01/24Bayer53✗/116
AUTH/3869/12/23Eli Lilly39✗/116
AUTH/3836/10/23GlaxoSmithKline21✗/116
AUTH/3835/10/23Moderna25✗/81
AUTH/3830/9/23Daiichi Sankyo148✗/232
AUTH/3825/9/23Janssen51✗/80
AUTH/3815/8/23Moderna81✗/197
AUTH/3777/6/23AstraZeneca72✗/116
AUTH/3763/4/23Novo Nordisk0✗/197
AUTH/3755/3/23Novo Nordisk14✗/197
AUTH/3714/11/22GlaxoSmithKline0✗/116
AUTH/3707/11/22AstraZeneca62✗/197
AUTH/3700/10/22Ethypharm34✗/81
AUTH/3683/8/22AstraZeneca71✗/232
AUTH/3676/7/22Roche11✗/197
AUTH/3668/6/22Galapagos189✗/232
AUTH/3663/6/22Roche74✗/232
AUTH/3650/5/22UCB3✗/81
AUTH/3631/4/22AstraZeneca and Daiichi Sankyo11✗/232
AUTH/3624/3/22Pacira BioScience73✗/81
AUTH/3614/2/22Eli Lilly17✗/429
AUTH/3599/1/22AstraZeneca43✗/232
AUTH/3592/12/21Janssen68✗/232
AUTH/3591/12/21Pfizer60✗/81
AUTH/3583/11/21ALK-Abello49✗/116
AUTH/3565/10/21Otsuka Europe and Otsuka UK0✗/232
AUTH/3545/7/21Novo Nordisk44✗/116
AUTH/3535/7/21SOBI38✗/81
AUTH/3524/6/21GlaxoSmithKline92✗/116
AUTH/3525/6/21Novo Nordisk61✗/429
AUTH/3509/5/21Tillotts70✗/116
AUTH/3499/4/21Daiichi Sankyo50✗/232
AUTH/3493/3/21AbbVie37✗/232
AUTH/3483/3/21Eli Lilly66✗/116
AUTH/3475/2/21Lundbeck25✗/232
AUTH/3454/1/21Sanofi94✗/232
AUTH/3452/1/21Lundbeck29✗/116
AUTH/3441/12/20Allergan49✗/81
AUTH/3432/12/20GlaxoSmithKline49✗/116
AUTH/3421/11/20Novartis41✗/116
AUTH/3428/11/20Leo51✗/197
AUTH/3417/11/20Daiichi Sankyo2✗/116
AUTH/3408/10/20Amgen1✗/116
AUTH/3396/10/20Daiichi Sankyo0✗/81
AUTH/3388/9/20Daiichi Sankyo0✗/81
AUTH/3364/6/20AstraZeneca90✗/313
AUTH/3361/6/20Merck Sharp & Dohme85✗/232
AUTH/3349/5/20Bristol-Myers Squibb and Pfizer33✗/232
AUTH/3343/5/20Shionogi Europe0✗/116
AUTH/3335/4/20Britannia92✗/197
AUTH/3333/4/20Novo Nordisk114✗/116
AUTH/3320/3/20Takeda93✗/116
AUTH/3322/3/20Bayer20✗/116
AUTH/3315/3/20Novo Nordisk0✗/197
AUTH/3301/1/20Vifor7✗/232
AUTH/3299/1/20Novo Nordisk0✗/197
AUTH/3285/12/19Daiichi Sankyo17✗/313
AUTH/3278/11/19Strides3✗/232
AUTH/3267/10/19Leo27✗/232
AUTH/3265/10/19Bracco37✗/232
AUTH/3259/10/19Otsuka Europe0✗/81
AUTH/3256/10/19Novartis9✗/81
AUTH/3246/9/19Pfizer and Bristol-Myers Squibb15✗/81
AUTH/3245/9/19Novo Nordisk0✗/116
AUTH/3228/7/19Ipsen185✗/197
AUTH/3226/7/19Teva7✗/81
AUTH/3202/6/19Napp16✗/116
AUTH/3190/4/20Takeda32✗/116
AUTH/3191/4/19Amgen34✗/116
AUTH/3193/4/19Novartis33✗/116
AUTH/3194/4/19GlaxoSmithKline69✗/313
AUTH/3195/4/19Chiesi27✗/313
AUTH/3197/4/19Ethypharm4✗/116
AUTH/3176/3/19Orion98✗/232
AUTH/3167/2/19Novartis30✗/116
AUTH/3151/1/19Otsuka Europe29✗/116
AUTH/3141/12/18Merck Sharp & Dohme43✗/81
AUTH/3137/12/18Gilead31✗/348
AUTH/3134/12/18Shield65✗/232
AUTH/3131/12/18Napp0✗/116
AUTH/3125/11/18Merck Sharp & Dohme2✗/232
AUTH/3118/11/18Tesaro6✗/116
AUTH/3087/9/18GlaxoSmithKline17✗/116
AUTH/3102/9/18Lilly76✗/116
AUTH/3056/8/18GlaxoSmithKline46✗/116
AUTH/3053/7/18Daiichi Sankyo0✗/116
AUTH/3044/6/18Roche154✗/232
AUTH/3045/6/18Pfizer114✗/197
AUTH/3046/6/18AstraZeneca58✗/81
AUTH/3037/4/18Shield0✗/116
AUTH/3028/3/18AbbVie12✗/197
AUTH/3022/2/18Servier55✗/232
AUTH/3010/1/18Daiichi Sankyo1✗/116
AUTH/2978/9/17Janssen24✗/81
AUTH/2965/8/17Biogen52✗/81
AUTH/2962/7/17Pierre Fabre0✗/81
AUTH/2943/3/17Bayer185✗/232
AUTH/2923/12/16Merck Sharp & Dohme79✗/197
AUTH/2913/12/16Janssen1✗/232
AUTH/2884/10/16Intrapharm7✗/232
AUTH/2879/10/16Bristol-Myers Squibb2✗/116
AUTH/2836/4/16Janssen24✗/197
AUTH/2834/4/16Amdipharm Mercury110✗/232
AUTH/2833/4/16Bausch & Lomb3✗/232
AUTH/2830/3/16Vifor60✗/196
AUTH/2828/3/16Vifor2✗/197
AUTH/2825/3/16Boehringer Ingelheim and Lilly55✗/116
AUTH/2824/2/16Chiesi40✗/232
AUTH/2823/2/16Grunenthal114✗/116
AUTH/2811/12/15Daiichi Sankyo54✗/81
AUTH/2813/12/15Pfizer9✗/232
AUTH/2808/12/15Napp32✗/81
AUTH/2805/12/15Guerbet100✗/232
AUTH/2806/12/15Mallinckrodt (part of Guerbet)47✗/116
AUTH/2795/9/15Napp84✗/197
AUTH/2779/7/15Merck Sharp & Dohme33✗/232
AUTH/2781/7/15Bristol-Myers Squibb91✗/232
AUTH/2763/5/15AstraZeneca5✗/116
AUTH/2739/11/14Daiichi Sankyo1✗/116
AUTH/2721/7/14Genzyme2✗/81
AUTH/2714/5/14Pfizer139✗/197
AUTH/2705/3/14Merck Serono7✗/81
AUTH/2699/2/14Merck Sharp & Dohme68✗/197
AUTH/2685/12/13Galderma1✗/116
AUTH/2684/12/13Galderma4✗/116
AUTH/2674/11/13GlaxoSmithKline19✗/116
AUTH/2653/11/13Sanofi18✗/116
AUTH/2645/10/13GlaxoSmithKline and Amgen0✗/116
AUTH/2644/10/13Galen33✗/197
AUTH/2643/10/13Pharmaxis2✗/116
AUTH/2633/8/13Leo8✗/232
AUTH/2624/8/13HRA Pharma49✗/116
AUTH/2618/7/13Chiesi57✗/232
AUTH/2615/7/13Amdipharm Mercury119✗/232
AUTH/2614/6/13Allergan74✗/81
AUTH/2611/6/13Janssen4✗/116
AUTH/2607/5/13GlaxoSmithKline71✗/197
AUTH/2605/5/13UCB154✗/232
AUTH/2604/5/13Sanofi0✗/116
AUTH/2600/4/13Sanofi50✗/81
AUTH/2580/2/13Gedeon Richter45✗/116
AUTH/2575/2/13Gedeon Richter3✗/116
AUTH/2571/12/12Pharmacosmos28✗/116
AUTH/2570/12/12Napp25✗/116
AUTH/2535/10/12Pfizer1✗/197
AUTH/2528/8/12Shire153✗/197
AUTH/2519/6/12Lilly63✗/116
AUTH/2509/6/12Roche61✗/116
AUTH/2483/2/12Pfizer0✗/232
AUTH/2482/2/12Novo Nordisk1✗/81
AUTH/2481/2/12ProStrakan35✗/197
AUTH/2475/1/12Johnson & Johnson8✗/81
AUTH/2470/1/12Novo Nordisk50✗/197
AUTH/2471/1/12Sanofi31✗/81
AUTH/2464/12/11Novartis27✗/232
AUTH/2449/11/11Boehringer Ingelheim and Lilly25✗/81
AUTH/2448/10/11Boehringer Ingelheim72✗/116
AUTH/2439/9/11Grunenthal70✗/81
AUTH/2422/7/11Vifor0✗/232
AUTH/2410/6/11Bristol-Myers Squibb74✗/81
AUTH/2420/7/11Biogen Idec42✗/81
AUTH/2414/6/11AstraZeneca74✗/197
AUTH/2417/6/11Servier5✗/313
AUTH/2397/4/11Warner Chilcott114✗/197
AUTH/2394/3/11Lundbeck and Teva11✗/116
AUTH/2389/2/11Alcon10✗/116
AUTH/2385/2/11Novo Nordisk34✗/116
AUTH/2383/2/11Bayer93✗/116
AUTH/2377/12/10Merck Sharp & Dohme35✗/81
AUTH/2367/10/10Takeda47✗/116
AUTH/2358/9/10Genus76✗/232
AUTH/2355/9/10Takeda2✗/197
AUTH/2345/8/10Daiichi Sankyo5✗/232
AUTH/2333/7/10Bayer6✗/232
AUTH/2329/7/10Genus99✗/232
AUTH/2325/6/10Boehringer Ingelheim2✗/116
AUTH/2322/5/10AstraZeneca15✗/81
AUTH/2320/5/10AstraZeneca9✗/116
AUTH/2310/4/10Lilly1✗/81
AUTH/2308/4/10Norgine14✗/197
AUTH/2301/3/10Lilly63✗/232
AUTH/2291/1/10Flynn Pharma114✗/116
AUTH/2287/12/09Roche and Chugai53✗/81
AUTH/2278/11/09Lundbeck101✗/116
AUTH/2273/10/09Novo Nordisk3✗/116
AUTH/2274/10/09Allergan8✗/197
AUTH/2270/10/09Merz16✗/81
AUTH/2264/9/09GlaxoSmithKline76✗/232
AUTH/2256/8/09Napp78✗/232
AUTH/2245/6/09Lundbeck86✗/232
AUTH/2243/6/09Takeda12✗/232
AUTH/2239/6/09Bracco5✗/232
AUTH/2232/5/09Cephalon203✗/232
AUTH/2225/4/09Astellas65✗/232
AUTH/2211/3/09Boehringer Ingelheim22✗/232
AUTH/2209/2/09Leo24✗/81
AUTH/2198/1/09Solvay32✗/232
AUTH/2189/12/08ProStrakan35✗/116
AUTH/2177/10/08Merz8✗/197
AUTH/2158/8/08Sanofi-Aventis7✗/81
AUTH/2137/6/08Ranbaxy120✗/232
AUTH/2119/4/08Merz9✗/232
AUTH/2101/2/08GlaxoSmithKline9✗/81
AUTH/2096/1/08Trinity-Chiesi37✗/232
AUTH/2093/1/08Pfizer11✗/197
AUTH/2091/1/08Guerbet15✗/116
AUTH/2089/1/08Goldshield25✗/116
AUTH/2071/11/07Takeda7✗/116
AUTH/2063/10/07Abbott33✗/116
AUTH/2056/10/07Novo Nordisk1✗/116
AUTH/2054/10/07Sanofi-Aventis1✗/232
AUTH/2045/9/07Novo Nordisk4✗/116
AUTH/2038/8/07Sanofi-Aventis87✗/116
AUTH/2035/8/07Leo83✗/232
AUTH/2028/7/07Sanofi-Aventis80✗/81
AUTH/2023/7/07Pfizer1✗/197
AUTH/2020/7/07Merck Sharp & Dohme73✗/81
AUTH/2017/7/07Teva17✗/116
AUTH/2007/5/07Teva29✗/116
AUTH/1999/5/07Pfizer96✗/232
AUTH/1997/5/07Takeda112✗/197
AUTH/1984/4/07Sanofi-Aventis and Procter & Gamble70✗/348
AUTH/1981/3/07ALK-Abello138✗/232
AUTH/1971/3/07Roche and GlaxoSmithKline19✗/81
AUTH/1941/1/07Altana0✗/116
AUTH/1940/1/07Bayer10✗/116
AUTH/1933/12/06Roche and GlaxoSmithKline86✗/116
AUTH/1918/11/06Amgen67✗/232
AUTH/1915/11/06Novartis119✗/313
AUTH/1910/11/06GlaxoSmithKline50✗/81
AUTH/1902/10/06Sanofi-Aventis and Procter & Gamble0✗/81
AUTH/1872/7/06Shire89✗/116
AUTH/1871/7/06Sanofi-Aventis11✗/81
AUTH/1864/7/06Lilly28✗/232
AUTH/1857/6/06Roche0✗/197
AUTH/1855/6/06Roche14✗/81
AUTH/1849/6/06AstraZeneca98✗/232
AUTH/1848/6/06Janssen-Cilag18✗/81
AUTH/1847/6/06Altana86✗/197
AUTH/1824/4/06Servier49✗/81
AUTH/1802/2/06Merck110✗/116
AUTH/1801/2/06GlaxoSmithKline101✗/197
AUTH/1799/2/06Abbott118✗/197
AUTH/1800/2/06AstraZeneca13✗/232