Validation Tools Taxonomy

Working specification

GenAI
Pharmacometrics
working document
The working specification for the Validation Tools Taxonomy project: what gets validated in a pharmacometrics activity, the tools that already exist for each kind, the gaps, and a validation plan that applies to an agent, an intern or a senior associate alike.
Last edited

October 21, 2026

Started 2026-10-14. The taxonomy in Section 3 began as eleven kinds, written from the tools on the Pharmacometrics Tools page. On 2026-10-21 it became thirteen, adding study design and assay, after the search recorded in the validation taxonomy. The project’s earlier question, validating code an agent wrote, is now one document under this one.

1. Goal

Two deliverables:

  1. A taxonomy of what gets validated in a pharmacometrics (PMX) activity, with the tools that already exist for each kind and the gaps where none does.
  2. A validation plan template: for any activity, which kinds of validation apply, which tool checks each, how deep the check goes given the risk, and who does it.

Both are meant for work done by anyone: a coding agent, an intern, a postdoc, a junior or a senior associate.

Out of scope:

  • Evaluating an AI model in general, such as benchmarks. The question is whether one analysis is right.
  • Computerized system validation in the regulatory sense, such as 21 CFR Part 11 for software that handles clinical records.
  • Writing the missing tools. This project finds the gaps and ranks them. Filling one becomes its own project.

2. One Plan for Any Author

The validation an analysis needs is set by the analysis and by the cost of getting it wrong, and the same questions apply whoever did the work. What changes with the author is where errors are expected, and so which checks carry the most weight. The rows below are a working guess.

Where errors are expected, by author
Author Errors to expect Where the weight goes
Coding agent Code that runs cleanly and computes the wrong quantity; a silent change to data handling; a fluent summary of a paper it misread Independent reproduction of the main results; checks on outputs; claims against sources
Intern or new hire Not knowing what a plausible parameter value looks like; unfamiliar data conventions; not asking The model’s plausibility; the data; whether the result answers the question
Experienced modeler Template code reused where it does not fit; assumed data conventions; the question drifting during the work The question; the data

Working with a person and working with an agent share the same steps. Each needs background, instructions, checking and correction, and the steps differ in where the result is kept.

The steps of working with a person and with an agent
Step A person An agent
Background Onboarding, a reading list, the team’s conventions Skill files and instruction files such as CLAUDE.md
Instructions The specification The same specification
Checking The validation plan The same validation plan
Correction Feedback, which the person carries forward A change to the skill or instruction file; anything not written down is gone next session
Explaining the work later Can be asked, and the final presentation of an internship tests it Only what is in the record

The checking row is what this project is about. An intern who directs an agent is also the agent’s first reviewer, which makes the validation plan part of what an intern learns. The PMX internship specification uses this.

3. The Taxonomy

Thirteen kinds of validation, in the order an analysis meets them. The table is the summary; each kind has a subsection below with the same three parts. The validation taxonomy uses the same numbers and lists the published guidances, papers, checklists and tools for each kind; the subsections here keep to the tools on this site and the main outside ones.

Thirteen kinds of validation and the tools for each
# Kind Asks Existing tools Gap
1 Question Is this the right question for the decision? Context of Use table, Dose Finding Framework No check after the work
2 Study design Can this nonclinical or clinical study answer the question? Sizing Studies project, PopED No check that an existing study supports the model
3 Causal assumptions What causes what, and could a confounder explain the relationship the model estimates? Causality checklist, causal diagrams talk Not yet a step in any plan
4 Assay Does the assay measure what the model assumes, accurately enough? ICH M10, FDA immunogenicity assay guidance Assays are validated for the laboratory, not for the model
5 Analysis dataset Is the analysis dataset the right one, built correctly? Validation checklist (import, modify), xGx No dataset assembly checklist
6 Sources and prior knowledge Does each claim and prior say what its source says? Status markers in references.qmd, Reading Papers guide Not a portable tool
7 Code Do the model, plotting and simulation code compute what they claim? R coding checklist, Validation checklist Agent-written code
8 Model Does the model describe the data, with plausible parameters? ggPMX, xgxr, See-Value No model evaluation checklist
9 Predictions Is the uncertainty in a prediction stated, and does the model predict better than a simple benchmark? Uncertainty cheat sheet and checklist, pedigree table, Stein and Looby 2018 No benchmarking procedure
10 Conclusions Does the conclusion follow from the result? Causality checklist, and the diagram from kind 3 To assess
11 Communication Do the figures and slides show what was found, without overstating it? Graphics principles cheat sheet, Presentation checklist, evaluation form Written reports
12 Reproducibility Can someone else rerun it and get the same numbers, and is there a record of what was checked? Validation checklist Record of an agent session
13 Use Was the answer used for the decision it was meant for? None No follow-up practice

Kinds 7, 8 and 9 are the three the project started from: the R coding checklist, the diagnostic plots, and the uncertainty checklist. Kinds 1, 3, 5, 6 and 10 to 13 were added in the first version of this list, and study design (2) and assay (4) were added after the search in the validation taxonomy. Section 6 question 4 asks whether thirteen is complete.

The names are what make the plan in Section 4 work. Once a kind of validation has a name, a plan can ask of it whether the question at hand needs it, and record the answer. A kind with no name is never asked about, so it gets skipped without anyone deciding to skip it. The first version of this list left out causal assumptions, although the Causality Checklist already existed, and the gap was found by reading down the names. This is the line attributed to Confucius, “The beginning of wisdom is calling things by their right names”, applied to validation.

3.1 Question

Asks. Is this the right question for the decision, with a named person who will use the answer, and is the analysis sized to it?

Existing tools.

Gap. All three are used before the work. Nothing checks at the end that the answer still answers the question asked at the start.

3.2 Study Design

Asks. Can the nonclinical or clinical study answer the question the analysis will put to it? For a model this means doses spread widely enough to estimate a dose-response, samples taken when the drug and the response are changing, enough subjects, and a population that matches the one the prediction is for.

Existing tools.

  • The Sizing Studies project.
  • PopED, which computes how well a proposed design would estimate a model’s parameters before the study runs.
  • ICH E8(R1) and E9(R1) for clinical studies, and the ARRIVE 2.0 guidelines for reporting animal studies, listed in the taxonomy.

Gap. The design literature optimizes a study before it runs, or says how to report it. Nothing asks, before an analysis starts, whether a study already done can support the model about to be fitted: whether its dose range, sampling and sample size are enough for the parameters the decision rests on. PopED could answer that, and is rarely run on a study already done.

3.3 Causal Assumptions

Asks. What causes what in the system the analysis describes, and could a variable that affects both the exposure and the response explain the relationship the model estimates? Such a variable is a confounder. In an exposure-response analysis of a monoclonal antibody, sicker patients often clear the drug faster, so they have lower exposure and worse outcomes, and the analysis sees exposure predicting response where disease severity is driving both.

The check is to draw a causal diagram for the question before the data are analyzed, with an arrow from each cause to each effect, and to look in it for paths from exposure to response that do not go through the drug’s effect. Each such path is a confounder to adjust for, or a reason the analysis cannot answer the causal question as asked.

Existing tools.

Gap. The tools exist and are not a step in any plan. The proposal here is that every validation plan carries this row, and that the row holds either a causal diagram or a sentence saying why the question needs none, such as a randomized dose comparison where dose is the only exposure. Kind 9 asks how uncertain an analysis’s assumptions are; this kind asks whether the causal structure behind them is right. Whether every model needs a diagram is Section 6 question 6.

3.4 Assay

Asks. Does the assay measure what the model assumes it measures, such as free or total drug, or free or total target, and is it accurate and precise enough across the range the analysis uses? How are values below the limit of quantification and anti-drug antibodies handled?

Existing tools.

  • ICH M10, on bioanalytical method validation for drug concentrations.
  • The FDA guidance on immunogenicity testing, for anti-drug antibody assays.
  • Beal (2001) on fitting a model when some data are below the limit of quantification.

All three are in the taxonomy, with the guidance on biomarker assays.

Gap. Assay validation is written for the laboratory. Nothing is written for a modeler checking that the measured quantity is the one in the model: free against total drug in a target-mediated drug disposition model, or whether an anti-drug antibody assay’s drug tolerance hides the patients whose clearance the model attributes to antibodies.

3.5 Analysis Dataset

Asks. Is the analysis dataset built from the right files and versions, are doses, times, missing values and values below the limit of quantification handled as intended, and do derived variables and exclusions match the plan?

Existing tools.

  • The Validation Checklist, sections on importing and modifying data.
  • xGx exploratory graphics, which show the data before any model sees it.

Gap. No checklist for assembling a PMX analysis dataset against its data specification: dosing records, event types, time since first dose, covariate merges. Errors here run cleanly through every later step.

3.6 Sources and Prior Knowledge

Asks. Does each claim drawn from a paper, a label or a regulatory review say what the source says, and does each parameter value or prior taken from the literature match its source and apply to this population?

Existing tools.

  • The status markers this site’s projects put on every entry of a references.qmd (❌ not checked, ⚠️ transcribed and unverified, ✅ checked), shaped as in the TCE-IPDE references.
  • The Reading Papers guide.

Gap. The markers are a convention of this repository, not a tool someone else can pick up. There is no procedure for checking an agent’s summary of a paper, which is where this check is most often skipped.

3.7 Code

Asks. Does the code compute what it claims to, can someone else read it, and does it run cleanly from the start?

Existing tools.

  • R Coding Checklist: PDF · Google Doc.
  • The Validation Checklist, whose planning section chooses the depth of review from the risk: unit tests, spot checks, code review, rerunning step by step, or independent reproduction.

Gap. Both assume a person wrote the code and a second person reviews it. Validating Code Written by an AI Agent works out what changes when an agent is the author.

3.8 Model

Asks. Does the model describe the data, and are its parameters plausible? The usual diagnostics: goodness-of-fit plots, residuals (conditional weighted residuals, normalized prediction distribution errors), visual predictive checks, parameter precision, shrinkage, and parameter values against physiology and published models.

Existing tools.

Gap. No written model evaluation checklist. The skills project is writing one for an agent, and the same text could serve a person. Plausibility against prior knowledge is the part least covered by any plot.

3.9 Predictions

Asks. Two questions. Is the uncertainty in a prediction stated, including the uncertainty from assumptions and not only from parameter estimates, and is the prediction inside the range the data support? And does the model predict better than a simple one would? A complex model, such as a quantitative systems pharmacology (QSP) model, is checked by fitting a simple model to the same data and comparing how well each predicts data neither was fitted to: a later cohort, a new dose, another study. If the simple model predicts as well, the complex model’s extra structure is not what the prediction rests on.

Existing tools.

Gap. The paper is an argument for benchmarking, not a procedure: which simple model to pick, which data to hold out, and what difference in predictive performance counts. Writing that down is the gap. The uncertainty checklist also needs checking for extrapolation, to doses, populations or durations outside the data.

3.10 Conclusions

Asks. Does the conclusion follow from the result? A model can be right and the claim drawn from it wrong, most often a causal claim from observational exposure-response data. Kind 3 sets up the causal diagram at the start; this kind checks the final claims against it.

Existing tools.

  • The causal diagram from kind 3, which a causal claim is checked against, and the Causality Checklist that goes with it (3.3).

Gap. To assess. Multiple comparisons and selective reporting are not obviously covered by anything listed here.

3.11 Communication

Asks. Do the figures, slides and report show what the analysis found, with the uncertainty, and without overstating it?

Existing tools.

Gap. Nothing for a written report. This repository’s writing guide does part of the job for an agent’s drafts, and is not written for a reviewer.

3.12 Reproducibility

Asks. Can someone else rerun the analysis from its inputs and get the same numbers, and is there a record of what was checked, by whom?

Existing tools.

  • The Validation Checklist, overall section.
  • Version control, and on this site the frozen computation cache, which replays a page’s results exactly.

Gap. When an agent did the work, the prompts, the specification and the session may be part of the record. Question 5 of the agent-written code document asks which.

3.13 Use

Asks. Was the answer used for the decision it was meant to inform, and did the decision go the way the analysis said it should?

Existing tools. None. The PMX internship specification checks use three months after an internship, for one project.

Gap. No practice of following an analysis to its decision. Without it, nothing tells whether the validation upstream was at the right depth.

4. The Validation Plan

A validation plan is written at the start of an activity, next to its specification, and filled in as the checks are done. One plan per activity: a model fit, a simulation, a literature review, a dose recommendation.

The header records:

  • Activity and the decision it informs.
  • Consequence of a wrong answer, and how likely one is, as in the Validation Checklist’s risk assessment.
  • Author: the person or agent doing the work, and who directs the agent if there is one.

Then one row per kind from Section 3:

Validation plan, one row per kind
# Kind Applies? Tool Depth Checked by Done
1 Question
2 Study design
3 Causal assumptions
…
13 Use

Depth takes one of the Validation Checklist’s levels: spot check, review, rerun step by step, or independent reproduction. Every row is kept, and a kind that does not apply says so with a reason, so that two plans can be compared and an empty row is not read as a check that passed.

5. Gaps, Ranked

Ranked by how often the gap is likely to let a wrong answer through, which is a judgment until Section 7 step 2 gives evidence.

  1. Data assembly checklist (3.5). Data errors pass every later check.
  2. Model evaluation checklist (3.8). Shared with the skills project.
  3. Whether a study already done can support the model (3.2).
  4. Whether the assay measures the model’s quantity (3.4).
  5. A causal diagram as a standing step (3.3). The checklist exists; the gap is that no plan asks for it, so confounding is considered only when someone thinks of it.
  6. A benchmarking procedure (3.9): which simple model, which held-out data, what difference counts.
  7. Work done by an agent (3.7), already under way for code in its own document. The taxonomy found studies measuring how often AI-written PMX code fails, and no review procedure.
  8. Checking claims against sources (3.6), as a tool someone outside this repository can use, including on an agent’s summaries.
  9. The question after the work, and its use (3.1, 3.13).
  10. Written reports (3.11).

6. Open Questions

  1. Is “validation” the right word for all thirteen kinds? Regulatory and engineering usage separates verification (was it built right) from validation (is it the right thing). The taxonomy marks kinds 5, 6, 7 and 12 as verification, assay (4) as both, and the rest as validation. Whether the plan should use those terms is still open.
  2. Does depth scale with the author? For people it does: a senior associate’s model gets a lighter review than an intern’s. For an agent there is no track record per kind of task yet. The validation plans themselves, once kept, would build one.
  3. Can an agent do some of the checking? A second agent reviewing the first may share its blind spots. Question 1 of the agent-written code document asks the same for code; the question is wider here.
  4. Are thirteen kinds the right number? Study design and assay were added on 2026-10-21. Missing candidates include safety and regulatory consistency (does the analysis agree with the label and with earlier submissions), and the author’s understanding, which the internship’s final presentation checks for a person.
  5. What prior art covers this? The taxonomy lists it. The ASME V&V 40 standard sets the depth of credibility evidence for a computational model by its risk, the same move as the Validation Checklist’s planning section, and the ICH M15 guideline on model-informed drug development, final in January 2026, uses a similar framework for PMX models. Neither has been read here.
  6. Does every model need a causal diagram? A diagram for a population PK model, where the question is not causal, may be wasted time; a diagram for an exposure-response model usually is not. The lighter rule is that every plan considers one and says why when it skips it. Step 2 will show which rule holds up.

7. Steps

  1. Read ASME V&V 40 and ICH M15 against Section 3, and mark them in the validation taxonomy, which serves as this project’s reference list.
  2. Fill in a validation plan for one past activity, the Model Based TDP worked comparison, and one for the next internship project. Record what each check caught.
  3. Re-rank Section 5 from what step 2 found, and open a project for the gap at the top.
  4. Once the template has been used twice, add it to the Validation section of the Pharmacometrics Tools page.
Back to top