The Bitter Lesson for Pharmacometricians
The Bitter Lesson
Ethan Mollick’s recent post on The Bitter Lesson recounts Rich Sutton’s observation that across many domains (e.g. chess, language translation, computer vision, protein folding), general methods that learn from data and scale with computation have repeatedly overtaken methods built on handcrafted expert knowledge. Claude created a music video that describes the lesson well.
Stockfish, the strongest chess engine, replaced its human-designed evaluation of a position with a neural network evaluation that was trained on positions from games that chess programs played against themselves. It still uses a search algorithm that humans designed and tuned.
In considering where we, as pharmacometricians, focus our efforts in the coming years, it’s worth keeping The Bitter Lesson in mind. I’ve been attracted to the idea of creating skill files and agentic work flows for model building and evaluation, but if other fields serve as a guide, then it could very well be in a year or two, AI can efficiently build a model and write a report without any guidance from us. Mollick admits he fell prey to the Bitter Lesson, and I think I did too: in March I listed templates and skill files as a place to focus. Whether that effort pays off depends on three separate questions:
- Can AI do the task without expert guidance? Increasingly, yes.
- Does expert guidance still improve the result? Possibly, even when the AI can work alone.
- Is the improvement worth the cost of writing and maintaining the guidance?
The third question is the practical one, and its answer can change with each model release. One way to answer it is to give the same model-building tasks to Claude working alone, Claude with a curated skill file, and Claude working with an experienced pharmacometrician, then compare model quality, diagnostic coverage, time and cost. My guess is that guidance still helps on hard, ambiguous problems today, and that the advantage shrinks as models improve. If so, where can we add the most long-term value?
I posit that the following items will hold their value, because they sit where work passes from people to an AI and back. I don’t assume we will stay better than AI at any of them, and AI may already scope better than we do. What the three share is that a person stays accountable at each handoff: for what the work is meant to answer, and for the decision it supports.
- Scoping before handoff. What has to be decided, written down and checked before work goes to an AI. Thinking carefully about this is the difference between agentic engineering and vibe coding.
- Validation after delivery. How to validate and take responsibility for AI-generated deliverables (e.g. code, modeling report, or decision recommendation).
- A human-curated list of the best references. The sources a person has read and vouches for as high quality, as opposed to whatever a search returns.
Scoping
Agents already help with scoping in software development. The brainstorming step in Superpowers interviews you one question at a time and writes the design for you, and GitHub’s Spec Kit has you define what is to be built and why before deciding how. AI may already be better at helping us scope than we are at scoping alone, though it may be worth tailoring tools like those above to drug development activities. For a modeling analysis, a good starting point is the Context of Use table I adapted from the FDA’s credibility assessment framework (Kuemmel et al. 2019). It records the question the model addresses, how much the decision depends on the model, the risk that the model leads to a wrong decision, and the consequence if it does.
Validation
Validation isn’t only asking whether the model is right. It’s asking whether the model, and the conclusions that come from the analysis, lead to a good decision.
AI will get better at checking work, and a second agent reviewing the first will become routine. But AI can be misaligned, and that problem is far from being solved. Even when an AI recommends a decision, a person has to sign off on it. In drug development, where a model can inform the dose a patient receives and put a patient’s life at risk, that responsibility can’t be handed off.
Validation can be supported by checklists, such as this one for R code and our cheat sheet and checklist for model uncertainty assessment, which assesses the uncertainty in a model’s predictions in a structured way. The uncertainty cheat sheet is meant to be used twice: during scoping, to put effort on the parts of the model that drive the decision, and again at the end, to check that the model is fit to inform it. Further thought as to how to validate AI deliverables is warranted.
References
An agent can find and rank sources, and eventually it may judge the literature better than we do. But curating a short list of the best references doesn’t take long. It narrows what a general AI needs to read, and it records that a person read and evaluated each source. The best way to build and maintain such a list is likely together with an agent.
Deskilling and Apprenticeship
Validation only works if the validator can tell a flawed model from a good one. If AI does most of the model building, that ability may fade. And if new pharmacometricians only ever review AI-built models, and never build their own, they may never develop a strong intuition. Furthermore, if AI eventually judges models better than we do, human judgment becomes a weaker check over time. Maintaining human expertise may be one of the big challenges we face going forward.
Where This Is Heading
AI tools are improving quickly, and the more AI can do, the more our value as pharmacometricians lies in deciding what the work is for and in taking responsibility for the decisions it supports.
Addendum: Datasets for Benchmarking
The International Society of Pharmacometrics (ISoP) AI/ML Special Interest Group discussion on 14 October 2026 raised something I left out: development of a database of curated, realistic datasets for evaluating and benchmarking different approaches.
Building that database may be a challenge, but the SIG has started one in its PMX benchmarks repository.
Addendum: Theory-Crafting
20 October 2026.
Grant Sanderson’s video on the future of mathematics and Terence Tao’s essay Mathematics in the Age of AI suggest another place to add value: theory-crafting, which supports validation. Sanderson makes the case that even if AI gets very good at proofs, there are large opportunities in building theory, and Tao argues that a proof no human can properly explain should be viewed as incomplete.
Similarly, even after a pharmacometrician has a good model that characterizes the data well and can simulate to inform predictions, there is often more that can be done. A model may contain additional insight. For instance, a mathematical analysis of the model may indicate the most relevant parameters (e.g. how clearance is generally the most important parameter for characterizing average concentration).
With AI to derive and simulate, math is now much cheaper and easier to do, and so the development of further pharmacometrics theory is well within reach. It would also help with validation, since a person who understands why a model behaves as it does is better placed to tell a flawed model from a good one and to understand and explain the results.