Pharmacometrics Modeling Skills: A Proposal
Transforming modeling guidance into a SKILL file that a human or AI assistant can use
Goal. Develop a model evaluation guidance in SKILL files that any human or AI assistant can refer to.
The Challenge with Guidance Documents
A modeling guidance document takes years to write. Once published it is rarely maintained, so it goes stale, and modelers have to remember where the guidance is stored. When its authors move on, the document is often lost.
A skill file is a human-readable Markdown file of instructions that an AI assistant loads before a task. It can be easily stored and maintained in public repositories.
| Guidance documents | Skill files |
|---|---|
| Have to be found and remembered by humans. | Automatically loaded by AI assistant whenever it is needed. |
| Rarely updated after publication | Can be rebuilt regularly by AI from curated sources, and reviewed by people |
| Written from scratch by a committee | Drafted by AI; based upon existing documents |
| Generally tried to a single company and kept internally, though occasionally published. | Works best when stored in a public repository. |
Consensus and Position Pages
Most of modeling practice is already agreed, and a smaller part is argued about. The two are kept in different places so that the argument never holds up the skill.
- The skill holds only the agreed practice, as instructions. For example: “Use a prediction-corrected VPC when dose differs across subjects.”
- A contested topic gets one line in the skill and a link. For example: “Practice differs on stepwise covariate selection versus full-model approaches; see the position page and ask the modeler which to use.” The assistant asks instead of choosing.
- A position page is written by the people who hold that position. Each view is stated at its strongest, with its references. Position pages are not merged into consensus; when the field settles, the skill’s one line becomes an instruction.
Review Rules
The AI can always write more than people can read, so rules limit what anyone is asked to read.
- Each skill has a length limit, a few hundred lines. A new instruction replaces an old one or fits within the limit.
- Reviewers read a change list. Every proposed change carries a one-line reason and a citation, and the list is ranked; a review covers the top ten, and the rest waits for the next round.
- A change merges after two weeks unless an editor objects. An objection does not block the change indefinitely: the topic moves to a position page. Apache projects call this rule lazy consensus.
- Every skill is tested on example tasks. Each task lists the checks a good run should report. Reviewers check that the skill still produces them, which is quicker than rereading its text.
Governance
- Editors. One to start, Andy, who merges changes and rules on objections. A person or two join once the skill is useful to someone else. A larger group, if it comes, is chosen from people who see the need, work well together and respect each other’s thinking.
- Contributors. Anyone, through an issue or a pull request on GitHub.
- Yearly rebuild. An AI assistant rebuilds each skill from the current skill, the curated references and checklists, a review of literature published since the last rebuild, and a second AI pass asking what is missing. The rebuild arrives as a change list under the review rules above.
- Full review, every three years as a first guess. The editors re-read the topics, references and position pages from scratch, and decide whether the set of skills should change.
- Home. pmxskills.com and a public GitHub repository, independent of any one company or tool. ISoP is the natural long-term home if the project outgrows a small group.