How to upskill your use of AI in SMARTbiomed research
2026-06-19
I have some slides, but please interrupt with questions/comments
Opening poll — how do you use AI today?
Tip
Ground rule
AI accelerates the work. You are responsible for it.
A conditional probability model over text
\[P(\text{next token} \mid \text{text so far})\]
How it’s trained
Why code is a strength
Code is abundant, structured, and correct by execution — a uniquely checkable signal in training data.
sample_next <- function(model, word) {
counts <- model$bigram[[tolower(word)]]
if (is.null(counts)) counts <- model$unigram
sample(names(counts), 1, prob = counts)
}
generate <- function(model, start, n = 20) {
out <- start
for (i in seq_len(n)) {
nxt <- sample_next(model, tail(out, 1))
if (is.na(nxt)) break
out <- c(out, nxt)
}
paste(out, collapse = " ")
}
generate(model, "bird", 10)[1] "bird and myself in the mystery of the dark and a"
Hallucination
The model generates plausible text, not verified truth.
Reasoning models (o3, R1, extended thinking) trade compute for better multi-step work, but the same failure modes apply at higher stakes.
Important
Always verify
Every citation · every number every derivation · every API call
Treat model output as a hypothesis, not a result.
Warning
Quiet deskilling
Outsourcing judgement builds dependency, not expertise. Stay in the loop intentionally.
What the model “sees”
| Typical range | |
|---|---|
| Small / fast models | 8k–32k tokens |
| Standard assistants | 128k–200k tokens |
| Long-context leaders | 1M+ tokens |
A context window is not memory — each new conversation starts fresh.
What counts against it
How to work within limits
.md files persist context across sessionsCLAUDE.md project file that persistsTip
Practical rule
If the conversation is getting long, start fresh — older context degrades response quality before it truncates.
Orient fast in a new subfield · surface candidate methods · anchor literature work
What AI does well here
Important
Big caution: fake citations
The model will generate plausible author names, journal names, and years — that do not exist.
Rule: verify every reference against the real source before citing or acting on it.
Tools
The problem: LLMs answer from training memory → hallucinated citations
The fix: retrieve real documents first, then generate from them
The model cannot fabricate a citation it was never given.
Tip
Tools you may already use
Elicit · Semantic Scholar · NotebookLM are RAG systems — they retrieve real papers before generating any answer.
ragnarBuild your own pipeline to chat with a PDF corpus
Pass chunks into your ellmer chat as context — the model answers from your documents, not from memory.
Move up the tiers as the task grows in scope and complexity.
Tier 1 Chat interfaces
ChatGPT · Claude · Gemini · Copilot Chat
Tier 2 In-IDE assistants
GitHub Copilot · RStudio Copilot ellmer · chattr · gander · chatlas
gander knows your data frame’s columns — context-aware suggestionsTier 3 Coding agents
Claude Code · Codex · Copilot · Pi.dev · NanoClaw
Tip
Suggested on-ramp: start with copilot in RStudio/vscode. It is the lowest-friction step up from copy-pasting to the chat window.
What makes agents different
DESCRIPTION, existing tests, READMEHuman-in-the-loop is not optional
Workflow
You: "Write a function coverage() that
computes CI coverage, with tests"
Agent: reads repo → writes function
→ writes tests → runs tests
→ iterates on failures
→ proposes diff
You: review diff → run tests yourself
→ commit if happy
Warning
The agent proposes; you decide. Never let an agent push to main without a human reviewing the diff.
Beyond Claude Code and Codex, three open-source agents are worth knowing.
| Pi.dev | Hermes | NanoClaw | |
|---|---|---|---|
| Made by | Earendil (MIT) | Nous Research (MIT) | NanoCo (MIT) |
| Interface | CLI | CLI + messaging apps | Messaging apps |
| Model choice | 15+ providers, swap mid-session | 300+ models via Nous Portal | Claude (Claude Code native) |
| Local / private | Yes | Configurable | Yes — Docker-isolated containers |
| Persistent memory | Project instructions file | Yes — learns project context | Per-session containers |
| Standout feature | Extensible via TypeScript packages | Runs across Slack · Telegram · Discord | ~15 source files — fully auditable |
The key axes: model lock-in · privacy · interface familiarity
Warning
All agents call external APIs by default
Running locally ≠ processing locally. Check which model endpoint your agent is calling — data still leaves your machine unless you use a self-hosted model.
Agents are only as reliable as the foundations they work in. These are the skills that make agent output reviewable and reversible.
| Foundation | Why it matters for agents |
|---|---|
| git | every change is a diff you can inspect and revert |
| Unit tests | the objective signal the agent iterates against |
| Terminal / IDE | you run and inspect the output yourself |
| Reproducible envs | renv, Docker, seeds — results don’t change between machines |
| CI | automated checks catch regressions across commits |
Tip
Practical priority
If you only do two things:
Everything else helps, but these two give you a safety net for everything the agent touches.
Agents live in the terminal and work with plain text. A little fluency here pays outsized dividends.
Key skills
Navigate a project directory from the command line
Read and write plain text files (.md, .R, .py, .txt)
Understand what a project root looks like:
my-project/
├── R/ # functions
├── analysis/ # scripts
├── tests/
├── README.md # what is this project?
├── CLAUDE.md # instructions for the agent
└── renv.lock # reproducible packagesMarkdown files that guide agents
| File | Purpose |
|---|---|
README.md |
human-facing project overview |
CLAUDE.md / agents.md |
standing instructions for the AI agent |
skills.md |
reusable task templates |
Tip
A well-written CLAUDE.md is worth hours of re-prompting — the agent reads it at the start of every session.
The five operations you need
| Command | What it does |
|---|---|
git commit |
checkpoint — a snapshot you can return to |
git branch |
parallel work — experiment without risk |
git diff |
what changed — what you review before committing |
git revert |
undo a commit cleanly |
| pull request | propose changes for review on GitHub/GitLab |
Why git makes agents safe
An agent proposes work as diffs you review and commit — always visible, always reversible.
Workflow with an agent:
agent writes → you diff → you test
→ you commit → repeat
If anything looks wrong: git checkout -- . or git revert puts you back exactly where you were.
A test is a function that runs your code on a known input and asserts the expected output. It turns “I think this is right” into a pass/fail signal.
In R with testthat:
Test-driven development (TDD):
The agent iterates against your tests — without tests, it has no objective criterion.
For statisticians: what to test
Tip
Tests are not bureaucracy — they are the contract between you and the code.
Prompt quality determines output quality.
A rigorous prompt:
“Write an R function
coverage(truth, lower, upper)that computes the empirical coverage probability of a set of confidence intervals.truthis a scalar,lowerandupperare equal-length numeric vectors. Return the proportion of intervals that containtruth. Validate thatlowerandupperhave equal length and thatlower <= upperelement-wise. Return a scalar between 0 and 1.”
Generated output:
Important
Generated code is a hypothesis, not a result.
Read it · understand it · test it.
Would you submit a simulation result without checking the DGM?
What made this prompt good?
Vague prompt → vague output
Tests make the agent’s loop self-correcting.
library(testthat)
test_that("coverage() is correct", {
# Correctness — worked out by hand
expect_equal(coverage(0, c(-1, -1), c(1, 1)), 1) # all cover
expect_equal(coverage(0, c( 1, 2), c(3, 4)), 0) # none cover
expect_equal(coverage(0, c(-1, 1), c(1, 2)), 0.5) # half cover
# Boundary — closed endpoint counts
expect_equal(coverage(1, 1, 2), 1)
# Validity — bad input rejected
expect_error(coverage(0, c(1, 2), 3))
})The TDD loop with an agent:
Write tests → agent writes function
→ tests FAIL (Red)
→ agent revises
→ tests PASS (Green)
→ you review diff and commit
Tip
The test numbers are yours — you worked them out by hand. The agent cannot fake correctness against truth you define.
The question:
How do the bias and variance of a doubly-robust GLM estimator compare to non-DR estimators when both the outcome model and the exposure model are misspecified?
Why this is the stress test
A DR estimator is consistent if either nuisance model is correct. Breaking both models removes the safety net entirely.
Estimand: marginal causal contrast
\[\tau = \mathbb{E}[Y(1)] - \mathbb{E}[Y(0)]\]
Binary outcome, binary exposure, confounders Z.
Tool: stdReg2::standardize_glm_dr() (Sachs, Sjölander, Gabriel, Ohlendorff & Brand)
Competing estimators
| Estimator | Outcome model | PS model |
|---|---|---|
| DR standardisation | misspecified | misspecified |
| Non-DR standardisation | misspecified | — |
| IPW | — | misspecified |
DGM: confounder Z with nonlinear X|Z and Y|X,Z — linear working models are wrong by construction.
Performance: bias · variance · RMSE · CI coverage · Monte Carlo SEs
ADEMP (Morris, White & Crowther, Stat Med 2019): a principled framework for simulation design.
| Element | What you decide | What AI accelerates |
|---|---|---|
| Aims | Compare DR vs non-DR under dual misspecification | Drafts study skeleton and runner |
| Data-generating | Confounder Z; nonlinear X\|Z and Y\|X,Z so linear working models are wrong |
Writes the simulator — you verify it hits the true estimand |
| Estimand | Marginal contrast \(\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]\), known from DGM | Computes the true value numerically |
| Methods | DR standardize_glm_dr (both models misspecified) vs non-DR / IPW |
Wires up the competing method calls |
| Performance | Bias, variance, RMSE, CI coverage + Monte Carlo SEs | Codes the metrics — you insist on the MCSEs |
Toolkit: SimDesign · rsimsum · future/furrr · set.seed()
Lazy prompt:
“Write me a simulation study comparing doubly-robust and non-doubly-robust estimators.”
What you get:
Warning
Garbage in → convincing-looking garbage out. The model will produce something — it will look complete and plausible.
Rigorous ADEMP-structured prompt:
“Design a simulation study using ADEMP. Aim: compare
stdReg2::standardize_glm_dr()against non-DR standardisation and IPW for estimating \(\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]\). DGM: binary Y, binary X, continuous confounder Z; use nonlinear relationships so linear working models are misspecified for both outcome and PS. Estimand: marginal risk difference. Methods: the three above, all using misspecified models. Performance: bias, empirical variance, RMSE, nominal-95% CI coverage — all with Monte Carlo SEs. UseSimDesignwithset.seed(42).”
What you get:
AI speeds the mechanics. Judgement stays human.
Where AI helps
rsimsumImportant
Reproduce the headline numbers
Before accepting AI-assisted output, compute the key numbers yourself — in a fresh script, from the raw results.
If the numbers disagree, investigate.
Warning
Quiet deskilling
If the AI always explains the output, you stop reading it directly. Stay engaged — use AI to check your interpretation, not replace it.
Where AI helps
AI reporting guideline checklist:
| Guideline | Scope |
|---|---|
| TRIPOD+AI | prediction model development & validation |
| PROBAST-AI | prediction model risk-of-bias assessment |
| DECIDE-AI | decision support / clinical evaluation |
| CONSORT-AI | randomised trials with AI interventions |
Important
The methodologist guarantees correctness
AI can improve clarity and structure. It cannot guarantee the methods are correct for your research question.
That is your job.
Tip
Practical tip
Ask the model to evaluate your methods section against TRIPOD+AI checklist items — it will surface gaps you missed.
Privacy
Reproducibility
Verification
Disclosure
Tip
A one-paragraph “AI use statement” at submission is becoming standard practice. Write it as you go, not at the end.
1. Install gander
Try it in RStudio on a real dataset. Ask it to write one function using your actual column names.
Lowest friction entry point into in-IDE AI assistance.
2. Draft a simulation with ADEMP
Write out the five ADEMP elements in plain text first. Then paste them into a structured prompt. Verify the DGM hits the true estimand.
The structure is the value — not just for AI.
3. Set one team ground rule
Privacy: “No patient-level data in public chat interfaces.”
Disclosure: “Log the prompts used in any analysis.”
A shared norm beats individual policy every time.
Think–pair–share prompt:
Name one task you would hand to AI tomorrow — and one task you would never hand to AI.
Share with a neighbor (2 min), then we’ll hear a few pairs.
Future seminar series — help us plan it
We are scoping a series on mastering the technical skills of AI use in biomedical research. Tell us what you want:
Open floor — questions and comments welcome.

SMARTbiomed Annual Review