From methods development to the clinic

How to upskill your use of AI in SMARTbiomed research

Michael Sachs

smartbiomed.dk

2026-06-19

Warmup

I have some slides, but please interrupt with questions/comments

Opening poll — how do you use AI today?

  • Not yet
  • Writing (emails, methods text, …)
  • Coding (autocomplete, chat)
  • Methods & analysis
  • Other

The pipeline

The pipeline view

Discover
orient · literature · methods
Develop
code · simulate · test
Validate
check · diagnose · explain
Translate
write · report · tailor
Govern · privacy · reproducibility · verification · disclosure

Tip

Ground rule

AI accelerates the work. You are responsible for it.

How LLMs work

What is an LLM? A statistician’s view

A conditional probability model over text

\[P(\text{next token} \mid \text{text so far})\]

  • Text is chopped into tokens (~¾ word each)
  • A context window is the max tokens it sees at once
  • Temperature \(\tau\) controls sampling sharpness: \(\tau \to 0\) → greedy; \(\tau \to \infty\) → uniform

How it’s trained

  1. Next-token prediction on massive text corpora
  2. Instruction tuning — follow prompts, be helpful
  3. RLHF / RLAIF — human (or AI) preference ranking

Why code is a strength

Code is abundant, structured, and correct by execution — a uniquely checkable signal in training data.

Exercise - Build a lLM in 5 minutes

  • Find a corpus to train the model and tokenize:
## Works of Edgar Allan Poe, Volume I
corpus <- readLines("https://www.gutenberg.org/cache/epub/2147/pg2147.txt")
tokens <- unlist(tokenizers::tokenize_words(corpus))
  • Train a Markov model
bigram  <- lapply(
    split(tokens[-1], tokens[-length(tokens)]),
    table
  )
unigram <- table(tokens)

model <- list(bigram = bigram, unigram = unigram)

Continued

  • Sample from the model
sample_next <- function(model, word) {
  counts <- model$bigram[[tolower(word)]]
  if (is.null(counts)) counts <- model$unigram
  sample(names(counts), 1, prob = counts)
}

generate <- function(model, start, n = 20) {
  out <- start
  for (i in seq_len(n)) {
    nxt <- sample_next(model, tail(out, 1))
    if (is.na(nxt)) break
    out <- c(out, nxt)
  }
  paste(out, collapse = " ")
}

generate(model, "bird", 10)
[1] "bird and myself in the mystery of the dark and a"

Discussion

  • What LLM concepts does this illustrate?
  • How can this model be improved?
  • What else can it be used for?

Why they’re confident — and sometimes wrong

Hallucination

The model generates plausible text, not verified truth.

  • Fabricated citations that look real (always verify)
  • Confident wrong code that compiles but is subtly incorrect
  • Invented API arguments or package functions

Reasoning models (o3, R1, extended thinking) trade compute for better multi-step work, but the same failure modes apply at higher stakes.

Important

Always verify

Every citation · every number every derivation · every API call

Treat model output as a hypothesis, not a result.

Warning

Quiet deskilling

Outsourcing judgement builds dependency, not expertise. Stay in the loop intentionally.

The context window and memory

What the model “sees”

Typical range
Small / fast models 8k–32k tokens
Standard assistants 128k–200k tokens
Long-context leaders 1M+ tokens

A context window is not memory — each new conversation starts fresh.

What counts against it

  • Your prompt
  • The entire conversation so far
  • Any files or code you paste in
  • Tool call results

How to work within limits

  • Chunking: split long documents; summarise sections before sending
  • System prompts: keep standing instructions short and reuse across sessions
  • External memory: project .md files persist context across sessions
  • Agent memory: tools like Claude Code maintain a CLAUDE.md project file that persists

Tip

Practical rule

If the conversation is getting long, start fresh — older context degrades response quality before it truncates.

Stage 1: Discover & frame

Discover & frame

Orient fast in a new subfield · surface candidate methods · anchor literature work

What AI does well here

  • Rapid orientation: “explain doubly-robust estimation to a statistician”
  • Surfacing related methods and their trade-offs
  • Summarising abstracts, structuring a literature map
  • Drafting search strings for PubMed / Web of Science

Important

Big caution: fake citations

The model will generate plausible author names, journal names, and years — that do not exist.

Rule: verify every reference against the real source before citing or acting on it.

Tools

  • Chat interfaces for exploration
  • Elicit · Semantic Scholar for literature search

Concept: Retrieval Augmented Generation

The problem: LLMs answer from training memory → hallucinated citations

The fix: retrieve real documents first, then generate from them

  1. Retrieve — embed the query; find the closest documents in a vector store
  2. Augment — paste the retrieved text into the context window
  3. Generate — model answers are grounded in that text, not in weights

The model cannot fabricate a citation it was never given.

Tip

Tools you may already use

Elicit · Semantic Scholar · NotebookLM are RAG systems — they retrieve real papers before generating any answer.

RAG in R: ragnar

Build your own pipeline to chat with a PDF corpus

library(ragnar)
store <- ragnar_store_create("papers.db")
ragnar_store_insert(store, ragnar_read("mypaper.pdf"))
ragnar_store_build_index(store)

chunks <- ragnar_retrieve(store, "What endpoints were used?")

Pass chunks into your ellmer chat as context — the model answers from your documents, not from memory.

Stage 2: Develop

Three tiers of tools

Move up the tiers as the task grows in scope and complexity.

Tier 1 Chat interfaces

ChatGPT · Claude · Gemini · Copilot Chat

  • Paste code, ask questions
  • One-shot generation
  • No access to your full repo
  • Good for: explaining, drafting snippets, quick checks

Tier 2 In-IDE assistants

GitHub Copilot · RStudio Copilot ellmer · chattr · gander · chatlas

  • Autocomplete in your editor
  • gander knows your data frame’s columns — context-aware suggestions
  • Good for: writing functions, transforming data, iterative coding

Tier 3 Coding agents

Claude Code · Codex · Copilot · Pi.dev · NanoClaw

  • Repo-aware — reads your whole project
  • Can run code, read errors, iterate
  • Work toward a goal you set
  • Good for: multi-file tasks, test-driven development, simulation scaffolding

Tip

Suggested on-ramp: start with copilot in RStudio/vscode. It is the lowest-friction step up from copy-pasting to the chat window.

Coding agents: from autocomplete to delegate

What makes agents different

  • Repo-aware: they read your DESCRIPTION, existing tests, README
  • Tool-use: they can run R/Python, call the shell, read error messages
  • Iterative: they retry when tests fail — the loop is automated
  • Transparent: all changes come as a diff you review and commit

Human-in-the-loop is not optional

  • Read every diff before accepting
  • Run your own checks — don’t trust “tests pass” without verifying
  • Commit deliberately: each commit is a checkpoint you can revert to

Workflow

You: "Write a function coverage() that
      computes CI coverage, with tests"

Agent: reads repo → writes function
       → writes tests → runs tests
       → iterates on failures
       → proposes diff

You:  review diff → run tests yourself
      → commit if happy

Warning

The agent proposes; you decide. Never let an agent push to main without a human reviewing the diff.

Open agent alternatives

Beyond Claude Code and Codex, three open-source agents are worth knowing.

Pi.dev Hermes NanoClaw
Made by Earendil (MIT) Nous Research (MIT) NanoCo (MIT)
Interface CLI CLI + messaging apps Messaging apps
Model choice 15+ providers, swap mid-session 300+ models via Nous Portal Claude (Claude Code native)
Local / private Yes Configurable Yes — Docker-isolated containers
Persistent memory Project instructions file Yes — learns project context Per-session containers
Standout feature Extensible via TypeScript packages Runs across Slack · Telegram · Discord ~15 source files — fully auditable

Choosing an agent for research work

The key axes: model lock-in · privacy · interface familiarity

  • Sensitive data, must stay local → NanoClaw or Pi.dev (both run on your machine; no data leaves unless you call an external API)
  • Want to switch between Claude, GPT-4o, Gemini → Pi.dev or Hermes (both support multiple providers; useful when one model is better at a specific task)
  • Lab group already lives in Slack or Telegram → Hermes (the agent joins your existing channels; no new interface to learn)
  • Want to understand exactly what the agent is doing → NanoClaw (~15 source files, written to be read)

Warning

All agents call external APIs by default

Running locally ≠ processing locally. Check which model endpoint your agent is calling — data still leaves your machine unless you use a self-hosted model.

Stage 2: Foundations

What coding agents stand on

Agents are only as reliable as the foundations they work in. These are the skills that make agent output reviewable and reversible.

Foundation Why it matters for agents
git every change is a diff you can inspect and revert
Unit tests the objective signal the agent iterates against
Terminal / IDE you run and inspect the output yourself
Reproducible envs renv, Docker, seeds — results don’t change between machines
CI automated checks catch regressions across commits

Tip

Practical priority

If you only do two things:

  1. Put your project in a git repo
  2. Write at least a few unit tests

Everything else helps, but these two give you a safety net for everything the agent touches.

Text-based computer skills

Agents live in the terminal and work with plain text. A little fluency here pays outsized dividends.

Key skills

  • Navigate a project directory from the command line

  • Read and write plain text files (.md, .R, .py, .txt)

  • Understand what a project root looks like:

    my-project/
    ├── R/              # functions
    ├── analysis/       # scripts
    ├── tests/
    ├── README.md       # what is this project?
    ├── CLAUDE.md       # instructions for the agent
    └── renv.lock       # reproducible packages

Markdown files that guide agents

File Purpose
README.md human-facing project overview
CLAUDE.md / agents.md standing instructions for the AI agent
skills.md reusable task templates

Tip

A well-written CLAUDE.md is worth hours of re-prompting — the agent reads it at the start of every session.

git: the safety net that makes agents safe

The five operations you need

Command What it does
git commit checkpoint — a snapshot you can return to
git branch parallel work — experiment without risk
git diff what changed — what you review before committing
git revert undo a commit cleanly
pull request propose changes for review on GitHub/GitLab

Why git makes agents safe

An agent proposes work as diffs you review and commit — always visible, always reversible.

Workflow with an agent:

agent writes → you diff → you test
→ you commit → repeat

If anything looks wrong: git checkout -- . or git revert puts you back exactly where you were.

Unit tests: making “correct” checkable

A test is a function that runs your code on a known input and asserts the expected output. It turns “I think this is right” into a pass/fail signal.

In R with testthat:

test_that("my_function is correct", {
  expect_equal(my_function(input), known_output)
  expect_error(my_function(bad_input))
})

Test-driven development (TDD):

  1. Red — write a failing test for the behaviour you want
  2. Green — write the code that makes it pass
  3. Refactor — clean up; tests still pass

The agent iterates against your tests — without tests, it has no objective criterion.

For statisticians: what to test

  • Correctness: numbers you can work out by hand (simple DGMs, closed-form results)
  • Boundary conditions: edge cases, empty inputs, length-one vectors
  • Statistical properties: symmetry, monotonicity, bounds on estimators
  • Reproducibility: same seed → same result

Tip

Tests are not bureaucracy — they are the contract between you and the code.

Illustration (1/2) — Ask: write the function

Prompt quality determines output quality.

A rigorous prompt:

“Write an R function coverage(truth, lower, upper) that computes the empirical coverage probability of a set of confidence intervals. truth is a scalar, lower and upper are equal-length numeric vectors. Return the proportion of intervals that contain truth. Validate that lower and upper have equal length and that lower <= upper element-wise. Return a scalar between 0 and 1.”

Generated output:

coverage <- function(truth, lower, upper) {
  stopifnot(length(lower) == length(upper))
  stopifnot(all(lower <= upper))
  mean(lower <= truth & truth <= upper)
}

Important

Generated code is a hypothesis, not a result.

Read it · understand it · test it.

Would you submit a simulation result without checking the DGM?

What made this prompt good?

  • Named the function and arguments
  • Specified the return type
  • Specified validation behaviour
  • Specified the estimand (proportion)

Vague prompt → vague output

Illustration (2/2) — Test it: validity and correctness

Tests make the agent’s loop self-correcting.

library(testthat)

test_that("coverage() is correct", {
  # Correctness — worked out by hand
  expect_equal(coverage(0, c(-1, -1), c(1,  1)), 1)   # all cover
  expect_equal(coverage(0, c( 1,  2), c(3,  4)), 0)   # none cover
  expect_equal(coverage(0, c(-1,  1), c(1,  2)), 0.5) # half cover

  # Boundary — closed endpoint counts
  expect_equal(coverage(1, 1, 2), 1)

  # Validity — bad input rejected
  expect_error(coverage(0, c(1, 2), 3))
})

The TDD loop with an agent:

Write tests → agent writes function
→ tests FAIL (Red)
→ agent revises
→ tests PASS (Green)
→ you review diff and commit

Tip

The test numbers are yours — you worked them out by hand. The agent cannot fake correctness against truth you define.

Worked example: simulation

Double robustness under misspecification

The question:

How do the bias and variance of a doubly-robust GLM estimator compare to non-DR estimators when both the outcome model and the exposure model are misspecified?

Why this is the stress test

A DR estimator is consistent if either nuisance model is correct. Breaking both models removes the safety net entirely.

Estimand: marginal causal contrast

\[\tau = \mathbb{E}[Y(1)] - \mathbb{E}[Y(0)]\]

Binary outcome, binary exposure, confounders Z.

Tool: stdReg2::standardize_glm_dr() (Sachs, Sjölander, Gabriel, Ohlendorff & Brand)

Competing estimators

Estimator Outcome model PS model
DR standardisation misspecified misspecified
Non-DR standardisation misspecified
IPW misspecified

DGM: confounder Z with nonlinear X|Z and Y|X,Z — linear working models are wrong by construction.

Performance: bias · variance · RMSE · CI coverage · Monte Carlo SEs

Build it with ADEMP — and let AI accelerate

ADEMP (Morris, White & Crowther, Stat Med 2019): a principled framework for simulation design.

Element What you decide What AI accelerates
Aims Compare DR vs non-DR under dual misspecification Drafts study skeleton and runner
Data-generating Confounder Z; nonlinear X\|Z and Y\|X,Z so linear working models are wrong Writes the simulator — you verify it hits the true estimand
Estimand Marginal contrast \(\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]\), known from DGM Computes the true value numerically
Methods DR standardize_glm_dr (both models misspecified) vs non-DR / IPW Wires up the competing method calls
Performance Bias, variance, RMSE, CI coverage + Monte Carlo SEs Codes the metrics — you insist on the MCSEs

Toolkit: SimDesign · rsimsum · future/furrr · set.seed()

The skill is in the prompt

Lazy prompt:

“Write me a simulation study comparing doubly-robust and non-doubly-robust estimators.”

What you get:

  • Generic DGM with no meaningful misspecification
  • No estimand stated
  • No performance measures beyond bias
  • No Monte Carlo SEs
  • Code you cannot easily verify

Warning

Garbage in → convincing-looking garbage out. The model will produce something — it will look complete and plausible.

Rigorous ADEMP-structured prompt:

“Design a simulation study using ADEMP. Aim: compare stdReg2::standardize_glm_dr() against non-DR standardisation and IPW for estimating \(\mathbb{E}[Y(1)]-\mathbb{E}[Y(0)]\). DGM: binary Y, binary X, continuous confounder Z; use nonlinear relationships so linear working models are misspecified for both outcome and PS. Estimand: marginal risk difference. Methods: the three above, all using misspecified models. Performance: bias, empirical variance, RMSE, nominal-95% CI coverage — all with Monte Carlo SEs. Use SimDesign with set.seed(42).”

What you get:

  • Structured, reviewable code
  • Correct estimand and DGM
  • MCSEs included automatically
  • Something you can actually critique

Stages 3–4 + Govern

Validate & analyse

AI speeds the mechanics. Judgement stays human.

Where AI helps

  • Interpreting R/Python output: “what does this warning mean?”
  • Running and explaining model diagnostics
  • Explaining a method back to you in plain language (a surprisingly useful cross-check)
  • Suggesting additional sensitivity analyses
  • Formatting simulation results with rsimsum

Important

Reproduce the headline numbers

Before accepting AI-assisted output, compute the key numbers yourself — in a fresh script, from the raw results.

If the numbers disagree, investigate.

Warning

Quiet deskilling

If the AI always explains the output, you stop reading it directly. Stay engaged — use AI to check your interpretation, not replace it.

Translate & communicate — to the clinic

Where AI helps

  • Draft and refine methods sections
  • Tailor language for a clinical vs. statistical audience
  • Respond to reviewer comments (draft, then edit — never send raw)
  • Structure grant text against a funder’s criteria
  • Check a clinical prediction model report against guidelines

AI reporting guideline checklist:

Guideline Scope
TRIPOD+AI prediction model development & validation
PROBAST-AI prediction model risk-of-bias assessment
DECIDE-AI decision support / clinical evaluation
CONSORT-AI randomised trials with AI interventions

Important

The methodologist guarantees correctness

AI can improve clarity and structure. It cannot guarantee the methods are correct for your research question.

That is your job.

Tip

Practical tip

Ask the model to evaluate your methods section against TRIPOD+AI checklist items — it will surface gaps you missed.

Govern: use it responsibly

Privacy

  • Never paste patient-level data into public chat interfaces
  • Use approved / on-premise deployments for sensitive data (check with your IT/data protection officer)
  • GDPR applies to identifiable data in prompts — when in doubt, ask before pasting

Reproducibility

  • Log the prompts you used for any analysis — treat them like code
  • Record tool versions (model ID, package version)
  • Your methods section should be reproducible from prompts + code + seed

Verification

  • Every citation — verify against the real source
  • Every number — reproduce independently
  • Every derivation — check the algebra yourself
  • Every API call — check the documentation

Disclosure

  • Most journals now require disclosure of AI use in writing or analysis
  • ICMJE: authors are responsible for all AI-assisted content
  • Check your target journal’s policy before submission

Tip

A one-paragraph “AI use statement” at submission is becoming standard practice. Write it as you go, not at the end.

Closing

Three things to try this week

1. Install gander

Try it in RStudio on a real dataset. Ask it to write one function using your actual column names.

Lowest friction entry point into in-IDE AI assistance.

2. Draft a simulation with ADEMP

Write out the five ADEMP elements in plain text first. Then paste them into a structured prompt. Verify the DGM hits the true estimand.

The structure is the value — not just for AI.

3. Set one team ground rule

Privacy: “No patient-level data in public chat interfaces.”

Disclosure: “Log the prompts used in any analysis.”

A shared norm beats individual policy every time.

Think–pair–share prompt:

Name one task you would hand to AI tomorrow — and one task you would never hand to AI.

Share with a neighbor (2 min), then we’ll hear a few pairs.

Future seminar series — help us plan it

We are scoping a series on mastering the technical skills of AI use in biomedical research. Tell us what you want:

  • Topic: What skill or tool would make the biggest difference to your work right now? (git · unit testing · RAG · coding agents · simulation frameworks · clinical reporting guidelines · other)
  • Format: Hands-on workshop · lecture + Q&A · live coding demo · reading group
  • Speaker: Is there someone — inside or outside SMARTbiomed — whose practice you would like to learn from?
  • Level: Absolute beginner · already using AI but want to go deeper · want to build pipelines from scratch

Thank you — let’s discuss

Open floor — questions and comments welcome.