Build your own AI-assisted personal health record

open forum
1

Build your own AI-assisted personal health record

A write-up of something I’ve been doing for a few months as a bit of an experiment. This ‘How-to’ requested by @Alan_Hassey @mike.bainbridge @nick.booth @JW148 on yesterday’s Virtual Mabel's call, but I thought - why not put in in a public place for all to use?

The idea: collect every scrap of your own medical record into one folder on your computer, then let an LLM read, organise, and find patterns in it. Everything stays on your machine.

You’ll be surprised at how astute the LLM can be when it has access to lots of facts about your history. I haven’t noticed any significant factual inaccuracies with this approach. I theorise that LLM hallucination is more likely to occur when there is an absence of factual information, and LLMs have to make up the gaps with bluff in order to please you.

What you need (one-off, ~20 min)

  1. Install VS Code

  2. Install Claude Code - install the VS Code extension from the Extensions panel (search “Claude Code”), or run the installer from that page. You will need a Claude subscription (Pro, at about £20 a month, is fine to start with).

  3. Privacy: In your Claude account, got to Settings > Privacy and switch OFF the default-on setting ‘Allow the use of your chats and coding sessions to train and improve Anthropic AI models’.

  4. Make one folder for everything, e.g. Documents/health. In VS Code: File > Open Folder and pick it. Open the Claude panel (the Claude icon in the sidebar). Claude can now read and write inside that folder, and only that folder.

The workflow

  1. Collect all your medical data. Drag in PDFs of hospital letters, discharge summaries, clinic letters, blood results, scan reports, optician prescriptions, vaccination/COVID records, occupational-health and pre-employment serology certs. Screenshots and photos of paper letters are fine too.

  2. Get the records you don’t have yet. Most of your history sits in NHS and private systems. Ask Claude to draft a Subject Access Request (UK GDPR Article 15) to each GP practice, trust, and clinic you have used. Ask for imaging studies, not just the reports. Using the Claude Browser Extension you can easily automate extraction of data even when it’s in a hideous web UI (like the NHS App)

  3. Ask Claude to organise it. It will read the PDFs, extracts the facts, and writes plain Markdown files you can read in any editor forever. A first prompt of anything like this will work:

Read every file in this folder.
Transcribe each PDF letter into a structured Markdown file next to the original.
Don't delete anything.
Then build me a current problem list, medication list, and a chronological blood-results table

  1. You decide the rules! You can go for a completely chronological record - or a Problem Oriented Record, or some hybrid approach. You can have any kinds of summary documents you want - medication lists

  2. Mine for insight. Now the record is structured, ask questions across it: “Plot my blood pressure trend”, “What immunity is actually documented vs inferred?”, “Which results have drifted over ten years?”, “What’s missing that I should chase?” This is where it stops being a filing cabinet and becomes a tool.

Hints and tips

  • Keep originals. The LLM writes a structured copy alongside each source PDF, never instead of it. Every derived fact stays traceable to its source.

  • Beware complex clinical calculators. Simple scoring tools (eg BMI) will likely be accurate. However for more complex ones, (eg QRISK2-3) I would not trust that the LLM has the ability to calculate it accurately. Web-based calculators might be something it can use via the Browser Extension. (This highlights the need for a suite of fast, LLM-friendly command-line clinical calculators, which I am working on [1] [2])

  • Backup continuously. Ideally to some kind of secure, private cloud - for example Dropbox or Google Drive, using an automatic sync tool.

  • It seems to be quite good at rare diseases. Due to the breadth of its training data, it is quite good at pointing out the possibility of a rare condition, even when experienced clinicians haven’t ‘connected the dots’. Clinicians simply can’t know about every possible rare genetic cause of a given symptom. LLMs naturally just do.

Things to try

  • Ask for a condensed list of the most important key medical facts about you - the ‘In Case Of Emergency’ wallet summary - and see how well it does.
  • Ask it to ask you questions to fill in any gaps or inconsistencies that it considers to be present.
  • Ask it for recommendations for things that need to be addressed at your next few GP consultations!

I’ve been surprised by how easy it’s been to accrue the basics of a comprehensive record, and actually what you start to realise is that even in this very rudimentary form, it’s still more comprehensive than anyone else’s medical record about you! Even your GP probably doesn’t have as complete a record, and they absolutely certainly don’t have it all at their fingertips, easy to browse, read, and search. AVTs? waste of a decent LLM.

2

A Claude Skill to create an evidence review to inform specific health questions

3

That’s great stuff. Thanks @spdegabrielle

4

Have you gone down the Hermes-Agent rabbit hole yet? I have resisted at present (due to my already stretched personal context window :sweat_smile: ) but seems like it would be possible to create a personalised, local agent that could work well with this.

5

I haven’t gone down the custom agent route, and it is my belief following this kind of experimentation that it’s not about the agent or the model (as long as you have some recent-ish model, anything GPT5+ or Claude Sonnet 4.5+ and upwards).

It’s about the context and information base. Garbage in = garbage out.

Get a high quality knowledge base about your conditions and history in one easily accessible (to you) and secure place. For me that’s a folder on my (encrypted SSD) laptop, synced to GDrive (2FA).

6

Do you encrypt before uploading ?

There have been reports of people irrevocably losing access to their data on cloud storage due to automated processes.

7

It’s HTTPS obvs so encrypted in transit. I don’t separately encrypt it before uploading, it’s synced using a third party service, and encryption would make it much harder to use. On Gdrive it is (probably) encrypted at rest, but if an attacker gained access to my Google account then they would be able to see it all. That doesn’t worry me at this stage, I think the risk is exceedingly low.

I am working on something better than this though - the whole ‘folder full of medical stuff’ was a POC experiment. We need something that would have encryption at rest and in transit and has signatures and provenance and a timeline.

I’ve made a fair bit of progress on GitEHR this year. The CLI is getting to near demoable. The website is very WIP and needs me to review it all and clarify the message a bit. But take a look, it’s happening.

8

Sounds good! It can all be run fully offline using local LLMs like Qwen 3.6. LM studio will run a server with an API endpoint you can call the same as you would Claude or OpenAI.

The interesting things you could do with a local agent would be around the fact it can co-ordinate the inputs and create a structured memory. Think of things like setting it up to receive sleep data from wearables which it automatically updates in a database. Send it photos of a meal which it adds to calorie tracker/dietary monitoring. Can also be linked to your calendar - e.g. recommend an exercise plan to improve x and arrange these workouts to fit in with my schedule over the next 4 weeks. They can also parse pdfs and documents automatically as it’s ingested so data like blood test results could be extracted and stored in a structured way.

9

Some really great ideas and yes, local models make this even better from a privacy POV.

Extracting data from wearables is likely possible, especially if they sync to an account which has an API. I suspect some might have this, but many won’t because the data they have on you is part of their business model, and lock-in is critical to that.

On the plus side, if there is a portal then these days that’s as good as an API, you use Claude for Chrome and get it to go get all the data and give you it in JSON format and it will. When I tried this with blood tests spread across the usual janky 12 pages in the NHS App, it worked first time and every test was correctly extracted.

This is I think where it might start getting less reliable. “There’s no art to find the meal’s construction in the photo” so to speak.

LLMs have quite simple limits

If you ask your LLM what it can reliably do it will be quite honest, I’ve found. I asked Claude Opus 4.7 how it ‘knew’ whether an NHS number was valid and it responded that the algorithm is small and widely published, and it’s just about computable in an LLM without hallucination. I tested it and it did get them all right, but perhaps for something a little more gnarly like a CHI number, it might make errors.

I asked if Claude ‘knew’ all of SNOMED by heart (it would likely have been in the training data since copyright and SNOMED’s licence terms only apply to ordinary people, not Anthropic) and it said no, it felt that asking for “give me a SNOMED term for this free text clinical entity” would result in high hallucination rates.

So for anything more complex than NHS number validation, we need to give it tools.

Build all the CLI tools!

What we need to do is give LLMs better tools. They can use CLI tools very quickly and efficiently - much less context is wasted compared to APIs or GUIs or even MCPs.

CLI tools for SNOMED - ie sct.
CLI tools for clinical calculations like QRISK3…
CLI tools for centile values (working on the CLI, but the API is done)
CLI tools for FHIR may be of help in enabling a better interface with patient portals maybe (but TBH I think will be of limited value)

(Arguably these are the same tools that we should have built and deployed as GUIs to clinicians decades ago, but there was no massive enthusiasm for it back then. We’ve been negotiating for a decade to still not have QRISK3 available in the GP systems.)

The good news it that we can have both. If the CLI backend works then you can put a decent GUI on the front for clinicians to use.