PsychScribe Local

The Problem

Clinical documentation eats a private-practice psychiatrist’s day, and the cloud “AI scribe” tools that promise relief route patient audio and notes through someone else’s servers. For a solo clinician handling sensitive psychiatric sessions, strict local/offline operation isn’t a nice-to-have — it’s the precondition for using the tool at all.

What I Built

PsychScribe Local is a macOS desktop concept for capturing clinical sessions and producing clinician-reviewed notes, with strict offline operation as the default. The shared architecture is a local-first clinical scribe for physicians and therapists; the first specialty pack is psychiatry, optimized for medication-management and therapy visits.

The repo is further along than a spec scaffold on the app-slice track: PsychScribeCore is a real Swift package whose test suite passes against real GRDB 6.29.3 and WhisperKit 0.18.0, and a SwiftUI macOS app builds and runs an Import → Transcribe → Review → Draft → Export slice end to end. Live chunked microphone capture is a compiling stub; file import is the working capture path for v0. Transcript clinical accuracy is explicitly unvalidated until a large-v3 benchmark run on target M1 hardware — the review gate described below is the safety boundary until then.

Architecture

One imported audio file traces through seven stages, all local:

1. Import — AudioImporter reads the source file; live mic capture goes through SessionRecorder/ChunkCrypto but recording itself is a stub in v0, so import is the working entry point.
2. Chunked transcription — ChunkedTranscriptionPipeline reads the file in fixed 30-second windows with a 3-second trailing overlap (mono 16 kHz Float32) rather than decoding the whole file into RAM, because a single long transcribeFile call was measured at ~3.8 MB/min of decoded audio with no bound for multi-hour sessions. Each window is transcribed in isolation by WhisperKitTranscriber, so only one window plus one temp file exist in memory/disk at a time regardless of total session length.
3. Stitching — TranscriptStitcher reconciles the overlapping regions between adjacent windows by timestamp proximity and text similarity into one TranscriptDocument in a provisional state.
4. Transcript review — the clinician edits segment text and marks the transcript approved in the Transcript Review UI. ReviewGateEngine.canExport blocks everything downstream — note generation and export — until transcript.state == .approved.
5. Note drafting — once approved, OllamaNoteGenerator.draftNote sends the transcript and a NoteTemplate to a local Ollama server (localhost:11434, OpenAI-compatible /v1/chat/completions, falling back to /api/chat) running llama3.2:3b, with a system prompt that forbids stating anything not present in the transcript and requires [clinician to confirm] placeholders for anything the model can’t source.
6. Deterministic safety flagging — ClinicalFactFlagger, a regex/lexicon scanner with no LLM involved, runs twice: once over the raw transcript, once over the LLM-drafted note text. It flags every medication name, dose/frequency pattern, diagnosis cue, risk/SI/HI/self-harm phrase, safety-plan mention, and level-of-care phrase as a pending ReviewGate. This is the actual export gate: ReviewGateEngine blocks Markdown/TXT export until every gate is resolved (a dev-only override exists, labeled as such in the UI) — the LLM drafts, the deterministic scanner is what enforces the AGENTS.md hard rule that clinical content requires human review before it leaves the app.
7. Export — MarkdownExporter (v0) or TranscriptTextExporter writes an ExportArtifact to local disk; no network call is made at this stage. PDF export is planned, not yet implemented.

Audio is encrypted at rest throughout via ChunkCrypto (AES-GCM/CryptoKit, symmetric key in the macOS Keychain, custom authenticated file format PSENC002), decrypted only transiently in memory for transcription. Structured records (Project, Encounter, TranscriptDocument, ClinicalNote) persist through GRDBClinicalStore — GRDB 6 over plain SQLite, one JSON-blob column per row keyed by id (encounter-indexed for transcripts/notes). SQLCipher (DB-at-rest encryption) is deferred to a named follow-up hardening step (ADR 0005); v0 accepts plain SQLite because only synthetic, non-PHI fixtures are used — the higher-sensitivity artifact, audio, is already encrypted independently.

A separate track, tools/asr-bench, benchmarks WhisperKit models against synthetic non-PHI fixtures (WER, medication-term recall, real-time factor) to decide which model is clinically viable for the accuracy pass — the first real benchmark run, openai_whisper-tiny over 6 fixtures, measured 52.8% avg WER and 48.1% avg term recall, missing most medication names and dose values, which is why the app default stays tiny for dev/demo and large-v3 is the named target for a real accuracy pass, not yet run on target hardware.

flowchart LR
  A["Imported audio file<br/>(mic capture is a stub)"] --> B["ChunkCrypto<br/>AES-GCM, Keychain key"]
  B --> C["ChunkedTranscriptionPipeline<br/>30s windows, 3s overlap"]
  C --> D["WhisperKitTranscriber<br/>tiny default -> base/small/large-v3 toggle"]
  D --> E["TranscriptStitcher<br/>overlap reconciliation"]
  E --> F[("GRDBClinicalStore<br/>SQLite, JSON blobs")]
  E --> G["Transcript Review UI<br/>clinician edits + approves"]
  G --> H{"ReviewGateEngine<br/>blocks until approved"}
  H --> I["OllamaNoteGenerator<br/>llama3.2:3b @ localhost:11434"]
  I --> J["ClinicalFactFlagger<br/>deterministic regex/lexicon scan<br/>meds, doses, dx, SI/HI, safety-plan"]
  J --> K["Note Review UI<br/>resolve pending gates"]
  K --> L{"ReviewGateEngine<br/>blocks export until gates resolved"}
  L --> M["MarkdownExporter / TranscriptTextExporter<br/>local disk, no network"]
  N["tools/asr-bench<br/>WER / term-recall / RTF harness"] -.picks model for.-> D

Models — which, where, why:

Stage	Model / tier	Where it runs	Why there
Transcription	WhisperKit; default openai_whisper-tiny, with a UI toggle to base / small / large-v3	On-device, WhisperKitTranscriber (Argmax WhisperKit 0.18.0)	Native Apple Silicon Whisper port; tiny is the dev/demo default and the only model benchmarked so far — the first real run missed most medication names and doses, confirming large-v3 is required before any clinical use
Note drafting + fact extraction	Ollama llama3.2:3b	Local Ollama server, localhost:11434, OpenAI-compatible /v1/chat/completions (falls back to /api/chat) — OllamaNoteGenerator uses URLSession only, no SDK	Keeps note generation off any network; a 3B model fits the M1/16 GB target machine; the system prompt bans stating anything not present in the transcript
Clinical safety flagging	Models: none — deterministic regex/lexicon scan	ClinicalFactFlagger, in-process, run over both the transcript and the LLM-drafted note	Medication/dose/diagnosis/risk/SI-HI/safety-plan detection has to be reliable and auditable, not probabilistic — this is the actual export-blocking check, independent of whatever the LLM drafted

Tools & infra — which, why:

• Storage — GRDB 6 over SQLite for structured records (Project/Encounter/TranscriptDocument/ClinicalNote, JSON-blob columns); plain SQLite in v0, not SQLCipher, because only synthetic non-PHI fixtures exist during prototyping — DB-at-rest encryption is a tracked next step (ADR 0005).
• Audio encryption — CryptoKit AES-GCM with the symmetric key held in the macOS Keychain (Security framework), a custom authenticated file format (PSENC002) versioned past an earlier PSENC001; audio is the highest-sensitivity artifact so it’s encrypted independently of the DB-at-rest gap above.
• Local LLM runtime — Ollama, reached over plain HTTP at localhost:11434; no vendor SDK, just URLSession against the OpenAI-compatible chat endpoint.
• ASR engine — WhisperKit (Argmax) 0.18.0 as a Swift package dependency; downloads the selected model once, then runs fully offline.
• App shell — native SwiftUI macOS app, XcodeGen-generated Xcode project (xcodegen generate + xcodebuild) — chosen over the Tauri + React alternative weighed in the tech-stack direction doc, in favor of direct WhisperKit/Keychain/AVFoundation integration.
• Benchmark harness — tools/asr-bench, a standalone Swift executable scoring WhisperKit models against synthetic (non-PHI) fixtures via WERScorer / TermRecallScorer / BenchMetrics (WER, medication-term recall, real-time factor, peak RSS).
• Hosting — none; the app is 100% local with no cloud deploy. The repo’s “Cloud Boundary” policy requires explicit opt-in, a signed BAA, and documented data flow before any cloud ASR/LLM/telemetry vendor can be added.

How it works

1. The clinician imports a session audio file (live chunked mic recording exists as a stub, not the working path yet).
2. ChunkedTranscriptionPipeline reads the file in 30-second windows with a 3-second overlap, transcribing each window through WhisperKitTranscriber (default openai_whisper-tiny) so memory use stays bounded regardless of session length.
3. TranscriptStitcher reconciles the overlapping regions by timestamp and text similarity into one provisional TranscriptDocument.
4. The clinician reviews and edits the transcript in the Transcript Review UI and marks it approved; ReviewGateEngine blocks every step past this point until that approval is set.
5. OllamaNoteGenerator sends the approved transcript plus a psychiatry note template to a local Ollama server (llama3.2:3b) and gets back a structured ClinicalNote draft, with sections that cite transcript spans or are marked [clinician to confirm].
6. ClinicalFactFlagger deterministically re-scans both the transcript and the drafted note for medication names, doses/frequencies, diagnosis cues, SI/HI/self-harm/safety-plan language, and level-of-care phrases, producing a pending ReviewGate for each hit.
7. The clinician resolves every pending gate in the Note Review UI; ReviewGateEngine.canExport only returns allowed: true once the transcript is approved, all gates are resolved, and consent wasn’t declined (a labeled dev-only override exists to unblock prototype work).
8. MarkdownExporter or TranscriptTextExporter writes the approved note to local disk (Markdown/TXT in v0; PDF planned) — no network call happens at export.
9. Independently, tools/asr-bench runs WhisperKit models against synthetic fixtures to measure WER, medication-term recall, and real-time factor, informing which model is promoted to the default before any real clinical use is considered.

Tech stack

SwiftUI macOS app (XcodeGen project) plus a shared PsychScribeCore Swift package consumed by both the app and the asr-bench harness. WhisperKit (Argmax, 0.18.0) for on-device transcription. Ollama (llama3.2:3b, local HTTP, OpenAI-compatible endpoint) for note drafting — no cloud LLM. GRDB 6 over SQLite for structured storage; CryptoKit AES-GCM plus macOS Keychain for audio-at-rest encryption. ClinicalFactFlagger is a from-scratch deterministic regex/lexicon scanner, not a library or a model. No cloud services, no hosting — the app runs entirely on-device.

Locked Defaults

• Platform — macOS desktop first (iPhone/iPad later), sized for an M1 MacBook Pro running back-to-back visits across a full clinical day.
• PHI posture — strict local/offline by default; audio retained encrypted only until the transcript and note are approved, then deleted.
• Clinician in the loop — notes are drafted for review, never auto-filed.

Status and Boundaries

The repo pairs a product/specification scaffold (design docs, ADRs, compliance planning) with a compiling, tested app-slice prototype (see Architecture above). Neither makes it clinical software: it is not HIPAA certified, transcript accuracy is unvalidated on target hardware, and it must not be used with real patient data until the compliance-evidence package and clinical-validation gates are complete. The portfolio entry documents the product direction and the privacy-first architecture, not a shipping medical device.