Medical RAG: DICOM, HIPAA, and Clinical Data Sovereignty

Why Medical RAG Must Be On-Premise

Every other industry discussed in this technical library treats on-premise deployment as a best practice. In healthcare, it is a legal and ethical floor, not a ceiling. The moment patient data—protected health information, or PHI—leaves a controlled environment, the liability calculus changes entirely. Cloud-hosted medical RAG is not a cost-saving optimization. It is a compliance violation waiting to be discovered.

HIPAA Is Not a Suggestion

The Health Insurance Portability and Accountability Act (HIPAA) establishes national standards for protecting individually identifiable health information. The Privacy Rule governs how PHI is used and disclosed. The Security Rule governs how electronic PHI (ePHI) is safeguarded. Both apply to covered entities—hospitals, clinics, insurers, clearinghouses—and their business associates.

A RAG system that ingests clinical documents, embeddings them into a vector database, and generates responses from them is processing ePHI. There is no carve-out for "AI research" or "internal tooling." If the system touches patient data, it is governed by HIPAA. Period.

The consequences of non-compliance are severe and tiered:

These are per-violation figures. A single breach exposing 10,000 patient records is not one violation—it is 10,000 violations. The numbers compound rapidly. Beyond financial penalties, covered entities face mandatory breach notification requirements: affected individuals must be notified within 60 days, HHS must be notified, and if the breach affects more than 500 residents of a state, media notification is required. The reputational damage to a hospital or health system is often more consequential than the fine.

Why Cloud Hosting Fails the Compliance Test

Cloud-hosted medical RAG introduces failure modes that cannot be engineered away through configuration:

• Multi-tenancy risk: Cloud providers share physical infrastructure across customers. While logical isolation exists, the hypervisor and the cloud provider's staff represent attack surfaces you do not control. A misconfigured IAM policy, a compromised cloud engineer, or a zero-day in the virtualization layer can expose patient data across tenant boundaries.
• Data residency ambiguity: Cloud regions replicate data for durability. A patient record ingested in a US-East data center may be replicated to a region in a jurisdiction with different privacy laws. HIPAA does not permit this ambiguity for ePHI without explicit business associate agreements covering every region.
• Sub-processor sprawl: A cloud-based RAG pipeline touches the embedding model service, the vector database, the LLM inference endpoint, the object storage, the logging service, and potentially the model fine-tuning service. Each is a sub-processor. Each requires a business associate agreement. Each is a potential breach vector.
• Law enforcement access: Cloud providers respond to law enforcement requests for stored data. A hospital that believes its patient records are private may discover they were disclosed under a legal process applied to the cloud provider, not the hospital itself. On-premise systems require a warrant directed at the hospital, which has legal counsel to challenge it.
• Vendor lock-in as a compliance risk: If a cloud RAG vendor changes its terms of service, experiences an outage, or discontinues a product, the hospital's ability to migrate patient data out is constrained by the vendor's export capabilities and migration timeline—not the hospital's needs.

The core principle is sovereignty: the entity that holds the data must be the entity legally accountable for it, with physical and logical control over the infrastructure. Cloud-hosted medical RAG divides accountability across multiple parties, none of whom have full control, and all of whom can point to someone else when something goes wrong. On-premise deployment restores a single chain of accountability: the hospital owns the hardware, the software, the data, and the liability. That clarity is what HIPAA was designed to enforce.

DICOM Image Ingestion

Medical imaging is not JPEGs with different file extensions. The Digital Imaging and Communications in Medicine (DICOM) standard is a comprehensive protocol that defines both the file format for medical images and the network protocol for transmitting them between devices. A single DICOM file contains the pixel data (the actual image) and a rich header of metadata describing the patient, the study, the series, the equipment, and the acquisition parameters. Understanding this dual nature is essential for building RAG systems that can retrieve and reason about imaging data.

What DICOM Files Contain

DICOM files are produced by virtually every medical imaging modality in clinical use:

• Computed Tomography (CT): Cross-sectional X-ray images reconstructed into slices. CT scans produce detailed density maps of tissue, bone, and contrast agents. A single CT study may contain hundreds of axial slices, each a separate DICOM instance.
• Magnetic Resonance Imaging (MRI): Images produced by aligning hydrogen atoms in the body with magnetic fields and radio pulses. MRI excels at soft tissue differentiation. Studies include T1-weighted, T2-weighted, and FLAIR sequences, each with distinct contrast characteristics.
• Radiography (X-ray): Projection images—chest X-rays, extremity views, mammograms. Single images or small series, typically stored as 16-bit grayscale.
• Ultrasound: Real-time imaging using sound wave reflections. DICOM ultrasound files may include cine loops (multi-frame images) stored as a single multi-frame instance.
• Nuclear Medicine (PET, SPECT): Functional imaging showing metabolic activity. Often co-registered with CT for anatomical context (PET-CT fusion studies).
• Fluoroscopy: Continuous X-ray imaging used during procedures. Stored as time-stamped frame sequences.

Each DICOM file follows a hierarchical structure: Patient → Study → Series → Instance. A patient may have multiple studies (e.g., a baseline CT and a follow-up CT three months later). Each study contains one or more series (e.g., axial, coronal, and sagittal reconstructions). Each series contains instances (individual slices or frames). This hierarchy is not cosmetic—it maps directly to how clinicians think about and query imaging data.

Parsing DICOM with pydicom

The pydicom library is the standard Python tool for reading and manipulating DICOM files. It parses the DICOM header into a Python object where each data element is accessible by its tag keyword. Here is a practical ingestion pipeline that extracts the metadata needed for RAG retrieval:

import pydicom
from pydicom.dataset import Dataset
from pathlib import Path
import json
from datetime import datetime

# --- DICOM Metadata Extractor ---

DICOM_FIELDS = {
    'patient_id':          (0x0010, 0x0020),  # Patient ID
    'patient_name':        (0x0010, 0x0010),  # Patient's Name
    'patient_birth_date':  (0x0010, 0x0030),  # Birth Date
    'study_date':          (0x0008, 0x0020),  # Study Date
    'study_time':          (0x0008, 0x0030),  # Study Time
    'study_description':   (0x0008, 0x1030),  # Study Description
    'study_instance_uid':  (0x0020, 0x000D),  # Study Instance UID
    'series_description':  (0x0008, 0x103E),  # Series Description
    'modality':            (0x0008, 0x0060),  # Modality
    'body_part_examined':  (0x0018, 0x0015),  # Body Part Examined
    'manufacturer':        (0x0008, 0x0070),  # Manufacturer
    'institution_name':    (0x0008, 0x0080),  # Institution Name
    'accession_number':    (0x0008, 0x0050),  # Accession Number
    'slice_thickness':     (0x0018, 0x0050),  # Slice Thickness
    'kvp':                 (0x0018, 0x0060),  # KVP
    'contrast_bolus_agent':(0x0018, 0x0010),  # Contrast/Bolus Agent
    'image_rows':          (0x0028, 0x0010),  # Rows
    'image_cols':          (0x0028, 0x0011),  # Columns
    'pixel_spacing':       (0x0028, 0x0030),  # Pixel Spacing
}


def extract_dicom_metadata(dicom_path: str) -> dict:
    """Extract structured metadata from a DICOM file for RAG indexing."""
    ds = pydicom.dcmread(dicom_path, stop_before_pixels=True)

    record = {}
    for field_name, tag in DICOM_FIELDS.items():
        try:
            value = ds[tag].value
            # Convert pydicom data types to JSON-serializable types
            if hasattr(value, 'original_string'):
                value = str(value)
            elif isinstance(value, (pydicom.uid.UID,)):
                value = str(value)
            elif field_name == 'pixel_spacing':
                value = [float(v) for v in value]
            record[field_name] = value
        except KeyError:
            record[field_name] = None

    # Format study date from DICOM format (YYYYMMDD) to ISO
    if record.get('study_date'):
        try:
            d = datetime.strptime(record['study_date'], '%Y%m%d')
            record['study_date_iso'] = d.strftime('%Y-%m-%d')
        except ValueError:
            record['study_date_iso'] = record['study_date']

    return record


def generate_text_description(metadata: dict) -> str:
    """Generate a natural-language description for embedding.

    This text is what the embedding model vectorizes. It should capture
    enough clinical context that semantic search can retrieve relevant
    studies without examining pixel data.
    """
    parts = []
    modality = metadata.get('modality', 'unknown modality')
    body_part = metadata.get('body_part_examined', 'unspecified region')
    study_desc = metadata.get('study_description', 'no study description')
    series_desc = metadata.get('series_description', 'no series description')
    study_date = metadata.get('study_date_iso', 'unknown date')

    parts.append(f"{modality} imaging study of the {body_part}")
    parts.append(f"performed on {study_date}")
    parts.append(f"Study description: {study_desc}")
    parts.append(f"Series: {series_desc}")

    if metadata.get('contrast_bolus_agent'):
        parts.append(f"Contrast agent: {metadata['contrast_bolus_agent']}")

    if metadata.get('slice_thickness'):
        parts.append(f"Slice thickness: {metadata['slice_thickness']}mm")

    if metadata.get('kvp'):
        parts.append(f"Tube voltage: {metadata['kvp']}kVp")

    parts.append(f"Image dimensions: {metadata.get('image_rows', '?')}x"
                 f"{metadata.get('image_cols', '?')} pixels")

    return '. '.join(parts) + '.'


def ingest_dicom_directory(dicom_dir: str) -> list[dict]:
    """Walk a directory of DICOM files and produce RAG-ready records."""
    records = []
    for path in Path(dicom_dir).rglob('*.dcm'):
        try:
            meta = extract_dicom_metadata(str(path))
            meta['text_for_embedding'] = generate_text_description(meta)
            meta['file_path'] = str(path)
            meta['ingested_at'] = datetime.now().isoformat()
            records.append(meta)
        except Exception as e:
            print(f"Failed to parse {path}: {e}")
            continue
    return records

Note the stop_before_pixels=True argument in dcmread. For metadata extraction and embedding generation, you do not need the pixel data loaded into memory. DICOM files for CT studies can be 512KB each, and a single study may have 300+ slices. Loading pixel data for every file during ingestion is unnecessary and memory-intensive. When the RAG system needs to display the actual image (for a clinician reviewing retrieved studies), it reads the pixel data on demand.

Web-Based DICOM Viewing with Cornerstone.js

Clinicians need to see the images a RAG system retrieves, not just text descriptions. The cornerstone.js family of libraries provides a browser-based DICOM viewer that runs entirely client-side. No image data is sent to an external rendering service. The viewer runs on the hospital's internal web server and renders DICOM pixel data using WebGL. Key libraries in the ecosystem:

• cornerstone-core: The rendering engine. Handles window/level adjustment, zoom, pan, and annotations on 2D medical images.
• cornerstone-wado-image-loader: Decodes DICOM files in the browser using WebAssembly. Supports JPEG, JPEG 2000, and raw pixel data transfer syntaxes.
• cornerstone-tools: Measurement and annotation tools—length, angle, ROI, probe, elliptical ROI, arrow annotate, and more.
• cornerstone-math: Geometric calculations for measurements.
• dicom-microscopy-viewer: For whole-slide imaging (pathology slides).

A minimal integration that displays a DICOM image fetched from an internal Orthanc server:

<!-- HTML -->
<div id="dicomViewer" style="width:512px;height:512px;position:relative;"></div>

<script type="module">
import cornerstone from 'https://hospital-internal.cdn/cornerstone-core.min.js';
import dicomImageLoader from 'https://hospital-internal.cdn/cornerstone-wado-image-loader.min.js';

// Initialize the rendering engine
cornerstone.enable(document.getElementById('dicomViewer'));

// Load a DICOM image from the internal Orthanc PACS server
// The URL points to the hospital's own DICOM web endpoint
const imageUrl =
    'wadouri:https://pacs.internal.hospital/dicom/wado?requestType=WADO' +
    '&studyUID=' + studyUID +
    '&seriesUID=' + seriesUID +
    '&objectUID=' + objectUID +
    '&contentType=application/dicom';

cornerstone.loadAndCacheImage(imageUrl).then(function(image) {
    const viewport = cornerstone.getDefaultViewportForCanvas(
        document.getElementById('dicomViewer').querySelector('canvas'),
        image
    );
    cornerstone.displayImage(
        document.getElementById('dicomViewer'),
        image,
        viewport
    );
}).catch(function(err) {
    console.error('DICOM load failed:', err);
});
</script>

This viewer is embedded in the RAG query interface. When a clinician searches for similar studies and the system retrieves matching DICOM series, each result card includes a thumbnail rendered by Cornerstone. Clicking the thumbnail opens a full viewer with windowing controls, scroll through slices, and measurement tools. All rendering happens in the browser. The pixel data travels from the Orthanc server to the clinician's workstation over the hospital's internal network. It never touches the public internet.

Generating Text Descriptions for Embedding

The critical design decision in DICOM RAG is how to make images searchable via text. Embedding models operate on text or on images, but medical RAG queries are often text-based ("Show me CT scans of the chest with similar findings"). There are two approaches:

Metadata-driven text embedding (practical, immediate): The function generate_text_description above constructs a natural-language summary from DICOM header fields. This summary is embedded using a text embedding model. Queries are also embedded as text. Retrieval matches on textual similarity between the query and the generated descriptions. This approach is deployable today with no specialized medical imaging model required. It captures modality, body region, contrast usage, technique parameters, and study descriptions—enough context for many clinical queries.

Image embedding (research-grade, resource-intensive): A medical image encoder (e.g., a model trained on CheXpert, MIMIC-CXR, or RadImageNet) converts the pixel data directly into an embedding vector. Queries can be either text (via a text encoder aligned to the image encoder during training) or images (an example scan is embedded and nearest-neighbor search finds visually similar studies). This approach requires GPU inference on-premise and a model trained specifically for medical imagery. General-purpose CLIP models perform poorly on medical images because their training data contains almost no radiology.

For most hospital deployments, the metadata-driven approach provides immediate value and can be augmented with image embeddings later as the RAG system matures. The key is that the text description captures clinically meaningful attributes—not just "CT scan" but "CT of the chest with IV contrast, 5mm slice thickness, performed for suspected pulmonary embolism."

Multi-Modal Medical RAG

A clinician's question is rarely answered by a single data type. "Show me all CT scans with findings similar to this one, along with the radiologist's notes" requires the RAG system to cross-reference imaging data with dictated radiology reports. "What were this patient's last three lab results and the corresponding clinical notes from their nephrologist?" requires lab data, clinical notes, and an understanding of temporal relationships. Multi-modal medical RAG unifies these data sources into a single searchable knowledge base.

The Four Data Modalities

Unifying into a Single Vector Space

The challenge of multi-modal RAG is that text from different sources must be embeddable into a comparable vector space. The approach is to normalize each modality into a text representation before embedding:

import json
from datetime import datetime

# --- Multi-Modal Document Normalizer ---

def normalize_dicom_study(meta: dict) -> dict:
    """Convert DICOM metadata into a RAG document."""
    text = (
        f"[IMAGING] {meta.get('modality', 'Unknown')} study of "
        f"{meta.get('body_part_examined', 'unspecified body part')} "
        f"on {meta.get('study_date_iso', 'unknown date')}. "
        f"Study description: {meta.get('study_description', 'N/A')}. "
        f"Series: {meta.get('series_description', 'N/A')}. "
        f"Contrast: {meta.get('contrast_bolus_agent', 'none')}. "
        f"Accession: {meta.get('accession_number', 'N/A')}."
    )
    return {
        'source_type': 'dicom',
        'patient_id': meta.get('patient_id'),
        'study_uid': meta.get('study_instance_uid'),
        'text': text,
        'timestamp': meta.get('study_date_iso'),
        'metadata': meta,
    }


def normalize_clinical_note(note: dict) -> dict:
    """Convert an EHR clinical note into a RAG document."""
    text = (
        f"[CLINICAL NOTE] {note.get('note_type', 'Progress note')} "
        f"by {note.get('author', 'unknown provider')} "
        f"in {note.get('department', 'unknown department')}. "
        f"Date: {note.get('note_date', 'unknown')}. "
        f"Content: {note.get('body', '')}"
    )
    return {
        'source_type': 'clinical_note',
        'patient_id': note.get('patient_id'),
        'text': text,
        'timestamp': note.get('note_date'),
        'metadata': note,
    }


def normalize_voice_dictation(transcript: dict) -> dict:
    """Convert a Whisper-transcribed dictation into a RAG document."""
    text = (
        f"[DICTATION] {transcript.get('dictation_type', 'Radiology report')} "
        f"dictated by {transcript.get('author', 'unknown')}. "
        f"Date: {transcript.get('date', 'unknown')}. "
        f"Transcript: {transcript.get('text', '')}"
    )
    return {
        'source_type': 'voice_dictation',
        'patient_id': transcript.get('patient_id'),
        'text': text,
        'timestamp': transcript.get('date'),
        'metadata': transcript,
    }


def normalize_lab_result(lab: dict) -> dict:
    """Convert a lab result into a RAG document."""
    panels = lab.get('results', [])
    result_lines = []
    for r in panels:
        result_lines.append(
            f"{r.get('test_name', 'Unknown test')}: "
            f"{r.get('value', '?')} {r.get('unit', '')} "
            f"(ref range: {r.get('reference_low', '?')}-"
            f"{r.get('reference_high', '?')}, "
            f"flag: {r.get('abnormal_flag', 'normal')})"
        )
    text = (
        f"[LAB RESULT] Panel: {lab.get('panel_name', 'General')}. "
        f"Date: {lab.get('collected_date', 'unknown')}. "
        + "; ".join(result_lines)
    )
    return {
        'source_type': 'lab_result',
        'patient_id': lab.get('patient_id'),
        'text': text,
        'timestamp': lab.get('collected_date'),
        'metadata': lab,
    }


# --- Unified Indexing ---

def index_medical_document(doc: dict, embed_fn, qdrant_client):
    """Embed and store a normalized medical document in Qdrant.

    The collection is partitioned by patient_id so that document-level
    access control can be enforced at query time.
    """
    vector = embed_fn(doc['text'])

    qdrant_client.upsert(
        collection_name='medical_rag',
        points=[{
            'id': generate_uuid(doc),
            'vector': vector,
            'payload': {
                'source_type': doc['source_type'],
                'patient_id': doc['patient_id'],
                'timestamp': doc['timestamp'],
                'text': doc['text'],
                'metadata': json.dumps(doc['metadata']),
                # Access control: list of authorized user IDs
                'authorized_users': get_care_team(doc['patient_id']),
            }
        }]
    )


def generate_uuid(doc: dict) -> str:
    """Generate a deterministic ID from document source and content."""
    import hashlib
    raw = f"{doc['source_type']}:{doc['patient_id']}:{doc['timestamp']}:{doc['text'][:100]}"
    return str(hashlib.md5(raw.encode()).hexdigest())


def get_care_team(patient_id: str) -> list[str]:
    """Query the EHR for the patient's assigned care team member IDs."""
    # In production, this queries the hospital's EHR/FHIR system
    # Returns a list of provider identifiers authorized to view this patient
    return []  # Placeholder - implement per your EHR

The Multi-Modal Query Workflow

When a clinician asks "Show me all CT scans with findings similar to this one, along with the radiologist's notes," the system executes the following workflow:

1. Query parsing: The query is decomposed into its constituent intents: find similar CT scans (imaging search) and retrieve associated radiology notes (text search by association).
2. Access control check: The clinician's identity determines which patient records they can see. The query is filtered to only search documents where the clinician is in the authorized_users list.
3. Vector search: The query text (or a reference image, if provided) is embedded. Qdrant performs nearest-neighbor search within the authorized subset of the collection.
4. Cross-modal join: Retrieved DICOM studies are joined with clinical notes and dictations that share the same patient_id and fall within a relevant time window (e.g., notes dictated within 48 hours of the study).
5. LLM synthesis: The retrieved context—study metadata, radiologist dictation, relevant lab values—is assembled into a prompt. The local LLM generates a synthesized response that references the specific studies and notes retrieved.
6. Provenance linking: The response includes citations: which DICOM study, which note, which lab result. The clinician can click through to view the original image or read the full note.
7. Audit log entry: The entire interaction—query, retrieved documents, response, clinician identity, timestamp, stated purpose—is written to the audit log.

The power of multi-modal medical RAG is that the clinician does not need to know which system holds the data. They do not need to query the PACS for images, the EHR for notes, the LIS for labs, and the dictation system for reports separately. One query searches across all modalities, and the LLM synthesizes the results into a coherent clinical picture.

Six Security Layers for Medical RAG

A medical RAG system is not secure because it uses encryption. It is secure because defense-in-depth layers multiple controls, each addressing a different attack surface. If one layer fails, the others maintain protection. The following six layers should be implemented in any hospital-deployed RAG system handling PHI. They are ordered from the network boundary inward to the data itself.

Layer 1: Network Isolation

The RAG system must operate on a network segment that has no path to the public internet. There are two configurations:

Air-gapped deployment: The RAG infrastructure (GPU servers, vector database, DICOM ingestion pipeline, LLM inference) is on a physically separate network with no network interface connected to any routable external network. Data ingress is via physical media (encrypted USB drives, serial-attached storage) or a unidirectional data diode that permits one-way transfer of model updates only. This is the highest-security configuration, appropriate for environments where even the risk of outbound data exfiltration is unacceptable.

Isolated VLAN deployment: The RAG system operates on a dedicated VLAN with strict firewall rules. Inbound traffic is permitted only from the hospital's internal clinical network (PACS, EHR, dictation workstations). Outbound traffic to the internet is denied at the firewall with explicit rules. A separate, tightly controlled jump host permits administrators to access the system for maintenance without exposing it broadly. A web proxy with an allowlist can permit specific outbound destinations (e.g., OS package mirrors for security updates) if needed, with full proxy logging.

# Example firewall rules for isolated VLAN (iptables / nftables conceptual)

# RAG VLAN: 10.50.0.0/24
# PACS server: 10.10.0.50
# EHR FHIR endpoint: 10.10.0.80
# Dictation system: 10.10.0.90
# Admin jump host: 10.20.0.5

# Default deny
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]

# Allow inbound from PACS (DICOM ingestion)
-A INPUT -s 10.10.0.50 -d 10.50.0.0/24 -p tcp --dport 4242 -j ACCEPT

# Allow inbound from EHR (FHIR API)
-A INPUT -s 10.10.0.80 -d 10.50.0.0/24 -p tcp --dport 8443 -j ACCEPT

# Allow inbound from dictation system (audio upload)
-A INPUT -s 10.10.0.90 -d 10.50.0.0/24 -p tcp --dport 9000 -j ACCEPT

# Allow inbound from admin jump host (SSH only)
-A INPUT -s 10.20.0.5 -d 10.50.0.0/24 -p tcp --dport 22 -j ACCEPT

# Allow internal VLAN communication (RAG components)
-A INPUT -s 10.50.0.0/24 -d 10.50.0.0/24 -j ACCEPT
-A OUTPUT -d 10.50.0.0/24 -j ACCEPT

# DENY ALL OUTBOUND TO INTERNET (0.0.0.0/0 except internal ranges)
-A OUTPUT -d 10.0.0.0/8 -j ACCEPT
-A OUTPUT -d 172.16.0.0/12 -j ACCEPT
-A OUTPUT -d 192.168.0.0/16 -j ACCEPT
# Everything else falls through to DROP

COMMIT

The firewall rules are enforced at the network switch level, not just on individual hosts. This prevents a compromised server within the RAG VLAN from establishing an outbound connection even if its local firewall is misconfigured.

Layer 2: Authentication & Role-Based Access

Every human interaction with the RAG system requires authenticated identity. Anonymous access is not permitted, even for read-only queries. Authentication is multi-factor: something the user knows (password), something the user has (smart card, hardware token, or TOTP code), and optionally something the user is (biometric, where the hospital has deployed biometric authentication).

Behind authentication is role-based access control (RBAC). The RAG system defines roles that determine what a user can query, what data they can retrieve, and what actions they can perform:

RBAC is enforced at the application layer. The RAG query interface checks the user's role and filters results accordingly. A researcher querying for "CT scans of the chest" receives only de-identified records. A nurse querying for "recent lab results" receives only results for patients on their assigned care team. An administrator has access to system configuration and logs but cannot retrieve clinical data through the RAG interface. This separation of duties is a HIPAA requirement and a sound security principle.

Layer 3: Document-Level Access Control

Role-based access controls what types of data a user can see. Document-level access control (DLAC) controls which specific records they can see. A doctor may have the role to view clinical notes, but only for patients on their assigned care team. A nurse may be authorized to view labs, but only for patients under their direct care.

DLAC is enforced at the vector database query level. When the RAG system issues a similarity search to Qdrant, it includes a filter that restricts results to documents where the current user is in the authorized_users list:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue

def retrieve_with_access_control(
    query_vector: list[float],
    user_id: str,
    user_role: str,
    qdrant: QdrantClient,
    top_k: int = 10,
) -> list[dict]:
    """Retrieve relevant documents, filtered by user access rights."""

    if user_role == 'researcher':
        # Researchers only see de-identified data
        access_filter = Filter(
            must=[
                FieldCondition(
                    key='de_identified',
                    match=MatchValue(value=True),
                ),
            ]
        )
    elif user_role == 'administrator':
        # Administrators cannot retrieve clinical documents
        return []
    else:
        # Clinical staff: only documents where they are authorized
        access_filter = Filter(
            must=[
                FieldCondition(
                    key='authorized_users',
                    match=MatchValue(value=user_id),
                ),
            ]
        )

    results = qdrant.search(
        collection_name='medical_rag',
        query_vector=query_vector,
        query_filter=access_filter,
        limit=top_k,
        with_payload=True,
    )

    return [
        {
            'score': hit.score,
            'source_type': hit.payload['source_type'],
            'text': hit.payload['text'],
            'patient_id': hit.payload['patient_id'],
            'timestamp': hit.payload['timestamp'],
        }
        for hit in results
    ]


# The care team assignment is maintained in the EHR and synced to the
# RAG system's access control list. When a patient is admitted, their
# care team is defined (attending, residents, nurses, consulting
# physicians). When a patient is discharged, the care team is
# dissolved but historical access is retained for the continuity of
# care and for audit purposes.

This means that even if two patients have nearly identical clinical presentations and the vector representations of their records are similar, a doctor who is only on one patient's care team will only retrieve that patient's documents. The other patient's records are invisible at the database query level, not merely hidden in the UI. This is critical: access control must be enforced where the data is queried, not where it is displayed. UI-level filtering can be bypassed; database-level filtering cannot.

Layer 4: Audit Logging

HIPAA requires audit controls that record and examine activity in systems containing ePHI. For a medical RAG system, this means logging every interaction that touches patient data. The audit log is not a debug log or a performance log. It is a compliance record that must answer the question: who accessed what patient data, when, and for what purpose?

import json
from datetime import datetime, timezone
from dataclasses import dataclass, asdict
from typing import Optional


@dataclass
class AuditEntry:
    timestamp: str          # ISO 8601, UTC
    user_id: str            # Authenticated user identifier
    user_role: str          # Role at time of access
    action: str             # 'query', 'retrieve', 'view_image', 'export', 'config_change'
    patient_id: Optional[str]  # Patient accessed (if applicable)
    document_ids: list[str]    # Specific documents retrieved
    query_text: Optional[str]  # The user's query (if action is 'query')
    purpose: str            # Declared purpose: 'treatment', 'education', 'research'
    source_ip: str          # Client IP address
    rag_response_hash: Optional[str]  # Hash of the generated response
    retrieval_count: int    # Number of documents retrieved
    access_denied: bool     # Was access denied by DLAC?


class MedicalRAGAuditLogger:
    def __init__(self, log_path: str, encryption_key: bytes):
        self.log_path = log_path
        self.encryption_key = encryption_key  # For encrypting log entries

    def log(self, entry: AuditEntry):
        """Write an audit entry. Entries are append-only and encrypted."""
        record = asdict(entry)
        record['timestamp'] = datetime.now(timezone.utc).isoformat()

        # Encrypt the audit record at rest
        encrypted = self._encrypt(json.dumps(record))

        # Append to the log file (append-only, no modification permitted)
        with open(self.log_path, 'a') as f:
            f.write(encrypted + '\n')

        # Also send to a separate, tamper-evident log server
        # (e.g., via syslog to a dedicated audit log host)
        self._send_to_log_server(record)

    def _encrypt(self, plaintext: str) -> str:
        from cryptography.fernet import Fernet
        f = Fernet(self.encryption_key)
        return f.encrypt(plaintext.encode()).decode()

    def _send_to_log_server(self, record: dict):
        # In production: send via TLS-protected syslog to a
        # dedicated, hardened audit log host that only accepts
        # append operations
        pass


# Usage in the RAG query pipeline:

def execute_medical_query(
    query: str,
    user_id: str,
    user_role: str,
    purpose: str,
    client_ip: str,
    qdrant: QdrantClient,
    embed_fn,
    llm,
    audit: MedicalRAGAuditLogger,
) -> dict:
    """Execute a RAG query with full audit logging."""

    # Log the query attempt
    audit.log(AuditEntry(
        timestamp='',  # Set by logger
        user_id=user_id,
        user_role=user_role,
        action='query',
        patient_id=None,
        document_ids=[],
        query_text=query,
        purpose=purpose,
        source_ip=client_ip,
        rag_response_hash=None,
        retrieval_count=0,
        access_denied=False,
    ))

    # Embed the query
    query_vector = embed_fn(query)

    # Retrieve with access control
    results = retrieve_with_access_control(
        query_vector, user_id, user_role, qdrant
    )

    if not results:
        audit.log(AuditEntry(
            timestamp='',
            user_id=user_id,
            user_role=user_role,
            action='retrieve',
            patient_id=None,
            document_ids=[],
            query_text=None,
            purpose=purpose,
            source_ip=client_ip,
            rag_response_hash=None,
            retrieval_count=0,
            access_denied=True,
        ))
        return {'response': 'No accessible documents found.', 'sources': []}

    # Log each document retrieved
    doc_ids = [r['text'][:50] for r in results]  # Simplified
    patient_ids = list(set(r['patient_id'] for r in results if r['patient_id']))

    audit.log(AuditEntry(
        timestamp='',
        user_id=user_id,
        user_role=user_role,
        action='retrieve',
        patient_id=patient_ids[0] if patient_ids else None,
        document_ids=doc_ids,
        query_text=None,
        purpose=purpose,
        source_ip=client_ip,
        rag_response_hash=None,
        retrieval_count=len(results),
        access_denied=False,
    ))

    # Generate response
    context = '\n\n'.join(r['text'] for r in results)
    prompt = f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:"
    response = llm.generate(prompt)

    import hashlib
    response_hash = hashlib.sha256(response.encode()).hexdigest()[:16]

    audit.log(AuditEntry(
        timestamp='',
        user_id=user_id,
        user_role=user_role,
        action='generate_response',
        patient_id=None,
        document_ids=[],
        query_text=None,
        purpose=purpose,
        source_ip=client_ip,
        rag_response_hash=response_hash,
        retrieval_count=len(results),
        access_denied=False,
    ))

    return {'response': response, 'sources': results}

Audit logs must be tamper-evident. The log file is append-only (filesystem permissions deny write and delete to all users except the logging process). A copy is sent to a separate, hardened log server via TLS-protected syslog. If the RAG system's audit log is tampered with, the discrepancy between the local log and the remote log server reveals the tampering. Logs are retained for a minimum of six years per HIPAA requirements.

Layer 5: Encryption

Encryption protects data that has been accessed despite access controls. It ensures that even if an attacker obtains the storage media, the database files, or intercepts network traffic, the data is unreadable without the encryption keys.

Encryption at rest: All storage—the vector database files, the DICOM image archive, the embedding cache, the LLM model weights, the audit logs—is encrypted using AES-256. This is implemented via full-disk encryption (LUKS on Linux) on all storage volumes. The encryption keys are stored in a hardware security module (HSM) or a TPM on the server. Keys never leave the physical server. If a disk is removed from the server, it is encrypted and unreadable.

Encryption in transit: All network communication within the RAG system uses TLS 1.3. This includes the connection between the clinician's web browser and the RAG query interface, the connection between the ingestion pipeline and the vector database, and the connection between the query interface and the LLM inference server. TLS 1.3 is the minimum acceptable version; TLS 1.2 is permitted only for legacy components that cannot be upgraded, and only with strong cipher suites (ECDHE with AES-256-GCM). Self-signed certificates are generated on the hospital's internal certificate authority. No certificate authority is external.

Backup encryption: Backups are encrypted before they leave the server. The backup encryption key is different from the disk encryption key and is stored separately (ideally on the HSM or in a key management system that is itself backed up to a different physical location). This means that even if an encrypted backup tape is lost or stolen, the data remains protected by the backup encryption layer independent of the disk encryption.

# Full-disk encryption with LUKS (Linux Unified Key Setup)

# 1. Create an encrypted volume on a new disk
sudo cryptsetup luksFormat /dev/sdb1 \
    --cipher aes-xts-plain64 \
    --key-size 512 \
    --hash sha256 \
    --iter-time 5000

# 2. Open the encrypted volume (prompts for passphrase or reads from TPM)
sudo cryptsetup luksOpen /dev/sdb1 rag_storage

# 3. Create a filesystem on the decrypted volume
sudo mkfs.ext4 /dev/mapper/rag_storage

# 4. Mount for RAG data (vectors, DICOM archive, embeddings)
sudo mount /dev/mapper/rag_storage /opt/rag-data

# --- TLS configuration for the RAG query interface (nginx) ---

# server {
#     listen 8443 ssl http2;
#     server_name rag.internal.hospital;
#
#     ssl_certificate     /etc/ssl/internal/rag.crt;
#     ssl_certificate_key /etc/ssl/internal/rag.key;
#     ssl_protocols       TLSv1.3;
#     ssl_ciphers         TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256;
#     ssl_prefer_server_ciphers on;
#
#     # HSTS (enforce TLS for all future connections)
#     add_header Strict-Transport-Security "max-age=31536000" always;
#
#     location / {
#         proxy_pass http://127.0.0.1:8000;
#         proxy_set_header X-Real-IP $remote_addr;
#         proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#     }
# }

# --- Backup encryption (restic with age encryption) ---

# Initialize an encrypted backup repository
restic init \
    --repo /mnt/backup/rag-backups \
    --password-file /etc/rag/backup-keyfile

# Create an encrypted backup
restic backup /opt/rag-data \
    --repo /mnt/backup/rag-backups \
    --password-file /etc/rag/backup-keyfile \
    --tag daily \
    --tag medical-rag

Layer 6: De-Identification

Even within an authorized system, there are contexts where PHI should not appear. A researcher querying the RAG system for educational purposes should see de-identified data. A response generated by the LLM should not unnecessarily expose patient names, dates of birth, or other direct identifiers if the clinical question does not require them. De-identification is the final layer that strips identifiers from data before it reaches the user or before it is embedded.

HIPAA defines two methods of de-identification: Safe Harbor and Expert Determination. Safe Harbor requires the removal of 18 specific identifiers (names, geographic subdivisions smaller than a state, dates more specific than year, phone numbers, fax numbers, email addresses, Social Security numbers, medical record numbers, health plan beneficiary numbers, account numbers, certificate/license numbers, vehicle identifiers, device identifiers, URLs, IP addresses, biometric identifiers, full-face photographs, and any other unique identifying number). Expert Determination requires a qualified statistician to certify that the risk of re-identification is very small.

The RAG system implements automated de-identification using named entity recognition (NER) trained on medical text:

import re
from typing import Optional

# --- Automated PHI De-Identification ---

# Regex patterns for direct identifiers (Safe Harbor approach)
PHI_PATTERNS = {
    'ssn':          r'\b\d{3}-\d{2}-\d{4}\b',
    'mrn':          r'\bMRN[:\s]*\d{6,10}\b',
    'phone':        r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b',
    'email':        r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b',
    'date_full':    r'\b\d{1,2}/\d{1,2}/\d{2,4}\b',
    'date_iso':     r'\b\d{4}-\d{2}-\d{2}\b',
    'zip':          r'\b\d{5}-\d{4}\b',
    'account_num':  r'\bACCT[:\s]*\d{6,12}\b',
}

# NER model for patient names, provider names, facility names
# In production, use a medical NER model (e.g., a local instance of
# a model trained on i2b2 de-identification challenge data)


def deidentify_text(text: str, preserve_clinical_dates: bool = False) -> str:
    """Remove or mask PHI from text per HIPAA Safe Harbor method.

    Args:
        text: The clinical text to de-identify.
        preserve_clinical_dates: If True, replace dates with relative
            terms (e.g., '2024-03-15' -> '[DATE]'). If False,
            remove dates entirely.
    """
    # Apply regex-based replacements
    for phi_type, pattern in PHI_PATTERNS.items():
        if phi_type == 'date_full' or phi_type == 'date_iso':
            if preserve_clinical_dates:
                text = re.sub(pattern, '[DATE]', text)
            else:
                text = re.sub(pattern, '[REDACTED]', text)
        else:
            text = re.sub(pattern, f'[{phi_type.upper()}_REDACTED]', text)

    # NER-based de-identification (conceptual)
    # entities = ner_model.extract(text)
    # for entity in entities:
    #     if entity.label in ['PATIENT', 'DOCTOR', 'HOSPITAL']:
    #         text = text.replace(entity.text, f'[{entity.label}]')

    return text


def deidentify_for_research(doc: dict) -> dict:
    """Produce a de-identified copy of a document for researcher access."""
    deidentified = doc.copy()
    deidentified['text'] = deidentify_text(doc['text'])
    deidentified['patient_id'] = generate_pseudonymous_id(doc['patient_id'])
    deidentified['de_identified'] = True
    # Remove all metadata fields that contain identifiers
    if 'metadata' in deidentified:
        meta = json.loads(deidentified['metadata']) if isinstance(
            deidentified['metadata'], str
        ) else deidentified['metadata']
        for key in ['patient_name', 'patient_birth_date',
                    'accession_number', 'institution_name']:
            if key in meta:
                meta[key] = '[REDACTED]'
        deidentified['metadata'] = json.dumps(meta)
    return deidentified


def generate_pseudonymous_id(patient_id: str) -> str:
    """Generate a stable, non-reversible pseudonymous ID.

    The mapping from real patient_id to pseudonymous_id is stored in
    a separate, access-controlled lookup table. Only the re-identification
    authority (typically a compliance officer) can reverse the mapping.
    """
    import hashlib
    return 'PT-' + hashlib.sha256(
        (patient_id + REID_SALT).encode()
    ).hexdigest()[:12]


# --- Response de-identification ---

def sanitize_llm_response(response: str, user_role: str) -> str:
    """Ensure LLM responses do not leak PHI to unauthorized roles."""
    if user_role == 'researcher':
        return deidentify_text(response, preserve_clinical_dates=True)
    # For clinical roles, PHI in the response is appropriate
    # (they are authorized to see it). But we still strip SSNs,
    # insurance numbers, and other non-clinical identifiers that
    # should never appear in a clinical answer.
    response = re.sub(PHI_PATTERNS['ssn'], '[SSN_REDACTED]', response)
    response = re.sub(
        PHI_PATTERNS['account_num'], '[ACCOUNT_REDACTED]', response
    )
    return response

De-identification is not a one-time transformation applied at ingestion. It is applied dynamically based on the requesting user's role. The underlying database may contain identified data (necessary for clinical use), but the retrieval and response pipeline de-identifies on the fly for researchers and education contexts. The de-identified text is what gets embedded for the researcher's separate vector collection, ensuring that even the vector representation does not encode identifiers that could be extracted.

HIPAA Compliance Checklist for RAG Systems

The HIPAA Security Rule specifies four categories of technical safeguards. A medical RAG system must address each. The following checklist maps the rule's requirements to concrete implementation steps:

1. Access Controls (45 CFR § 164.312(a))

2. Audit Controls (45 CFR § 164.312(b))

3. Integrity Controls (45 CFR § 164.312(c))

4. Transmission Security (45 CFR § 164.312(e))

Additional Organizational Requirements

Beyond the four technical safeguards, HIPAA requires organizational measures that the RAG system supports but cannot implement alone:

• Security risk analysis: An annual risk assessment covering the RAG system, its data flows, its access controls, and its threat surface. The assessment identifies risks and documents mitigation plans.
• Workforce training: All clinicians, researchers, and administrators using the RAG system receive HIPAA training. They understand that querying patient data they are not authorized to view is a violation, even if the system permits it technically.
• Business associate agreements: If any component of the RAG system is provided by a third party (even an open-source vendor offering support contracts), a BAA is required. For a fully on-premise, self-hosted stack, no BAAs are needed—another advantage of sovereignty.
• Incident response plan: A documented procedure for responding to suspected breaches: detection, containment, notification, and post-incident review. The RAG system's audit logs are the primary evidence source for incident investigation.

What AI Can and Cannot Do With Clinical Data

The capabilities of medical RAG are powerful, but they have a hard boundary. Understanding this boundary is not a matter of caution—it is a matter of patient safety, legal liability, and professional ethics.

What AI Can Do: Research and Education

• Clinical literature retrieval: A RAG system indexed on medical literature, textbooks, and guidelines can answer "What does the current evidence say about first-line treatment for community-acquired pneumonia in immunocompromised patients?" and cite specific sources. This supports evidence-based medicine by making the literature searchable in natural language.
• Similar case retrieval: A clinician can find prior cases with similar presentations: "Show me cases where a patient presented with unilateral vision loss and was ultimately diagnosed with temporal arteritis." The RAG system retrieves matching clinical notes, imaging studies, and lab results from the hospital's own de-identified case archive.
• Medical education: Residents and medical students can query the system for teaching cases, anatomical variants, and imaging examples. "Show me examples of the CT signs of acute appendicitis" returns a curated set of de-identified studies with radiologist dictations explaining the findings.
• Workflow assistance: The RAG system can summarize a patient's recent encounters, lab trends, and imaging history for a clinician preparing for a consultation. This is information synthesis, not diagnosis—the clinician makes the clinical assessment.
• Radiology report drafting assistance: The system can suggest report structure based on retrieved similar studies and the study metadata. The radiologist edits, verifies, and signs the report. The AI does not interpret the image.
• Quality assurance: Retrospective review of cases can identify missed findings, documentation gaps, and care variations. The RAG system retrieves and organizes the data; the quality assurance team makes the clinical judgments.

What AI Cannot Do: Clinical Decision-Making

Specifically, the RAG system must not:

• Provide a diagnosis: The system can retrieve studies with similar findings and surface relevant differential diagnoses from the literature. It cannot state "This patient has a pulmonary embolism." That conclusion requires the radiologist's interpretation of the imaging in the full clinical context.
• Recommend a treatment plan: The system can summarize treatment guidelines and evidence from the literature. It cannot state "Start the patient on anticoagulation therapy." Treatment decisions depend on patient-specific factors—allergies, comorbidities, medication interactions, patient preferences—that require clinical judgment the AI does not possess.
• Replace clinical judgment: The system's retrieval may be incomplete, its embeddings may miss relevant nuances, and the LLM may hallucinate plausible-sounding but incorrect information. A clinician who treats the AI's output as authoritative is abdicating their professional responsibility.
• Make triage or prioritization decisions: The system cannot determine which patient should be seen first, which study is more urgent, or which finding warrants immediate intervention. These are clinical prioritizations that consider the full patient context.
• Determine patient consent capacity: The system cannot assess whether a patient can consent to a procedure, understands the risks, or has decision-making capacity. This is a clinical and ethical determination.

Human Oversight Requirements

Every output of the medical RAG system must be reviewed by a qualified clinician before it influences patient care. The system's response should be framed as a starting point for the clinician's own analysis, not as a conclusion. The query interface should display a prominent reminder: "AI-generated content is for reference only. Verify all clinical information against primary sources before acting."

This is not a limitation to be worked around. It is the correct and safe design. Medical AI systems that position themselves as decision-makers face a much higher regulatory bar (FDA approval as a medical device, clinical validation trials, liability frameworks for AI-caused harm). RAG systems that position themselves as information retrieval and synthesis tools operate in a more tractable regulatory space and provide genuine value without assuming clinical responsibility they are not equipped to bear.

The Sovereign Medical RAG Stack

A fully on-premise medical RAG system is built from open-source components, each running on hospital-owned hardware within the hospital network. No component requires an external API call. No data leaves the perimeter. The stack below is a complete, deployable architecture:

Hardware Requirements

The sovereign stack requires capable hardware, but the investment is modest compared to the cost of a single HIPAA breach:

• GPU server: 1–2 NVIDIA GPUs (e.g., RTX 4090 or A6000) for LLM inference and Whisper transcription. The LLM runs in 4-bit or 8-bit quantization to fit in VRAM. A 70B parameter model in 4-bit quantization requires approximately 40GB VRAM.
• Vector + application server: 64GB RAM, fast NVMe storage (1–2TB) for the Qdrant database and embedding cache. CPU is sufficient for Qdrant; no GPU needed.
• DICOM storage: 4–10TB encrypted storage for the Orthanc DICOM archive, depending on study volume. SSD for active studies, HDD or tape for archival.
• Log server: Separate, hardened machine for tamper-evident audit log storage. Minimal attack surface (no unnecessary services, no outbound network).
• Backup storage: Encrypted, offsite (within hospital system's physical control) for disaster recovery. Restic or BorgBackup with separate encryption keys.

Total hardware cost: approximately $20,000–$40,000 for a mid-sized hospital deployment. Compare this to the $2,068,928 maximum annual HIPAA penalty. The hardware pays for itself by eliminating the risk of a single breach, and it is a one-time capital expense, not a recurring cloud subscription that scales with data volume.

Example Architecture: A Hospital Deployment

Consider a 400-bed regional medical center that performs approximately 200 imaging studies per day (CT, MRI, X-ray, ultrasound), generates 150 clinical notes per day, and has 300 lab results reported per day. The hospital has an existing EHR system (Epic or Cerner), a PACS for DICOM storage, and a dictation system for radiology reports. The goal is to deploy a RAG system that allows clinicians to search across all of these data sources from a single interface.

System Architecture

+------------------------------------------------------------------+
|                    HOSPITAL INTERNAL NETWORK                      |
|                     (No Internet Access)                         |
|                                                                  |
|  +------------------+     +-------------------+                 |
|  |   PACS Server    |     |   EHR System      |                 |
|  |   (Existing)     |     |   (Epic/Cerner)   |                 |
|  |                  |     |                   |                 |
|  | DICOM images     |     | Clinical notes    |                 |
|  | stored here      |     | Lab results       |                 |
|  +--------+---------+     +---------+---------+                 |
|           |                         |                           |
|           | DICOM C-STORE            | FHIR API                 |
|           | (HL7)                    | (HTTPS)                  |
|           v                         v                           |
|  +--------+---------+     +---------+---------+                 |
|  | Orthanc Server   |     | FHIR Connector    |                 |
|  | (DICOM archive)  |     | (Python service)  |                 |
|  |                  |     |                   |                 |
|  | - Stores DICOM   |     | - Pulls clinical  |                 |
|  | - Serves via     |     |   notes via FHIR  |                 |
|  |   WADO-RS        |     | - Pulls lab       |                 |
|  |                  |     |   results         |                 |
|  +--------+---------+     +---------+---------+                 |
|           |                         |                           |
|           | pydicom extraction       | normalized text           |
|           v                         v                           |
|  +--------+-------------------------+---------+                 |
|  |           INGESTION PIPELINE                  |              |
|  |                                               |              |
|  |  +----------------+    +-------------------+ |              |
|  |  | DICOM Ingestor |    | Note/Lab Ingestor | |              |
|  |  | (metadata      |    | (FHIR -> text)     | |              |
|  |  |  extraction +   |    |                   | |              |
|  |  |  text desc)     |    |                   | |              |
|  |  +-------+--------+    +--------+----------+ |              |
|  |          |                      |            |              |
|  |          +----------+-----------+            |              |
|  |                     |                        |              |
|  |                     v                        |              |
|  |          +----------+----------+             |              |
|  |          | Whisper Transcriber |             |              |
|  |          | (dictation audio -> |             |              |
|  |          |  text)              |             |              |
|  |          +----------+----------+             |              |
|  |                     |                        |              |
|  +---------------------+------------------------+              |
|                        |                                       |
|                        | normalized documents                  |
|                        v                                       |
|  +---------------------+------------------------+              |
|  |              EMBEDDING SERVICE                |            |
|  |                                               |            |
|  |  Local embedding model (sentence-transformers |            |
|  |  or Ollama embeddings)                         |            |
|  |                                               |            |
|  |  Text -> vector                                |            |
|  +---------------------+-------------------------+            |
|                        |                                       |
|                        | vectors + payload                     |
|                        v                                       |
|  +---------------------+-------------------------+            |
|  |              QDRANT VECTOR DB                  |            |
|  |                                               |            |
|  |  Collection: medical_rag                      |            |
|  |  - vectors (embeddings)                      |            |
|  |  - payload (source_type, patient_id,         |            |
|  |    authorized_users, de_identified,           |            |
|  |    timestamp, text, metadata)                |            |
|  |                                               |            |
|  +---------------------+-------------------------+            |
|                        |                                       |
|                        | similarity search                     |
|                        v                                       |
|  +---------------------+-------------------------+            |
|  |           RAG QUERY ENGINE                     |            |
|  |                                               |            |
|  |  1. Receive clinician query                    |            |
|  |  2. Check authentication + RBAC               |            |
|  |  3. Embed query                                |            |
|  |  4. Qdrant search with DLAC filter            |            |
|  |  5. Cross-modal join (DICOM + notes + labs)   |            |
|  |  6. Assemble prompt context                    |            |
|  |  7. LLM generates response (Ollama)          |            |
|  |  8. De-identify if researcher                 |            |
|  |  9. Log to audit                               |            |
|  | 10. Return response + source citations        |            |
|  |                                               |            |
|  +---------------------+-------------------------+            |
|                        |                                       |
|                        v                                       |
|  +---------------------+-------------------------+            |
|  |        CLINICIAN WEB INTERFACE                 |            |
|  |                                               |            |
|  |  - Search bar (natural language)             |            |
|  |  - Results with source citations              |            |
|  |  - Cornerstone.js DICOM viewer               |            |
|  |  - Clinical note display                      |            |
|  |  - Lab result tables                          |            |
|  |  - Dictation transcripts                      |            |
|  |  - Role indicator (treatment/research mode)   |            |
|  |                                               |            |
|  +-----------------------------------------------+            |
|                                                               |
|  +-----------------------------------------------+            |
|  |        AUDIT LOG SERVER (HARDENED)              |            |
|  |                                               |            |
|  |  - Receives syslog from all components        |            |
|  |  - Append-only storage                        |            |
|  |  - No outbound network                        |            |
|  |  - 6-year retention                           |            |
|  |                                               |            |
|  +-----------------------------------------------+            |
|                                                               |
+------------------------------------------------------------------+

Data Flow: From Acquisition to Queryable

DICOM ingestion: When a CT scanner completes a study, it sends the DICOM files to the Orthanc server via the DICOM C-STORE protocol over the hospital network. The DICOM ingestor service, which subscribes to Orthanc's webhook (DICOM STOW-RS events), is notified of new studies. It retrieves the study metadata via pydicom, generates a text description, normalizes it into a RAG document, embeds it, and stores the vector in Qdrant. The study is available for search within minutes of completion.

Clinical note ingestion: The FHIR connector queries the EHR's FHIR API every 5 minutes for new or updated clinical notes. Each note (DocumentReference resource in FHIR) is fetched, normalized into a RAG document, embedded, and stored in Qdrant. The connector maps FHIR resource references to internal patient IDs and queries the EHR for the patient's care team to populate the authorized_users field.

Lab result ingestion: The FHIR connector also retrieves Observation resources (lab results). Each panel is converted to a text representation (as shown in the normalization code above), embedded, and stored. Lab results are linked to clinical notes by patient ID and timestamp, enabling cross-referential queries.

Voice dictation ingestion: Radiologists dictate findings using hospital-issued dictation microphones. Audio files are uploaded to the ingestion pipeline's audio receiver service. Whisper (running locally on the GPU server) transcribes the audio to text. The transcript is normalized as a "voice dictation" document, embedded, and stored in Qdrant. The transcription takes 10–30 seconds for a typical 2-minute dictation on a single GPU.

A Clinical Query in Action

A pulmonologist is reviewing a patient with an unusual CT finding and wants to compare it to similar cases. They log into the RAG interface with their smart card and password (multi-factor authentication). The system identifies them as an attending physician with access to their assigned patients.

They enter the query: "Show me all CT scans of the chest in my patients with ground-glass opacities, along with the radiologist's interpretation and any relevant prior imaging."

The system processes the query:

1. The query text is embedded using the local embedding model.
2. Qdrant searches the medical_rag collection with a filter requiring source_type='dicom', modality='CT', body_part_examined='CHEST', and the user's ID in authorized_users.
3. The top results include DICOM study descriptions and their text descriptions (which mention ground-glass opacity in the study or series description).
4. The system cross-references: for each retrieved study, it queries Qdrant for voice dictations and clinical notes with the same patient_id and a timestamp within 48 hours of the study.
5. The retrieved context (study descriptions, dictation transcripts, relevant notes) is assembled into a prompt.
6. Ollama generates a response: a structured summary of each matching study, the radiologist's interpretation (from the dictation transcript), and a note about prior imaging if available.
7. The response includes source citations with clickable links to view the DICOM study in Cornerstone.js and to read the full dictation transcript.
8. The entire interaction is logged to the audit system.

The pulmonologist sees a list of relevant studies with thumbnails, reads the synthesized summary, and clicks through to examine specific images and reports. They make the clinical assessment. The RAG system provided the information; the physician provides the judgment.

Deployment Considerations

Model Selection for Medical RAG

The LLM chosen for the RAG system should be evaluated on medical question-answering benchmarks. Open models that perform well on medical knowledge:

• Llama 3 (70B): Strong general reasoning, performs well on medical QA tasks when quantized to 4-bit. Requires approximately 40GB VRAM.
• Qwen 2.5 (72B): Excellent multilingual support, strong performance on clinical reasoning tasks. 4-bit quantization fits in 40GB VRAM.
• Mistral Large: Good instruction-following, handles multi-document synthesis well.
• Med-specific fine-tunes: Models fine-tuned on medical data (e.g., MedLlama, ClinicalMistral) may perform better on specific clinical tasks but should be evaluated against the base models to confirm the improvement justifies the narrower training.

For the embedding model, a domain-appropriate choice matters. General-purpose sentence embedding models (e.g., all-MiniLM-L6-v2) work adequately for metadata-driven descriptions. For clinical note embeddings, a model trained on medical text (e.g., BioBERT-based embeddings, PubMedBERT embeddings) may capture clinical terminology more effectively. Benchmark both on your specific data and choose based on retrieval quality, not parameter count.

Continuous Operation

The RAG system runs continuously. Ingestion happens in real time as new studies, notes, and dictations arrive. The system must handle:

• Backfill: When the system is first deployed, it must ingest the existing PACS archive (potentially years of studies) and the existing EHR notes. This is a batch process that may take days depending on volume. It should run at low priority to avoid impacting clinical systems.
• Updates: Model updates (new LLM versions, improved embedding models) are applied via the jump host. Model weights are transferred on encrypted physical media or via a one-way data diode, never via internet download from the RAG network.
• Monitoring: The system's health (GPU utilization, Qdrant response time, ingestion queue depth, error rates) is monitored by a local monitoring stack (Prometheus + Grafana). Alerts are sent to the on-call administrator via the hospital's internal paging system.
• Disaster recovery: Encrypted backups are taken daily and stored on separate hospital-controlled infrastructure. A documented recovery procedure specifies how to restore the vector database, DICOM archive, and configuration from backup. Recovery is tested quarterly.

Conclusion: Sovereignty as the Default

Medical RAG is not general-purpose RAG with a HIPAA sticker applied. It is a fundamentally different architecture designed around a non-negotiable constraint: patient data never leaves the hospital's physical and logical control. This constraint shapes every decision, from the choice of LLM (must run locally) to the choice of vector database (must be self-hosted) to the choice of transcription (Whisper, not a cloud API) to the network topology (isolated VLAN, no internet egress).

The six security layers—network isolation, authentication, document-level access control, audit logging, encryption, and de-identification—work together. No single layer is sufficient. Network isolation prevents external attacks; authentication prevents unauthorized internal access; document-level access control limits what authorized users can see; audit logging detects and deters misuse; encryption protects data if other layers fail; and de-identification minimizes exposure for contexts where full PHI is not needed.

The technology to build this exists today. Ollama, Qdrant, Orthanc, pydicom, Whisper, Cornerstone.js, and the supporting open-source ecosystem are mature, capable, and free. The hardware is affordable relative to the risk it mitigates. What is required is the institutional commitment to sovereignty—the decision that patient data is too important to trust to a third party's infrastructure, no matter how convenient the cloud alternative appears.

Hospitals that make this commitment gain more than compliance. They gain a research and education tool that their clinicians trust, that their compliance officers can audit, and that their IT teams can secure. They gain a system whose behavior is fully transparent because every component runs on hardware they own. And they gain the certainty that when a breach attempt occurs—and in healthcare, it will—the attack surface is a single, controlled, monitored network segment, not a sprawling cloud footprint shared with thousands of other tenants.