What an Actually HIPAA-Compliant AI Stack Looks Like (Architecture Walkthrough)

What an Actually HIPAA-Compliant AI Stack Looks Like (Architecture Walkthrough)

I keep seeing "HIPAA-compliant AI" thrown around in vendor pitches like it's a checkbox feature. It is not. A HIPAA-compliant AI deployment is an architecture decision that touches every layer of your stack, from the model inference endpoint down to how you rotate encryption keys. If OCR comes knocking (and their enforcement budget grew to $28 million in FY2024), you need to show a coherent, documented chain of compliance, not a marketing PDF from your vendor.

So here is what a reference architecture actually looks like when you build it to survive scrutiny.

The BAA Chain: Your Compliance Backbone

Every entity that touches, processes, stores, or transmits PHI in your AI pipeline needs a Business Associate Agreement. Every single one. This includes your cloud infrastructure provider, your model hosting service, your vector database vendor, your logging platform, and yes, your fine-tuning data pipeline if it ingests PHI at any stage.

The BAA chain is where most healthcare AI deployments quietly fall apart. You sign a BAA with AWS or Azure (both offer them), then you bolt on a third-party embedding service that has no BAA, and now you have an unprotected hop in your data flow. OCR's 2023 settlement with Banner Health ($1.25 million) reinforced that covered entities bear responsibility for their business associates' failures. You own the whole chain.

A practical BAA map for a typical AI stack looks like this:

• Cloud infrastructure (AWS, Azure, GCP): BAA required. All three offer them, but you need to configure eligible services only. AWS, for example, limits its BAA to specific services listed in their HIPAA Eligible Services documentation.
• Model inference (self-hosted or API): If you are calling an external LLM API with PHI, that provider needs a BAA. OpenAI offers a BAA through its Enterprise tier. Anthropic offers one through their API terms for eligible customers. If your provider does not offer a BAA, PHI cannot touch that endpoint. Full stop.
• Vector database / retrieval layer: Pinecone, Weaviate, or whatever you use for RAG. If PHI is embedded and stored there, BAA required.
• Monitoring and logging: Datadog, Splunk, or your SIEM. If logs contain PHI (and they often do, even accidentally), BAA required.
• Fine-tuning pipeline: Any service or contractor involved in model training on PHI-containing datasets.

Document this chain. Literally draw the diagram, label every node with the BAA status and execution date, and keep it updated. OCR investigators love diagrams.

Encryption Boundaries: At Rest, In Transit, In Use

The HIPAA Security Rule (45 CFR 164.312(a)(2)(iv) and 164.312(e)(1)) treats encryption as an addressable specification, which people sometimes misread as optional. It is addressable in the sense that you must either implement it or document why an equivalent alternative is reasonable. In practice, for an AI stack in 2025, there is no reasonable alternative. Encrypt everything.

In transit: TLS 1.2 minimum for all API calls, internal service communication, and data pipeline transfers. Every hop between the components in your BAA chain needs encrypted transport. Mutual TLS between microservices is a strong pattern here.

At rest: AES-256 for all PHI storage, including your vector database, model weights if they were fine-tuned on PHI, training datasets, and inference logs. Use your cloud provider's KMS, and enforce customer-managed keys so you control rotation and revocation.

In use: This is the harder one. When PHI is loaded into memory for inference, it is decrypted. Confidential computing environments (Azure Confidential Computing, AWS Nitro Enclaves) provide hardware-level memory encryption that keeps PHI protected even during processing. For high-sensitivity deployments, especially anything involving psychotherapy notes or substance abuse records under 42 CFR Part 2, this layer matters.

Audit Logging: What OCR Actually Wants to See

Section 164.312(b) of the Security Rule requires audit controls, and for an AI system, this means logging with enough granularity to reconstruct who accessed what PHI, when, through which model, and what the output was.

Your audit log architecture should capture:

• Authentication events: Who initiated the AI query. Map this to a specific user identity, not a service account.
• Query content: The input to the model, with PHI elements tagged. Yes, this means logging prompts.
• Model output: What the system returned. If the AI generated a clinical summary, that output is PHI and must be logged and retained.
• Data access patterns: Which records in your RAG retrieval layer were accessed during inference. This is your equivalent of database access logging.
• Administrative actions: Model deployments, configuration changes, BAA updates, access control modifications.

Retention period: HIPAA requires six years for security-related documentation (45 CFR 164.530(j)). Your audit logs should match or exceed this. Store them in immutable, append-only storage. CloudTrail with S3 Object Lock, or equivalent.

The 2024 OCR investigation into Inmediata Health Group (which resulted in a $4.4 million settlement) highlighted how insufficient audit logging made it impossible for the entity to demonstrate the scope of a breach. Without logs, you cannot contain, investigate, or report accurately.

Vendor Due Diligence Checklist

Before any AI vendor enters your PHI data flow, run through this:

• BAA availability: Will they sign one? What does it actually cover? Read the scope limitations carefully.
• SOC 2 Type II report: Current, not older than 12 months. Review the exceptions and management responses.
• Data residency: Where is PHI processed and stored? If inference happens in a region without adequate protections, you have a problem.
• Subprocessor transparency: Does the vendor use subprocessors? Are those subprocessors covered under the BAA chain? OpenAI's use of Azure infrastructure is covered; a smaller vendor's undisclosed use of a third-party GPU cloud might not be.
• Breach notification terms: The BAA should specify notification timelines. HIPAA requires covered entities to report within 60 days of discovery (45 CFR 164.404), but your BAA with vendors should require faster notification to give you time to investigate.
• Model training data handling: Confirm in writing that your PHI will not be used to train or improve the vendor's models. This is a dealbreaker if not explicitly excluded.
• Right to audit: Can you (or a third party you designate) audit the vendor's HIPAA controls? If the answer is no, reconsider.
• Incident response plan: Ask to see it. A vendor without a documented, tested IR plan is a vendor that will fumble a breach.

Minimum Viable Architecture

Putting it together, a defensible healthcare AI deployment looks roughly like this:

• Self-hosted or BAA-covered model inference running inside a VPC with no public endpoints
• PHI ingestion through an encrypted pipeline with de-identification where possible (Safe Harbor or Expert Determination per 164.514)
• Vector storage in a BAA-covered, encrypted database with row-level access controls
• Centralized audit logging to immutable storage with six-year retention
• Identity-aware access controls with MFA, mapped to minimum necessary standards (164.502(b))
• Documented risk analysis (required under 164.308(a)(1)(ii)(A)) that specifically addresses AI-related risks

Skip any of these layers and you are building on a foundation that will not hold up when tested.

How FirmAdapt Addresses This

FirmAdapt's architecture was designed around these requirements from the ground up, not retrofitted after the fact. The platform maintains a complete BAA chain across all infrastructure and subprocessors, enforces encryption at rest and in transit as default configurations rather than optional settings, and generates the granular audit logs that OCR expects, including prompt-level and output-level logging tied to authenticated user identities.

For healthcare organizations evaluating AI platforms, FirmAdapt provides the vendor due diligence documentation (SOC 2 Type II, BAA, subprocessor lists, data residency details) as part of the standard onboarding process. The goal is to give compliance teams the evidence they need without a six-month procurement cycle to extract it.