Self-Hosted AI for Data Privacy: A Complete Compliance Guide

Every time you send patient records, financial data, or legal documents to an external AI API, you're creating a compliance risk. The data crosses a boundary you don't control, gets stored in ways you can't audit, and may be used for model training unless you specifically opt out.

Self-hosted LLMs eliminate this problem entirely. The data never leaves your infrastructure.

The Compliance Landscape

Regulation	Industry	Key Requirement	Self-Hosted Advantage
PIPEDA	Canada (all)	Data must stay in Canada or equivalent jurisdiction	Full control over data residency
GDPR	EU (all)	Data minimization, right to deletion, no unauthorized transfers	No third-party data processors
HIPAA	US Healthcare	PHI must be encrypted, access-controlled, auditable	Complete audit trail, no external exposure
SOC 2	SaaS / Tech	Security controls, access monitoring, data protection	Your infrastructure, your controls

Architecture for Compliance

Here's the deployment pattern we use for regulated industries:

# Docker Compose for an air-gapped LLM deployment
version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ./models:/root/.ollama
    ports:
      - "127.0.0.1:11434:11434"  # Bind to localhost only
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  app:
    build: .
    environment:
      - LLM_URL=http://ollama:11434
      - AUDIT_LOG=/var/log/llm-audit.jsonl
    depends_on:
      - ollama
    networks:
      - internal

networks:
  internal:
    driver: bridge
    internal: true  # No external internet access

Key architectural decisions:

• Localhost binding — the LLM API is only accessible from within the server
• Internal network — containers cannot reach the internet
• Audit logging — every prompt and response is logged with timestamps and user IDs
• No telemetry — disable all phone-home features in the serving software

Audit Logging Implementation

import json, datetime

def audit_log(user_id, prompt, response, model):
    entry = {
        "timestamp": datetime.datetime.utcnow().isoformat(),
        "user_id": user_id,
        "model": model,
        "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest(),
        "response_length": len(response),
        "tokens_used": count_tokens(prompt + response)
    }
    with open("/var/log/llm-audit.jsonl", "a") as f:
        f.write(json.dumps(entry) + "\n")

Notice we hash the prompt rather than storing it in plaintext. This gives auditors traceability without creating another copy of potentially sensitive data.