4 Critical Findings in a Healthcare RAG Chatbot — Before It Shipped
A clinical AI team engaged us one week before their patient-facing RAG chatbot was due to go live. Our 374-vector red team found 4 critical findings — including PHI leakage via retrieval manipulation — before a single patient used the system.
4
Critical findings resolved pre-launch
48hr
Full report with evidence delivered
0
Patient data incidents post-launch
Industry|Healthcare AI
The Problem
The team had run internal prompt testing and QA cycles, but no systematic adversarial evaluation against the AI layer. One week before launch, their CISO requested an independent security assessment before sign-off. Neither their AI layer nor their web API endpoints had ever been tested against adversarial inputs or OWASP payloads. Both needed to happen within 48 hours.
What We Did
We ran our full 374-vector red team assessment: 90 prompt injection vectors from the Healthcare AI domain pack against the RAG pipeline, and 284 web-layer payloads against their API endpoints. In parallel, we ran the 6-dimension Continuous AI Evals suite with 57 healthcare-domain test cases to baseline output quality before launch. The full assessment ran over 31 hours. Findings report — full HTML with payload-and-response evidence — was delivered at the 48-hour mark.
The Outcome
We surfaced 4 critical and 3 high findings. The most severe: a PHI leakage vector where a crafted retrieval query caused the system to surface documents outside the requesting patient's own records. A second critical finding: system prompt extraction via role confusion — the full system prompt could be extracted in 3 turns. All findings were remediated and retested within 4 days. The product launched 9 days after our engagement began. Zero patient data incidents in the 6 months that followed.
Want results like this?
Book a free 30-minute scoping call. We'll review your stack and show you exactly where the risk is.