Chapter 25: Real-World Case Studies

The clearest lessons in this book came from production at the Medical SaaS Company, a regulated healthcare analytics platform with a multi-product portfolio. Two of its products were AI-native. As established in the Preface, I refer to them here as Watchdog and Speedy Clinical; the real names, the company, and its customers are anonymized to honor confidentiality. The metrics and the lessons are real.

Case Study 1: Watchdog - Agentic Supply-Chain & Contract Intelligence

Context: A greenfield, AI-native product combining contract performance with utilization analytics so that supply-chain leaders get real-time visibility into how contracts are performing and where spend is leaking - the kind of supply-cost variance that hides in line items until something forces a comparison.

Approach: Watchdog is the hybrid from Chapter 14 done in anger. An LLM extracts terms from contract documents (two-pass extraction with per-field confidence scoring and an OCR fallback for scanned pages); a rules engine performs the deterministic compliance and financial scoring; and ML handles anomaly detection and risk classification (Critical / High / Medium / Low) from committed-versus-actual spend. It runs on Python and FastAPI, and - because the data is sensitive - serves an open-weight Qwen-class model on self-hosted vLLM so inference stays inside the compliance boundary. Late in the build we consolidated a hybrid Java-plus-Python stack to Python-only to remove an unnecessary service hop. One concrete agentic feature was a hospital procurement agent that surfaced medical-supply pricing discrepancies no manual review had caught.

Contract pipeline: LLM extraction, a rules engine, and ML anomaly detection, with human review for low confidence — Use the LLM only where documents are unstructured, keep scoring deterministic, and route doubt to a human.

Lessons learned: First, do not reach for an LLM where a rules engine is more precise and auditable - in a compliance product, determinism is a feature, and the LLM earns its place only on the unstructured documents. Second, per-field confidence scoring with an automatic flag below 70% for human review is intent engineering in miniature: the system knows what it is unsure about and escalates instead of guessing. Third, the self-hosted model boundary was not a performance decision - it was the decision that made the product shippable with protected data at all.

Case Study 2: Speedy Clinical - Clinical Intelligence in a Regulated Build

Context: A clinical intelligence product that surfaces AI-detected patterns and test recommendations from diagnostic and clinical data to help care teams close care gaps and intervene earlier. The point was never to flood clinicians with more information - it was to give the right teams clearer signal so they could act sooner and with more confidence.

Approach: A .NET and C# build on HIPAA-compliant cloud infrastructure with direct EMR integration over HL7/FHIR, delivered with an external partner team. AI surfaces the patterns; clinicians make the decisions. Every clinically meaningful action sits behind a human approval gate, and every agent action is logged for a defensible audit trail.

Lessons learned: In a clinical context the human-in-the-loop design from Chapter 22 is not optional - AI augments judgment, it does not own it, and the approval gate is the product, not an afterthought. The hard engineering was rarely the model; it was the regulated integration, the data boundary, and the audit trail that let the feature be trusted and defended. Build the compliance architecture first, then add the intelligence on top.

Case Study 3: The AI-Native SDLC and Its Adoption

Context: Operationalizing agentic AI across the full software lifecycle at the Medical SaaS Company - and, just as importantly, getting the organization to actually adopt it.

Approach: Claude Code, GitHub Copilot, AWS Kiro, and Gemini wired as first-class CI/CD pipeline stages for code generation, test synthesis, cloud-architecture scaffolding, and documentation refresh, with MCP for governed tool access and quality gates in CI. Adoption ran in deliberate waves (Chapter 27): engineering and data first, then the rest of the organization, anchored by a hackathon, rolling Show and Tell sessions, a shared prompt library, and an AI literacy program.

Measured Outcomes:

* 5x deploy frequency, code-to-release cycle time down 40%
* 23% PR throughput gain with AI-assisted review and test generation
* Test coverage lifted from under 10% to 40% with no dedicated QA team
* New-engineer onboarding cut 70%

Lessons learned: Technology is the easy 20% of an AI transformation; adoption, trust, and process are the other 80%. And attribute honestly - some of those gains came from agentic AI, and some came from plain good engineering (disciplined CI/CD, a culture of quality ownership). Keeping the attribution clean is what lets you have a credible conversation about what AI specifically changed.

Case Study 4: Migration at Scale with Intent-Driven Agents

Context: In a separate effort, over 250 production sites migrated using intent-driven AI agents against real workloads.

Approach: Each page or service received an intent spec - objective, success criteria, cost ceiling, escalation triggers, and a stop rule. Agents transformed each item in an isolated environment with scoped MCP permissions (filesystem, browser screenshots, git commit but no push). An eval verified each result against a visual diff, a link check, and a performance threshold, and anything that tripped an escalation trigger routed to a human.

One intent spec fans out to parallel agents, passes through an eval gate, and either merges or escalates — Parallel agents gave the throughput; the intent spec and the eval gave the safety.

Lessons learned: The throughput came from async, parallel agents; the safety came from intent specs and evals. Self-reported success was never the signal - the eval result was. This is the Silent Partial Completion and Runaway Tool Use failure modes from Chapter 4 designed out in advance.