A pragmatic white paper for executives, product owners, and AI program leads
Executive summary
Most AI programs fail not because the models are weak, but because leaders track the wrong things (or nothing at all). This paper defines a focus-first KPI system that ties model quality, adoption, risk, and business outcomes into a single operating view you can run weekly. It is designed for large enterprises rolling out assistive AI (agent copilots, retrieval-augmented chat), predictive models (forecasting, scoring), and automation (workflow agents, RPA+LLM).
What to focus on:
- Business value realized (cash and time), 2) Adoption & behavior change (who uses it, how often, to what effect), 3) Model fitness (offline + online), 4) Operational reliability (SLOs, cost), and 5) Risk & governance (safety, compliance). If you cannot show credible movement on all five, your AI program will stall in year one.
We provide ready-to-use KPIs with formulas, targets, baselines, and a 13-week rollout plan.
KPI architecture at a glance
Outcome layer (board-level):
- Net Value Created (NVC) = Revenue Uplift + Cost Savings + Risk Loss Avoided − Run Cost
- Time Returned to Humans (TRH) in FTE-hours per month
- Adoption Penetration = % of eligible users/flows using AI in the last 30 days
System layer (program & product leads):
- Model Fitness (quality, generalization, drift)
- Operational SLOs (availability, latency, unit cost)
- Experiment Impact (A/B outcomes, causal attribution)
- Governance Health (policy, incidents, auditability)
Telemetry layer (engineering & data teams):
- Data contracts, eval harness, observability, cost & usage bills of materials
KPI catalog (with definitions & targets)
1) Business value & efficiency
- Revenue Uplift (RU)
- Definition: Incremental revenue attributable to AI vs. control (A/B or pre/post with controls).
- Formula: RU = (Treatment Rev − Control Rev) − External Trend Adjustments.
- Target: Positive within 1–2 quarters for customer-facing use cases.
- Cost Savings (CS)
- Definition: Avoided workload cost versus baseline process.
- Formula: CS = (Baseline Unit Cost − AI Unit Cost) × Volume.
- Target: ≥15–30% per automated task by Q2 of rollout.
- Risk Loss Avoided (RLA)
- Definition: Expected loss reduction (fraud, write-offs, safety incidents) from AI detection or guardrails.
- Formula: RLA = Baseline Expected Loss − Observed Expected Loss (adjust for mix).
- Target: Case-dependent; demonstrate statistical significance.
- Time Returned to Humans (TRH)
- Definition: Human time saved by assistive/automation features.
- Formula: TRH = Σ(Tasks × Minutes Saved per Task).
- Target: ≥2–4 hours/month per active user by month 3; ≥8 hours by month 6.
- Net Value Created (NVC)
- Definition: Program P&L view.
- Formula: NVC = RU + CS + RLA − (Cloud + Licensing + Engineering + Change Mgmt).
- Target: Break even in 6–9 months for top 3 use cases.
2) Adoption & behavior change
- Eligible User Coverage (EUC)
- Definition: % of target users with access and onboarding completed.
- Target: ≥80% by 8 weeks post-launch.
- 30-Day Active Use (30DAU/EUC)
- Definition: % of eligible users active in last 30 days.
- Target: ≥50% by month 2; ≥65% by month 4 for assistive tools.
- Task Conversion Rate (TCR)
- Definition: % of candidate tasks completed using AI vs. legacy path.
- Target: ≥40% by month 3; grow to ≥70% for well-fit tasks.
- Depth of Use (DoU)
- Definition: Median tasks per active user per week or median tokens/requests for LLM copilots.
- Target: Up-and-to-the-right trend; pair with value/task.
- User-Perceived Quality (UPQ)
- Definition: 1–5 rating after tasks; also NPS-style question “Did this save you time?”
- Target: ≥4.2/5 with ≥60% “Yes” time-savings.
3) Model fitness (offline + online)
- Task Success Rate (TSR)(LLM/agent)
- Definition: % tasks completed to spec without human correction (judged by eval harness or human review).
- Target: ≥80% for internal assistive tasks; ≥90% for external-facing.
- Grounded Accuracy (GA)(RAG)
- Definition: % answers supported by citations that actually contain the asserted facts.
- Target: ≥85% by month 2; ≥92% with retrieval tuning.
- Safety Non-Compliance Rate (SNCR)
- Definition: % of responses violating policy (PII, harmful, regulated claims).
- Target: ≤0.1% for internal; ≤0.01% for external.
- Calibration & Drift(predictive)
- Definition: ECE/Brier score; PSI/KS drift on key features and outputs.
- Target: Stable within control limits; alert >0.2 PSI or 10% relative change.
- Explainability Coverage (XC)
- Definition: % of predictions/decisions with traceable features, rationale, or chain-of-thought surrogate explanation logged for audit (never store raw chain-of-thought for sensitive domains; use structured rationales).
- Target: 100% for regulated decisions; ≥80% otherwise.
4) Operational reliability & cost
- Service Availability (SLO-A)
- Definition: % time meeting availability SLO.
- Target: 99.9% internal; 99.95% external.
- P95 Latency (SLO-L)
- Definition: End-to-end response time from user action to result.
- Target: Assistive ≤1.5s; customer support ≤2.5s; batch SLAs per domain.
- Unit Economics — Cost per Successful Task (CST)
- Definition: Cloud/compute + licensing + orchestration + guardrails per completed task with success criteria.
- Formula: CST = (Total Variable Cost in Period) / (# Successful Tasks).
- Target: Declining trend; CST < value/task by month 3.
- Hallucination Rework Rate (HRR)
- Definition: % tasks requiring rework due to incorrect/unsupported content.
- Target: ≤3% internal; ≤1% external.
- Incident Rate (IR)
- Definition: Number of Sev1/Sev2 AI incidents per 1k tasks.
- Target: ≤0.05/1k; fast time to detect/resolve.
5) Governance, compliance, and ethics
- Policy Coverage (PC)
- Definition: % of AI apps with documented use policy, data lineage, DPIA/PIA (as applicable), and model cards.
- Target: 100% before go-live.
- Auditability Score (AS)
- Definition: Presence/quality of logs: inputs, prompts, retrieval sources, tool calls, outputs, decisions, reviewer IDs.
- Target: 100% completeness; 90-day immutable retention minimum (longer for regulated).
- Sensitive Data Exposure Rate (SDER)
- Definition: % tasks where sensitive data leaves approved trust boundary.
- Target: 0% for regulated data; strict alerts & blocking.
- Human-in-the-Loop Coverage (HITL-C)
- Definition: % of high-risk tasks with mandated human verification gates.
- Target: 100% for high-risk categories.
- Third-Party Dependency Risk (TPDR)
- Definition: % of critical AI workflows with single-provider risk without viable fallback.
- Target: ≤30%; require at least one fallback (model or provider).
KPI sets by rollout phase
Phase 0–1: Pilot & proof (Weeks 1–6)
North stars: TSR, GA, SNCR, DoU, UPQ, CST.
- Baseline manual flow, define value/task, set guardrails, instrument telemetry.
- Exit criteria: TSR ≥75%, GA ≥80%, SNCR ≤0.3%, CST < baseline unit cost, UPQ ≥4.0.
Phase 2: Production & scale (Weeks 7–20)
North stars: 30DAU/EUC, TCR, TRH, NVC, SLO-A/L, HRR.
- Launch to ≥60% eligible users; run controlled experiments; prune low-value flows.
- Exit criteria: NVC ≥0 in at least one use case; HRR ≤3%; SLO met; adoption ≥50%.
Phase 3: Portfolio optimization (Quarter 2–4)
North stars: RU, CS, RLA (causally attributed), TPDR, AS.
- Consolidate platform, add fallbacks, negotiate pricing, retune prompts/retrieval, retire laggards.
- Exit criteria: NVC positive across top 3 use cases; >8 hours TRH per active user/month.
Measurement system & instrumentation
Foundations to implement in week 1–3:
- Event schema & data contracts: user_id, task_id, input class, retrieval sources, tool calls, output verdict, human review flags, timing, token counts, cost, version IDs.
- Eval harness: curated task sets with gold answers, rubric-based scoring, safety tests; nightly regression.
- Observability: latency/availability SLOs; drift monitors (PSI/KS), cost dashboards; incident runbooks.
- Experimentation: feature flags & holdouts; counterfactual logging where A/B is impractical.
- Model registry & lineage: data → features → model → policy → deployment → logs.
- Access & data boundaries: redact/Mask PII; retrieval scoping; secrets isolation; content filters.
Example KPI scoreboard (single view)
Board-level (monthly):
- NVC, RU/CS/RLA, TRH, Adoption Penetration, Top risks & mitigations.
Weekly operating review:
- Adoption (EUC, 30DAU, TCR, DoU, UPQ)
- Model fitness (TSR, GA, SNCR, Drift)
- Operations (SLO-A/L, CST, HRR, IR)
- Governance (PC, AS, SDER, HITL-C, TPDR)
Rule: If value is flat but usage climbs, re-assess fit (we may be doing the wrong tasks). If value rises but usage is flat, double down on change management and training.
Targets & benchmarks (pragmatic ranges)
- Assistive copilots (internal): TSR ≥80–90%; TRH ≥6–12 hrs/user/month; HRR ≤3%; CST ≤€0.40/task for short-form tasks.
- Customer support deflection: First-contact resolution +8–15 pts; Median handle time −20–35%; CSAT flat-to-up; SNCR ≤0.05%.
- Knowledge search/RAG: GA ≥90% with strict citations; P95 latency ≤2s; user trust (UPQ) ≥4.3/5.
- Predictive risk/fraud: 10–25% lift in precision/recall at fixed review budget; drift alerts <weekly; AS = 100%.
(Treat these as planning ranges; convert to firm targets after 2–3 weeks of your own baselining.)
Financial model & unit economics
- Value per task (VPT):
- For assistive: minutes saved × loaded labor rate.
- For growth: incremental conversion × margin per conversion.
- Cost per task (CPT):
- Tokens × provider rate + vector retrieval + orchestration + guardrails + observability.
- Contribution per task (ContPT):
- ContPT = VPT − CPT; aim for >€0.50/task early, rising as prompting and caching improve.
- Levers to improve ContPT:
- Prompt compression & caching; RAG to reduce context length; specialized smaller models for cheap steps; distillation; batch & streaming modes; routing high-value tasks to highest-quality models only.
Risk, safety & compliance playbook (condensed)
- Policy gating: Purpose, inputs, outputs, retention, human checkpoints.
- Safety tests: red-teaming suites (jailbreaks, PII leakage, hallucination traps).
- Data: DLP, provenance, retrieval allow-lists, hash checks on documents.
- Audit: immutable logs, model cards, change approvals, reproducible prompts & configs.
- People: training on responsible use; “stop-ship” authority for AI leads.
Change management & adoption
- Design for assist, not replace: focus on the most annoying 5–10 micro-tasks first.
- “Two-minute wins”: features that save two minutes but happen 50x/day beat hero projects.
- Embedded champions: 1 per 20–30 users; collect UPQ and examples weekly.
- In-product learning: inline tips, exemplars, “show reasoning” summaries users can trust (no raw hidden chain-of-thought).
13-week rollout plan (do-now template)
Weeks 1–2: pick the top 3 tasks, baseline them, instrument events, define value/task, set guardrails, create eval harness.
Week 3: pilot to 30 users, daily UPQ + TSR, fix prompts and retrieval.
Weeks 4–5: reach EUC 60%, start A/B on deflection or conversion, implement caching & cost dashboards.
Weeks 6–7: governance pack complete (policy, DPIA/PIA, model cards), drift monitors on, fallbacks wired.
Weeks 8–9: scale to 200–500 users, reduce HRR <3%, hit P95 latency goals, publish first NVC view.
Weeks 10–11: prune low-ROI tasks, double down on top 2 winners; negotiation on model/provider pricing.
Weeks 12–13: board readout: NVC, TRH, adoption, risk posture, next-quarter roadmap.
Example OKR tree (ready to paste)
- Objective: Make AI materially improve productivity and CX this quarter.
- KR1 (Value): Achieve €300k NVC across support deflection and sales assist.
- KR2 (Adoption): 65% 30DAU/EUC for targeted cohorts; TCR ≥60% on top 5 tasks.
- KR3 (Quality): TSR ≥85%, GA ≥90%, SNCR ≤0.05%.
- KR4 (Ops): P95 latency ≤1.5s, CST −30% vs. baseline, IR ≤0.05/1k.
- KR5 (Governance): AS = 100%, TPDR ≤30%, HITL-C = 100% for high-risk flows.
Pitfalls to avoid
- Vanity metrics: raw prompt counts or token usage without value/task.
- Overfitting to offline tests: ship, measure online, and iterate.
- One-vendor trap: no fallback path during outages or pricing changes.
- Invisible costs: retrieval, embeddings, orchestration, eval runs—include them in CST.
- No human checkpoints: especially for claims, compliance, finance, and safety-critical outputs.
Appendices
A. KPI glossary (quick copy)
- NVC, RU, CS, RLA, TRH, EUC, 30DAU/EUC, TCR, DoU, UPQ, TSR, GA, SNCR, Drift (PSI/KS), XC, SLO-A, SLO-L, CST, HRR, IR, PC, AS, SDER, HITL-C, TPDR.
B. Minimal dashboard spec
- Pages: Executive (NVC, TRH, Adoption, Risk), Product (TSR, GA, UPQ, HRR), Ops (SLOs, CST), Governance (AS, incidents).
- Drilldowns: by cohort, task, model version, retrieval source, and prompt template.
- Alert thresholds: PSI>0.2, SNCR>0.1%, P95 Latency>target+20%, HRR>3%.
C. Data schema (events)
event_id, timestamp, user_id, role, tenant, app_id, task_id, task_type, input_fingerprint, retrieval_sources[], tool_calls[], model_id, prompt_template_id, output_fingerprint, verdict (success/fail/assisted), minutes_saved, citations[], safety_flags[], tokens_in, tokens_out, cost, latency_ms, version_hash
