Applies toEnterprise GenAI programmes on any provider (Azure OpenAI, Bedrock, OCI GenAI, self-hosted models)
1. Use-case intake (every project)
- Purpose, users (internal or customers) and decisions influenced.
- Data involved: personal, confidential, regulated.
- Model(s) and provider, hosting region.
- Autonomy: does the AI only suggest, or can it act (send emails, change systems)?
- Owner accountable for the outcome.
2. Risk tiers
| Tier | Examples | Approval |
|---|---|---|
| Low | Internal drafting help, summarising public documents | Team lead + checklist |
| Medium | Internal knowledge assistant over confidential data; code assistants | AI governance board review |
| High | Customer-facing advice; decisions about people (hiring, credit, health); agents that change production systems | Board + legal/privacy + security review; human oversight required |
| Prohibited | Uses banned by law or policy (for example, the prohibited practices in the EU AI Act) | Not allowed |
3. Required controls by tier
| Control | Low | Medium | High |
|---|---|---|---|
| Approved model and provider list | ✓ | ✓ | ✓ |
| Content filters / guardrails on | ✓ | ✓ | ✓ |
| Data protection review (DPIA) | ✓ | ✓ | |
| Grounding and citations for factual answers | ✓ | ✓ | |
| Evaluation set and quality threshold before release | ✓ | ✓ | |
| Red teaming (prompt injection, jailbreaks, harmful output) | ✓ | ✓ (external) | |
| Human review of outputs or actions | Optional | Required | |
| Users told they're using AI | ✓ | ✓ | ✓ |
| Logging and monitoring | Basic | ✓ | ✓ + periodic audit |
4. Model and data rules
- A list of approved models and versions. New models go through an evaluation before use.
- Provider terms checked: no training on your data, retention, and data residency.
- Sensitive data is masked or minimised before it goes into prompts. Access is respected at retrieval time (security trimming in RAG).
5. Human oversight and agents
- Actions with impact (payments, deletions, external messages) need human approval.
- Least-privilege tool permissions, with rate limits on actions.
- A kill switch to disable the AI feature quickly.
6. Monitoring in production
- Quality: groundedness and relevance scores on samples, user feedback (thumbs up or down).
- Safety: content filter hits, jailbreak attempts, abuse reports.
- Drift: quality after model version upgrades (re-run the evaluation set).
- Cost and usage per use case.
7. Incidents
Treat harmful output, data leaks through prompts, and agent mistakes as incidents. Record them, find the cause, fix it, and update the evaluation set so the same issue is tested from now on.
8. Documentation
A short "system card" per use case: purpose, model, data, limitations, evaluation results, controls and owner. Review it yearly or after major changes.
Tip
Keep the low tier light (a 10-minute checklist). If governance is slow for harmless use cases, people go around it, and then you can't see the risky ones.