Checklist: decommissioning a GenAI application - deployments, keys, indexes, logs and data

#The goal

A pilot ends, a chatbot gets replaced, or a team moves to the shared platform. The app is "switched off", but its pieces stay behind: a model deployment that still accepts calls, an API key in three pipelines, a vector index full of internal documents, and prompt logs that hold customer data. Each one costs money, adds attack surface, or both.

Use this checklist to retire a GenAI application in a fixed order: confirm → freeze → revoke → delete → verify → record. Work through it with the app owner and someone who can approve deleting data.

Caution

Deleting indexes, Knowledge Bases, logs or a purged Azure resource can't be undone. Get written sign-off from the data owner on what has to be kept (retention, legal hold, audit) before you start the delete steps.

#1. Build the inventory

• List every resource the app uses: model deployments, gateway APIs, search or vector indexes, storage containers, Key Vault secrets, managed identities, IAM roles, diagnostic settings, dashboards and alerts.
• Find resources by tag if your landing zone tags by app:

az resource list --tag app=<app-name> -o table

aws resourcegroupstaggingapi get-resources \
  --tag-filters Key=app,Values=<app-name> \
  --query "ResourceTagMappingList[].ResourceARN"

• Check for shared resources. If the Azure OpenAI account or the Bedrock inference profile is shared with other apps, you remove only this app's deployments, keys and roles, not the whole resource.

#2. Confirm nothing still calls it

• Look at the last 14–30 days of request metrics. For Azure OpenAI, list the metric names first, because they vary by resource kind and API version:

RID=$(az cognitiveservices account show -g <rg> -n <account> --query id -o tsv)
az monitor metrics list-definitions --resource $RID --query "[].name.value" -o tsv
az monitor metrics list --resource $RID --metric AzureOpenAIRequests \
  --interval P1D --offset 30d --aggregation Total -o table

• If diagnostic logs go to Log Analytics, find who was calling and when:

AzureDiagnostics
| where TimeGenerated > ago(30d)
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where Resource =~ "<account>"
| summarize Calls = count(), LastCall = max(TimeGenerated) by OperationName, CallerIPAddress
| order by LastCall desc

Table and column names depend on whether your diagnostic setting uses resource-specific tables; check your workspace.

• For Bedrock, check invocations per model or inference profile:

aws cloudwatch get-metric-statistics --namespace AWS/Bedrock \
  --metric-name Invocations --dimensions Name=ModelId,Value=<model-or-profile-id> \
  --start-time $(date -u -d '-30 days' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 86400 --statistics Sum

• Any traffic in the last week? Find the caller before you go further. Look for a forgotten cron job, a notebook or another team.

#3. Freeze: stop traffic in a way you can undo

Run with access blocked for a week or two before you delete anything. If someone complains, you can turn it back on in minutes.

• Disable the gateway route, or return 410 from the app's API in API Management:

az apim api list -g <rg> --service-name <apim> -o table
# Remove the product association or apply a deny policy to the API before deleting it

• Turn off the feature flag or scale the app to zero (az webapp stop, kubectl scale deploy/<app> --replicas=0).
• Write down the freeze date and the planned delete date in the change ticket.

#4. Revoke credentials and access

• Regenerate both Azure OpenAI keys if the account is dedicated to this app (shared account: skip this and remove the app's role assignments instead):

az cognitiveservices account keys regenerate -g <rg> -n <account> --key-name key1
az cognitiveservices account keys regenerate -g <rg> -n <account> --key-name key2

• Disable the app's secrets in Key Vault instead of deleting them straight away:

az keyvault secret set-attributes --vault-name <kv> --name <secret> --enabled false

• Remove role assignments held by the app's managed identity or service principal:

az role assignment list --assignee <principal-id> --all -o table
az role assignment delete --assignee <principal-id> --scope <scope>

• On AWS, detach the app role's Bedrock policies, then delete the role once nothing uses it (aws iam list-attached-role-policies, aws iam detach-role-policy).
• Revoke any gateway subscription keys or LiteLLM virtual keys issued to the app.

#5. Delete paid and data-holding resources

Delete the things that cost money every hour first, then the things that hold data.

#Model capacity

• Azure OpenAI deployments, especially provisioned (PTU) ones:

az cognitiveservices account deployment list -g <rg> -n <account> -o table
az cognitiveservices account deployment delete -g <rg> -n <account> --deployment-name <name>

Warning

Provisioned deployments and Azure reservations are billed separately. Deleting the deployment doesn't cancel a reservation, and deleting a Bedrock provisioned throughput with a commitment term doesn't end the commitment. Check the commitment with your billing owner first.

• Bedrock provisioned throughput and application inference profiles:

aws bedrock list-provisioned-model-throughputs --query "provisionedModelSummaries[].[provisionedModelName,provisionedModelArn,status]" --output table
aws bedrock delete-provisioned-model-throughput --provisioned-model-id <arn>

aws bedrock list-inference-profiles --type-equals APPLICATION --output table
aws bedrock delete-inference-profile --inference-profile-identifier <arn>

#Retrieval data

• Export or snapshot the index only if the data owner asked you to keep it.
• Delete the search service, or only the app's indexes on a shared service:

az search service delete -g <rg> -n <search-service> --yes

• Delete Bedrock Knowledge Bases and their data sources. Deleting the Knowledge Base doesn't empty the vector store behind it (for example OpenSearch Serverless or Aurora), so delete that too:

aws bedrock-agent list-knowledge-bases --output table
aws bedrock-agent delete-knowledge-base --knowledge-base-id <kb-id>

• Remove source document copies, chunk caches and embedding staging files from blob or S3 containers.
• Clear any semantic cache entries (Redis keys or a cache index) that hold the app's prompts and answers.

#Safety and config

• Delete Guardrails and content filter configurations used only by this app (aws bedrock list-guardrails, aws bedrock delete-guardrail --guardrail-identifier <id>).
• Archive the prompt templates and evaluation sets into the repo, then delete them from the prompt registry.

#6. Logs, traces and diagnostic settings

• Decide on retention with the data owner: prompt and response logs often contain personal data.
• Remove diagnostic settings so no new logs arrive:

az monitor diagnostic-settings list --resource $RID -o table
az monitor diagnostic-settings delete --name <setting> --resource $RID

• If this account was the only Bedrock user, check model invocation logging and turn it off only after confirming that:

aws bedrock get-model-invocation-logging-configuration
aws bedrock delete-model-invocation-logging-configuration

• Set retention on, or delete, the app's log groups, Log Analytics tables or trace projects.
• Delete dashboards and alert rules, so nobody gets paged for an app that no longer exists.

#7. Delete the resource itself (dedicated resources only)

• Delete the Azure OpenAI or Foundry account. It goes into a soft-deleted state, which holds the name and may hold quota:

az cognitiveservices account delete -g <rg> -n <account>
az cognitiveservices account list-deleted -o table
az cognitiveservices account purge -l <location> -g <rg> -n <account>

Caution

Purging is permanent. Purge only after the freeze period, when you're sure you won't need to recover the account.

#8. Verify and record

• Rerun the inventory query from step 1: nothing tagged for the app should remain apart from what you agreed to keep.
• Check next month's cost report for the app's tag or cost centre.
• Remove the app from the AI use-case register, the gateway catalogue and the kill-switch runbooks.
• Attach to the change ticket: the inventory, traffic evidence, sign-offs, deletion commands and what was kept, with retention dates.

#Prevent it next time

• Tag every GenAI resource with app, owner and data-classification when it's created, and deny untagged deployments by policy.
• Give each app its own identity and deployments, so you can retire it without touching shared resources.
• Record an end-of-life date for pilots when you approve them, and review open pilots each quarter.
• Keep this checklist in your runbook repo and link it from the use-case intake form.