Home / AI & GenAI
LLM APIs, RAG pipelines, vector search and running GenAI in production. 189 posts.
Improve recall on vague, conversational and multi-part questions by transforming the query before retrieval. Implement standalone-question rewriting, multi-query with reciprocal rank fusion, HyDE and decomposition, then prove the gain with a retrieval evaluation set and keep latency in budget.
CPU-based HPA does not work for GPU inference. Scale vLLM or TGI replicas on waiting requests and KV cache usage with KEDA and Prometheus, pair it with the cluster autoscaler or Karpenter for GPU nodes, and hide multi-minute cold starts with warm capacity.
A step-by-step checklist for rolling a new NVIDIA driver across GPU node pools serving LLMs - compatibility checks, capacity headroom, GPU Operator upgrade policy, canary nodes, validation and rollback.
Embeddings turn text into lists of numbers where similar meaning means nearby vectors. Learn cosine similarity, why approximate nearest-neighbour indexes exist, and build a tiny semantic search over your runbooks with Python before reaching for a vector database.
Estimate how much GPU memory a model needs from its parameter count and precision, add KV cache for context and concurrency, understand FP16, FP8 and 4-bit quantization, and read nvidia-smi to see what is really using the GPU.
Make a first chat completion call to Azure OpenAI and Amazon Bedrock from Python, understand system/user/assistant messages, what tokens are and how they drive cost and limits, and which parameters (temperature, max tokens) to set first.
The nvidia-smi commands on-call engineers actually use on GPU inference nodes - what is installed, who is using memory, why clocks are low, ECC and row remap errors, and NVLink topology.
Make a RAG assistant return only what each user may read. Capture ACLs at ingestion, store them as filterable metadata, apply identity-based filters at query time, keep permissions fresh, and test for leaks before and after go-live.
Practical prompt patterns for day-to-day operations work. Give the model a role, the right context and constraints, ask for structured output, use examples, and review results safely, with ready-to-use prompts for log triage, runbook drafting and change summaries.
The four steps of RAG (chunk, embed, retrieve, generate) explained with a small Python example over your own runbooks, plus the prompt that keeps answers grounded, how to show sources, and the first checks when answers are wrong.
Treat every MCP tool as an API that an untrusted prompt can call. Authenticate clients with OAuth for remote servers, scope tools per role, separate read and write tools, validate arguments, require confirmation for destructive actions, and log every call.
Users report the assistant suddenly giving dangerous commands, phishing links or wrong policy answers. Trace answers back to the chunks that caused them, purge the bad documents from every index and cache, close the ingestion path that let them in, and add provenance checks.
A secret-scanning alert or a sudden token spike shows an Azure OpenAI, OpenAI or Bedrock credential is exposed. Revoke and rotate in minutes without a long outage, confirm abuse from usage logs, contain cost, purge the secret from Git history and move to keyless auth.
All AI features fail with 404 DeploymentNotFound after someone deleted, renamed or recreated a model deployment, or an IaC run replaced it. Recreate the deployment with the same name, model version and capacity, restore quota, and stop drift with locks and code.
A wave of spot or preemptible evictions removed most GPU nodes, model pods are Pending and replacements cannot get capacity. Fail traffic over, get any GPU capacity you can (other sizes, zones, on-demand), and redesign so spot can never take the whole service down.
After a config change, model upgrade or new GPU node type, vLLM fails at startup or under load with torch.OutOfMemoryError or "No available memory for the cache blocks". Restore service with known-good settings, find what changed in the memory budget, and stop it recurring.
Understand how vLLM splits GPU memory between weights, activations and KV cache, read the startup log to see how many tokens fit, size max-model-len and max-num-seqs for your traffic, use FP8 KV cache and prefix caching, and spot preemption before users feel it.
An agent creates two tickets, sends the same email twice or repeats a refund. Trace where the duplicate came from (HTTP retry, model re-planning, framework replay) and make side-effecting tools idempotent with keys, dedup stores and confirmation steps.
A FastAPI app with four Gunicorn workers, or an ingestion job running beside the API, all open the same Chroma directory. Writes fail with "database is locked", some workers don't see new data, and the index occasionally corrupts. Find the processes sharing the directory and move to a single owner.
The Chroma Python API on one page - choosing a client, creating collections with an explicit distance, add vs upsert, query and get with metadata and document filters, includes, updates, deletes, counts and the errors you'll hit most.
Retrieval from Chroma misses obvious matches, distances look odd, and changing ef_search on an existing collection doesn't help. Check the distance space, the HNSW parameters the collection was really built with, and rebuild into a correctly configured collection with a measured recall check.
Asking for 10 nearest neighbours with a tenant or date filter returns 2, or misses the best match. Understand how each engine applies filters to approximate search, then fix it with the right filter mode, iterative scans, partial indexes or partitioning.
Vector search misses exact terms like error codes, part numbers and names; keyword search misses paraphrases. Add a tsvector column next to the embedding, run both searches as CTEs and merge them with reciprocal rank fusion, all inside PostgreSQL.
Scaling out an LLM deployment is useless when each new pod spends most of its time downloading and loading 100+ GB of weights. Measure where the time goes, then fix it with caching, faster storage, streaming loaders, image design and probes.
Searches fail with "collection not loaded" after a restart or deploy, loads hang at 90%, or query nodes run out of memory as collections are added. Understand Milvus load state, size memory for loaded collections, and use release, partitions, replicas and mmap to stay within budget.
Back up Milvus collections to object storage with the milvus-backup tool, restore them under new names to verify before switching, move data between clusters, and run the whole thing as a scheduled, monitored job.
After deleting or upserting millions of entities, Milvus row counts look wrong, object storage keeps growing, memory doesn't drop and searches get slower. Understand how Milvus deletes work, check segment and compaction state, and get storage and latency back under control.
Which Milvus index to choose for your data size, memory budget and recall target, how to create it with pymilvus, which search parameter controls recall for each one, and the commands to inspect and rebuild indexes.
Milvus pods sit in CrashLoopBackOff, the standalone container exits after a minute, or clients get "connection refused" on 19530. Most of the time Milvus itself is fine and one of its dependencies isn't. Check each dependency in order and fix the usual causes.
Scrape Qdrant's /metrics endpoint, add the collection and cluster checks that metrics don't cover, wire Kubernetes probes correctly, and set a short list of alerts for latency, errors, memory, disk, optimizer state and replica health.
The RAG prototype uses chromadb.PersistentClient inside the app, and now three app instances need the same data. Run Chroma as a server in Docker with persistent storage, connect with HttpClient, put authentication and TLS in front of it, add health checks and backups, and know when to move to something bigger.
A tensor-parallel model server hangs at startup or under load with NCCL watchdog timeouts. Turn on NCCL debug logging, check shared memory, PCIe/NVLink topology and P2P, pick the right network interface, and prove the fix with nccl-tests.
Serving many customers or departments from one Pinecone project needs strict isolation, predictable cost and easy offboarding. Compare namespaces, metadata filters and separate indexes, then implement namespace-per-tenant with enforced routing, offboarding and monitoring.
One page for day-to-day pgvector work - column types and their index limits, which operator goes with which operator class, HNSW and IVFFlat syntax, query-time settings and the catalog queries to check what you have.
An HNSW build on 10 million embeddings runs all night, or slows to a crawl halfway with a "graph no longer fits into maintenance_work_mem" notice. Size memory for the graph, use parallel builds, watch progress, and plan loads so the build happens once.
A document store that re-embeds or deletes chunks every day ends up with an HNSW index twice the expected size, autovacuum running for hours on it and slower searches. Measure the bloat, reindex before vacuum, and change the write pattern that causes it.
Similarity search takes seconds and EXPLAIN shows a Seq Scan and a Sort even though an HNSW index exists. Work through the seven usual causes - wrong operator, ORDER BY expression, DESC order, missing LIMIT, untyped parameters, filters and planner costs - and fix each.
The Pinecone operations you use every day on one page - connecting by host, upsert, query with metadata filters and every filter operator, fetch, partial updates, deletes, stats and the metadata rules that cause most filter bugs.
A user uploads a document and asks about it immediately, but the assistant can't find it. Tests that upsert then query fail randomly. Understand Pinecone's eventual consistency, tell real ingestion failures apart from freshness lag, and make the application and tests behave correctly.
Someone ran delete_all on the production namespace, or a bad re-embed overwrote good vectors. Take backups before risky changes, restore a backup into a new index, verify it and point the application at it - with the commands and checks for each step.
Loading a few million embeddings into Pinecone takes all day, requests fail with 429 or "request size exceeds limit", and retries make it worse. Right-size batches, run upserts in parallel within the limits, retry with backoff, and use bulk import from object storage for big loads.
The Qdrant calls you need day to day - create and inspect collections, upsert and delete points, filter syntax, the Query API, scroll, count, aliases, snapshots and health endpoints - side by side in curl and Python.
After a large upload the Qdrant collection shows yellow for hours, or grey after a restart, searches are slow and indexed_vectors_count stays far below points_count. Read the status correctly, trigger pending optimisations, and upload large datasets without making the optimizer do the work twice.
Unfiltered Qdrant searches take a few milliseconds but the same search with a tenant or category filter takes hundreds, or returns fewer results than expected. Add the right payload indexes, mark tenant fields, and understand how Qdrant plans filtered searches.
A single Qdrant node is running out of memory or is a single point of failure. Turn it into a cluster, choose shard and replica counts you won't regret, balance data onto new nodes by moving and replicating shards, and handle a failed peer.
Tests that compare LLM outputs flap, and users get different answers to identical questions even with temperature 0. Understand batching, floating point and routing effects, use seeds and fingerprints where they exist, and design tests and caches that tolerate variation.
One base model with dozens of fine-tuned LoRA adapters saves GPUs, until adapters fail to load, requests slow down or memory runs out. Fix rank and module mismatches, size max-loras and CPU cache, and load adapters dynamically and safely.
Weaviate's hybrid query blends BM25 keyword search and vector search, but the defaults rarely fit every corpus. Build an evaluation set, then tune alpha, the fusion method, which properties BM25 searches, tokenization and the BM25 parameters - one change at a time.
A routine pip install --upgrade moved Chroma across a major version and the app now fails to open its data or talks to the server with the wrong API. Plan Chroma upgrades like database upgrades - pin versions, back up, test on a copy, upgrade client and server together, and keep an export-import path when the on-disk format changes.
Configure a Weaviate backup module, take full or per-collection backups, check their status, restore a collection after an accidental delete, and schedule backups with monitoring - plus the restrictions that trip people up during a real restore.
An import of 2 million chunks "finishes" but the collection holds 1.7 million. Weaviate batches report errors per object, and with text2vec modules most failures come from the embedding provider - rate limits, timeouts and token limits. Find the failed objects, fix the cause, and re-import safely.
Weaviate's native multi-tenancy gives every customer its own shard inside one collection. Set it up, create and manage tenants, deactivate or offload idle tenants to save memory, and fix the common "tenant not found" and "tenant not active" errors.
In a three-node Weaviate cluster one node restarts and writes start failing, or different requests return different results. Understand replication factor and consistency levels, choose settings for reads and writes, and handle node failures, repairs and rolling upgrades.
The az commands, test calls, metric names and KQL you need day to day for Azure OpenAI - from listing deployments and quota to keys, network rules and diagnostic logs.
Seven reasoning models in Azure OpenAI retire on 19 November 2026, with gpt-5.6-sol or gpt-5.6-terra as replacements. Standard deployments are auto-upgraded unless you opted out; provisioned deployments are not and will start returning 410 Gone. Inventory, evaluate, and switch before the date.
For about six hours, Azure OpenAI and Foundry requests in Sweden Central failed intermittently. The incident review recommends retries with backoff, BCDR planning and granular monitoring. Put a second region behind API Management with a circuit breaker, keep retries bounded, and measure per-backend error rates.
Token streaming works against the model API but not through your gateway. Fix buffering, timeouts and compression in NGINX, ingress-nginx, Azure API Management and load balancers.
An MCP server shows as failed, times out or lists no tools. How to test it outside the client, and fix stdio, Streamable HTTP, session and OAuth problems.
One table of the Foundry models that retire between mid-November and mid-December 2026, with replacements where Microsoft names one, plus what each retirement means for your app. Partner models such as Claude and Mistral follow a 12-month lifecycle and have shorter notice.
Deploy NVIDIA DCGM Exporter, scrape it with Prometheus, and use PromQL to catch GPU memory pressure, XID errors, thermal throttling and idle expensive GPUs.
Vectors from different embedding models can't be mixed, so when an embedding model retires (Cohere embed v3 on Foundry is deprecated in November 2026 and retires in February 2027) every document must be re-embedded. Build a second index in parallel, compare retrieval quality on a test set, then switch the query path in one step.
When gpt-4o 2024-05-13 retires on 9 December 2026, Standard deployments get auto-upgraded to gpt-5.6-sol. Decide per deployment whether that's what you want with versionUpgradeOption, and use the Models API's lifecycleStatus and deprecation fields to report what's coming - remembering the API's status names don't match the portal's.
The AWS CLI commands you need day to day on Amazon Bedrock. Find model IDs, send a test prompt, check guardrails and invocation logging, read CloudWatch metrics and look up quotas.
Poor RAG answers often come from poor retrieval. In Azure AI Search, combine keyword and vector search, add the semantic ranker, filter with metadata and boost fresh or authoritative content, then measure each change against a test set.
One table per provider mapping LLM API status codes and error types to their usual cause and fix, plus which errors to retry, how to back off, and what to log so incidents are quick to diagnose.
The NVIDIA GPU Operator installs the driver, container toolkit, device plugin and monitoring on GPU nodes. Install it with Helm, validate with a CUDA test pod and nvidia-smi, and fix the usual driver and plugin problems.
Ollama answers take seconds per word, nvidia-smi shows an idle GPU, or every first request is slow. Check where the model is loaded, then fix drivers, context size, concurrency and keep-alive.
Use Azure AI Content Safety Prompt Shields to detect jailbreak attempts in user prompts and indirect prompt injection in retrieved documents, either through Azure OpenAI content filter configurations or by calling the API directly, then log and act on the results.
When Triton reports a model as UNAVAILABLE, the cause is usually the repository layout, a config.pbtxt mismatch, a missing backend or framework version, or not enough GPU memory. Read the server log and the repository index, then fix the layer that failed.
A loop in an agent or a misused key can spend a month's AI budget in an afternoon. Layer near-real-time token caps at the gateway with budget alerts and automated actions that disable spend, and know how fast each layer really reacts.
Wrap each AI feature in a feature flag so on-call can switch it off in seconds without a deployment. Setup in Azure App Configuration, Python and .NET code with short refresh, a safe degraded response, and the CLI command to pull it.
Put the global and per-application AI kill switches in the gateway, so they work even if an application ignores its own feature flags. Covers an APIM named value with a policy that returns a clean 503, per-product and per-model switches, and the commands to flip them.
When every team builds its own model access, costs, keys and controls scatter. Offer a shared platform with a gateway, approved models, per-team quotas and keys or identities, usage-based showback, built-in guardrails and logging, and a clear path for teams to request new models or capacity.
GenAI workloads need the usual reliability, security, cost and operations checks plus AI-specific ones such as evaluation, grounding, model lifecycle and responsible AI. Use these questions before go-live and yearly, collect evidence, rate risks and keep a short owned action list.
Many assistant needs are covered by existing productivity copilots, low-code agent builders or features inside business applications. Compare adopting, extending and building on fit, data access, control, effort and running cost, and build custom only where the use case needs integration, control or behaviour others can't provide.
Many "agent" projects would be cheaper and more reliable as fixed workflows with LLM steps. Use workflows when the steps are known, agents when the path depends on what is found, give agents the least autonomy that works, and add budgets, approvals and evaluation before going to production.
Without evaluation, every prompt or model change is a guess. Define what quality means per use case, build offline test sets from real traffic, combine rule checks, judge models and human review, gate releases on results, and keep measuring in production with user feedback and sampled reviews.
Serving engines differ in throughput, hardware support, model coverage and operational fit. Match the engine to the environment (data-center GPUs, laptops, CPUs, edge), model formats and features you need (OpenAI-compatible API, structured outputs, LoRA adapters), and benchmark with your traffic before standardising.
The vector store affects retrieval quality, operations effort and cost for years. Decide by scale, the need for hybrid keyword search and filters, where your data already lives, security and networking requirements, and who will operate it. Start simple and move only when evidence says you've outgrown it.
The embedding model decides what retrieval can find, and changing it later means re-embedding everything. Evaluate candidates on your own queries and documents, check language and domain coverage, weigh dimensions and cost against quality, consider hosting constraints, and plan versioning from the start.
Leaderboards don't tell you which model works for your task, data and constraints. Write down the requirements (quality, latency, cost, data handling, deployment), shortlist hosted and open-weight candidates, evaluate them on your own test set, and plan for model changes because today's choice will be replaced.
Poor parsing silently caps RAG quality: tables flattened, headings lost, scanned pages empty. Classify your documents, pick the cheapest parser that preserves the structure each type needs, keep structure as Markdown or JSON for chunking, and measure parsing quality on a sample before indexing everything.
Human review is the main safety control for AI that takes actions, but badly designed reviews turn into rubber-stamping. Decide which actions need approval by risk, show reviewers the evidence they need, route by confidence, measure reviewer agreement and overrides, and reduce review only when data shows it's safe.
Teams collect dozens of GenAI ideas and build the most exciting rather than the most useful. Describe each idea in a standard way, score value, feasibility and risk, check data readiness, start with a few high-value low-risk cases with clear success measures, and stop those that don't prove themselves.
Teams often reach for fine-tuning when they need retrieval, or build RAG when the documents would fit in the prompt. Decide by what the model lacks (facts, behaviour or format), how often the knowledge changes, access control needs and volume, and combine approaches where it makes sense.
GenAI apps add new attack paths on top of normal web risks. Draw the data flows, walk through LLM-specific threats such as prompt injection, sensitive data disclosure, excessive agency and supply chain risks, map each to controls, and revisit the model whenever tools, data sources or models change.
People paste whatever they have into AI tools unless told clearly what's allowed. Map your data classification levels to approved model endpoints and conditions, enforce the mapping technically where you can, make the rules easy to understand, and review when providers, regions or contracts change.
A kill switch stops an AI capability quickly and predictably when it misbehaves, leaks data or burns money. Here are the six levels to design for, from a single tool to the whole platform. Each comes with where it lives, how fast it acts and what users see.
Some failures move faster than an on-call engineer can respond. Pick the signals worth automating (harmful-output rate, blocked-content spikes, write-tool bursts, token rate, groundedness drop), set thresholds from baselines, and wire alerts to a switch with a human confirmation step.
One account is sending jailbreak attempts, scraping your assistant or draining capacity. Identify the caller, block at the narrowest level (user, API key, tenant or IP), tighten their limits, and keep evidence, while everyone else keeps working.
An untested kill switch usually fails. Flags are cached for an hour, a background worker ignores them, or nobody has the permission to flip them. Here's a drill checklist to prove each switch works, measure how long it takes, and fix what the drill finds.
When a key has leaked, an unknown caller is using your model, or the model must be withdrawn, you need to stop every call to the endpoint, not just your app's calls. Here are the identity, key, network and deployment levers on Azure OpenAI and Amazon Bedrock, in order of speed and blast radius.
For open-weight models served with vLLM, TGI or Triton on Kubernetes, you own the off switch. Here are four levers, from blocking external traffic to stopping the GPUs, with commands for each, what keeps running, and how to bring the model back safely.
Output tokens usually cost several times more than input tokens and drive latency. Set max_tokens per feature, ask for the format and length you need, use structured outputs instead of prose for machine consumers, tune reasoning effort on reasoning models, and monitor output length per feature.
Users judge an assistant by how quickly it starts answering and finishes. Break latency into retrieval, model queueing, time to first token and generation, then stream responses, run independent steps in parallel, trim prompts and outputs, place services in the same region and choose models by latency as well as quality.
Fine-tuning jobs often leave GPUs idle while waiting for data or wasting memory on full precision. Profile utilisation first, use bf16 mixed precision, size batches with gradient accumulation, enable gradient checkpointing and parameter-efficient methods when memory is tight, fix the data loader, and checkpoint so spot or preempted capacity is usable.
Provisioned throughput is paid whether used or not, so low utilisation wastes money and saturation causes 429s. Measure utilisation by hour, size PTUs for the steady baseline, send peaks to pay-as-you-go deployments through spillover or a gateway, shift batch work off-peak, and review sizing as traffic changes.
Agents call the model many times per task and resend a growing context each time, so costs can grow quickly. Cap steps and spend per task, return compact tool results, summarise or drop old turns, cache stable instructions, use smaller models for sub-steps, and track cost per completed task.
Logging every prompt and response in full can cost more than the model calls and increases privacy risk. Keep metrics for all requests, sample full traces, cap and redact payloads, keep errors and flagged conversations in full, and tier retention so detail ages out.
Re-processing every document on every run wastes document parsing, OCR and embedding spend. Track content hashes to process only changed documents, skip duplicates and boilerplate, choose parsing and OCR per document type, batch embedding calls, and measure cost per document indexed.
Using the largest model for every request is the most common GenAI overspend. Classify requests by difficulty, route simple ones to small models and hard ones to large models, or cascade by trying cheap first and escalating on low confidence, and prove quality holds with an evaluation set.
Most LLM providers can reuse the processed start of a prompt when it repeats exactly, cutting input cost and time to first token. Put static content first and variable content last, keep the prefix byte-identical between calls, check cache hit fields in responses, and know each provider's rules for minimum length, lifetime and pricing.
Evaluation is essential for safe releases, but running every test with a large judge model on every commit gets expensive. Split suites into fast and full tiers, use rule-based checks before judge models, calibrate smaller judges against a strong one, cache results for unchanged cases and run big suites through batch APIs.
Small models, embedding services and dev notebooks often use a fraction of a GPU while holding the whole device. Measure real GPU use, partition supported GPUs with MIG for isolation, use time-slicing where isolation isn't needed, pack workloads with node pools and autoscaling, and keep large models on dedicated GPUs.
Vector indexes grow with document count times dimensions times bytes per value, and memory is the main cost. Use shorter embeddings where the model supports it, quantize vectors to int8 or binary, rescore top candidates with full-precision vectors, and measure recall so savings don't quietly hurt answer quality.
Speculative decoding uses a cheap draft method to propose several tokens that the main model verifies in one step, cutting inter-token latency when proposals are often accepted. Pick a draft method, measure acceptance rate and latency at your real concurrency, and keep it only where it helps.
Sending 20 chunks "just in case" makes answers slower, more expensive and sometimes worse. Measure retrieval quality, tune how many candidates you retrieve, how many you rerank and how many you send to the model, deduplicate and order context sensibly, and pick the smallest context that keeps answer quality.
A self-hosted model's cost per token depends on how many requests each GPU serves at once without breaking latency targets. Tune GPU memory use and batch limits, enable prefix caching and chunked prefill, choose tensor parallelism only when needed, and load-test against time-to-first-token and inter-token latency SLOs.
Switching an assistant off completely isn't the only option after harmful or wrong answers are reported. Design safe-mode levels in advance (stricter filters, retrieval-only answers, canned responses, human handoff) and switch between them with one flag.
Someone got your assistant to reveal its system prompt, ignore its rules, leak data from retrieved documents or call tools it shouldn't. Contain it, find every conversation affected, judge what was exposed or done, then close the specific hole and the class of problem.
An autonomous agent is deleting records, sending emails or changing systems it shouldn't. Stop new runs, halt runs in progress, cut its tool credentials, contain the damage, then work out what happened from traces before anything is switched back on.
After public network access was disabled, a hub-spoke change or a DNS server move, calls to Azure OpenAI fail with 403 "public access is disabled", timeouts or name resolution errors. Check what the name resolves to from the app's network, fix the private DNS zone links or forwarders, and test from each spoke.
Indexer runs fail with storage or quota errors, vector uploads are rejected, and the RAG assistant stops learning new content. Confirm which limit you hit (total storage, vector index size or document count), buy time, then add partitions, shrink the vector footprint or move to a bigger tier.
Streaming worked in development, but in production users see nothing for 30 seconds and then the whole answer, or the stream stops halfway with a timeout. Find the hop that buffers or times out (NGINX, Application Gateway, APIM, Front Door, corporate proxy) and fix buffering and idle timeouts.
Inference demand rises but new GPU VMs fail to start because the region or zone has no capacity for that size, or you've hit quota. Get capacity from other zones, sizes or regions now, cut demand per request, and set up reservations so baseline capacity is guaranteed.
Inference pods crash, CUDA calls fail, and nvidia-smi errors or hangs. Read the Xid codes in the kernel log, tell software faults from hardware ones, drain and reset or replace the node, and keep the rest of the fleet serving.
New model replicas never start and the scheduler says no node has GPUs, even though GPU nodes exist. Check whether GPUs are advertised, whether taints and selectors match, whether the device plugin or GPU operator is broken, and whether the node pool can scale at all.
Production calls suddenly return deployment or model-not-found errors, or behave differently overnight. The model version reached its retirement date or was auto-upgraded. Restore service on a supported version fast, then put lifecycle tracking in place so it doesn't surprise you again.
Requests to a custom model endpoint fail with 408/504 or upstream timeouts in the morning or during traffic spikes, then recover. New instances are loading a large model while traffic is already routed to them, or the endpoint scaled in too far. Fix minimum capacity, timeouts, readiness checks and autoscale rules.
Someone notices API keys, passwords or customer personal data in prompt logs, tracing tools or evaluation datasets. Stop new capture, rotate exposed secrets, find and purge every copy (logs, traces, caches, evaluation sets, provider-side storage), and fix the pipeline so sensitive data is redacted before it's written.
After a re-index, pipeline change or data refresh, the assistant says "I don't know" far more often or cites the wrong documents. Compare retrieval before and after, find the change (chunking, embeddings, filters, missing documents), roll back with an index alias, and add a retrieval gate to the pipeline.
A pipeline that parses model output (ticket classification, extraction, routing) starts failing on JSON parse or schema validation errors. Find whether it's truncation, model change, prompt drift or input size, switch to schema-enforced output, and add validation, retries and a dead-letter path.
Users can get answers from HR, finance or legal documents they have no access to. Exclude the source from retrieval at query time in minutes, purge it from the index and caches, check what was exposed, and fix permission trimming before re-adding it.
Users wait seconds for the first word, the request queue grows, and GPU utilisation looks fine. Separate queueing from prefill cost from KV cache pressure using vLLM metrics, then apply the fix that matches, whether that's scaling out, limiting long prompts, chunked prefill or prefix caching.
Qdrant, Milvus or Weaviate pods restart with OOMKilled after the collection grew or a bulk load ran. Get search back by adding memory or moving vectors to disk, slow the ingestion down, then size memory properly with quantization and on-disk indexes.
Search throws dimension errors, or still works but returns irrelevant results, after someone changed the embedding deployment, model version or dimensions setting. Find which side changed, restore matching query and document embeddings, and lock the embedding config to the index.
Often you want the agent to keep answering questions but stop it from using one risky tool. Build a tool registry with per-tool enable flags and a read-only mode, enforce it in both the agent and the MCP server, and give the model a clear message when a tool is off.
Many GenAI incidents start with a change, such as a new model version, a reworded system prompt or a new retrieval setting. Make the active versions configuration, not code, so on-call can switch back to the last known-good combination in seconds and with an audit trail.
Small and medium RAG workloads often run a separate vector database next to a PostgreSQL that already holds the metadata. Moving vectors into pgvector removes a system to operate and enables joins and transactional updates. Check the fit, design the table and HNSW index, bulk load, verify recall and switch reads.
Teams with a working MLOps platform for classic models don't need a second platform for LLM apps. Reuse the registry, pipelines and monitoring, and add what's missing - prompt versioning, evaluation datasets and judges, tracing, and token cost tracking - with a release gate that works for both.
Replace an intent-and-decision-tree chatbot with a retrieval-augmented assistant without losing what worked. Mine the old bot's intents and transcripts, build the knowledge index, keep deterministic flows for transactions, evaluate against real questions and roll out side by side.
An incident-summary script that pastes logs into one prompt hits limits fast. Turn it into a bounded agent that fetches what it needs through MCP tools. Start read-only, cap steps and tokens, log every tool call, require approval for actions, and compare results against the old workflow.
Upgrade an intranet, help-centre or ticket search from BM25 keyword matching to hybrid search. Add vector fields, run keyword and vector queries together with rank fusion, rerank the top results, and prove the gain with a relevance test set before switching.
When a high-volume task needs a long prompt full of rules and examples on a large model, a small model fine-tuned with LoRA can match quality at lower latency and cost. Decide if fine-tuning is the right fix, build a training set from logged outputs, train with PEFT, evaluate against the prompt baseline and serve adapters with vLLM.
Service desks spend hours reading, categorising and assigning tickets. Use historical tickets to define labels and measure accuracy, classify with schema-enforced LLM output, keep humans in the loop for low-confidence cases, and move from suggest mode to automatic routing step by step.
OpenAI has deprecated the Assistants API in favour of the Responses API with Conversations. Map assistants, threads, runs and vector stores to the new model, rewrite the call path, move conversation state, re-test tool use and file search, and remove the old polling code.
Teams that started on api.openai.com with personal keys move to Azure OpenAI for private networking, Entra ID authentication, regional data handling and central cost control. Map models to deployments, change the SDK client, switch to managed identity and cut over app by app.
Nightly summarisation, re-classification and embedding backfills often run as loops of synchronous API calls that fight production for quota. Batch inference endpoints take a file of requests and return results within a completion window, at a discount and with separate capacity. Identify candidates, build the JSONL, submit, track, merge results and handle failures.
Models served from a GPU VM with a Flask app and a cron restart are fragile and hard to scale. Move them to managed online endpoints for health checks, autoscaling, blue/green deployments and built-in monitoring. Package the model, define the deployment, test, shift traffic gradually and retire the VMs.
Serving open-weight models at FP16/BF16 uses the most GPU memory per request. Quantizing weights (and optionally the KV cache) to FP8 or INT4 can fit a model on fewer GPUs or serve more concurrent users. Pick the scheme your GPUs support, quantize or use a pre-quantized checkpoint, evaluate quality, load test and roll out behind the gateway.
Most GenAI POCs are a notebook, an API key and a demo. Turn one into a supported service by closing the gaps in order: identity and network, gateway, evaluation, observability, safety, cost limits, resilience and ownership, with a go-live gate at the end.
Runbooks live in wikis, Git repos, old tickets and people's heads. Collect and tag them, index them for retrieval with service and freshness metadata, answer with links to the exact runbook step, surface stale content, and keep the assistant read-only so it guides engineers instead of acting for them.
Self-hosting open-weight models can cut cost at high volume and meet strict data or latency needs, but it adds GPU operations. Decide with numbers, pick a model with an evaluation, size GPUs, serve with vLLM behind an OpenAI-compatible API, and migrate workloads one at a time.
The technical switch is the easy part. Decide in advance who can pull which switch without asking, when they must, how to communicate it, how to restore, and how to keep a record that satisfies auditors and AI governance requirements.
Fix model pulls that fail at startup with 401, 403 gated-repo, timeout or not-in-cache errors, then pin revisions and pre-stage weights so pods never depend on the Hub.
Detect and mask names, emails, phone numbers and internal IDs before text reaches an LLM API or your trace store, with reversible placeholders when the answer needs the real values.
Turn on Bedrock model invocation logging, then use CloudWatch metrics and Logs Insights to find the role, model and team behind a token or cost spike.
Rotate a leaked or expiring Azure OpenAI key with the key1/key2 swap, find who still uses keys, move apps to managed identity and disable key auth.
Put an AI gateway in front of every model endpoint. Covers authentication with managed identity, per-team token limits, usage metrics for chargeback, logging, backend pools for failover, and semantic caching, using Azure API Management's GenAI policies.
Work through model access, IAM actions and resources, inference profiles, region availability, SCPs and VPC endpoint policies when Bedrock calls fail with access or validation errors.
Set up a Bedrock Knowledge Base step by step. Covers model access, the S3 data source, IAM role, vector store choice, chunking options, sync jobs, querying with RetrieveAndGenerate, metadata filters and the operations to plan for.
Lay out a secure Azure AI Foundry foundation. Covers hub-and-spoke placement, private endpoints and DNS for the AI services, storage, search and Key Vault, managed identities and RBAC, policy guardrails, and the shared versus per-app split.
Azure AI Search has no built-in cross-region failover. Plan DR with a second search service in another region, defined as code and kept in sync by indexers or dual-push, then switch app traffic. Includes sync options, a failover runbook and cost notes.
Read indexer execution history, fix data source permissions, document size and parsing errors, skillset and embedding failures, and reset and rerun indexers safely.
Tell a prompt-side 400 content_filter error from a response cut off with finish_reason content_filter, read the per-category annotations, and fix false positives safely.
Create an Azure OpenAI deployment the right way. Covers choosing a deployment type (Standard, Global Standard, Data Zone, Provisioned), managing quota, version upgrade policy, content filters, keyless authentication, and a first test call.
Expose a few well-described, read-only operations tools (service health, recent errors, runbook search) to AI assistants through a Model Context Protocol server in Python, with guardrails.
Readiness checks for an AI assistant that summarises alerts, correlates signals and suggests runbook steps, covering data access, permissions, human approval, evaluation and rollback.
Understand context windows vs output limits, count tokens before calling the model, and trim chat history and RAG context so requests fit and answers aren't truncated.
Build a test set, write a clear grading rubric, calibrate the judge against human labels, and run evaluations in CI so prompt and model changes don't silently reduce quality.
An inventory-driven DR plan for GenAI apps. Covers which components hold state (conversations, prompts, indexes, evaluation sets, fine-tuned models), which are rebuildable from code, the recovery order, and the RTO/RPO trade-offs per component.
Plan capacity for GenAI before it becomes an outage. Covers estimating tokens per minute from user numbers, sizing pay-as-you-go quota versus provisioned throughput, sizing self-hosted GPUs from memory and throughput, cloud GPU quotas and reservations, and monitoring headroom.
Spread LLM traffic across several Azure OpenAI deployments and regions with APIM backend pools. Covers priority-based spillover from PTU to pay-as-you-go, weighted balancing, circuit breakers on 429 and 5xx errors, retries, and testing failover.
Run vLLM model servers on Kubernetes without single points of failure. Covers multiple replicas across zones, readiness probes that wait for the model to load, PodDisruptionBudgets, autoscaling on queue depth, fast model loading, and rolling updates that keep capacity.
Keep GenAI features working when a model or provider has problems. Covers a fallback chain of models, circuit breakers, timeouts per stage, which errors to retry, keeping prompts compatible across models, and graceful degradation for users.
Prepare for the day a whole LLM provider is unavailable, not just one region. Covers multi-provider readiness work, contract and data checks, degraded service modes, a decision tree during the incident, and returning to normal.
Fix the common tool/function calling failures in agent apps, including vague tool descriptions, invalid JSON arguments, missing tool results and loops, with validation and limits in code.
Treat prompts, model versions and retrieval settings like code. Covers prompts in Git, a golden evaluation set, automated evaluations in CI with quality gates, staged rollout, production tracing and a feedback loop.
Why vectors from different embedding models can't be mixed, and a blue-green runbook to re-embed, evaluate and switch a production vector index safely.
Understand driver vs toolkit versions, fix 'CUDA driver version is insufficient' and NVML mismatch errors, and get GPUs working in Docker and Kubernetes.
Set up IAM policies for OCI Generative AI, call the chat API with the right compartment and endpoint, and fix 404 NotAuthorizedOrNotFound, 400 and 429 errors.
Layered controls against direct and indirect prompt injection, covering least-privilege tools, human approval, content isolation, output handling, detection and testing.
Read the OOM message, check for other processes and fragmentation, then reduce memory with batch size, mixed precision, gradient checkpointing and inference_mode.
Recognise the symptoms of chunks that are too small or too large, choose a structure-aware chunking strategy, and test chunk settings against a retrieval benchmark.
Design RAG ingestion and query paths that keep working through failures. Covers event-driven ingestion with queues and dead-letter handling, idempotent chunk IDs, embedding API backoff, index versioning with aliases, and timeouts and fallbacks on the query side.
A practical governance model for GenAI. Covers use-case intake, risk tiering, required controls per tier, model and data approvals, human oversight, red teaming, incident handling and ongoing monitoring, aligned with NIST AI RMF and ISO/IEC 42001 concepts.
A step-by-step runbook for when model calls fail in one region. Covers confirming the scope, switching the gateway to DR deployments, checking quota and quality in the DR region, user communication, and failing back.
Size GPU memory for weights and KV cache, choose vLLM flags for throughput and latency, and watch queue depth, cache usage and time-to-first-token.
Cache LLM answers by meaning, pick a safe similarity threshold, scope the cache by tenant and permissions, invalidate on content changes, and use provider prompt caching too.
Build a private RAG proof of concept on one VM, with a local model served by Ollama and an embedding model, document chunks in PostgreSQL with pgvector, and a small Python script for ingesting and answering questions. Ends with what to change before production.
Stand up a production-minded single-GPU inference server with vLLM. Covers choosing a model that fits, installing drivers and the container, the OpenAI-compatible server, memory flags, a reverse proxy with authentication, systemd, monitoring and load testing.
Run a single-node Qdrant vector database properly. Covers Docker setup with persistent storage, API keys and TLS, collection design with HNSW and quantisation, payload indexes for filters, memory sizing, snapshots and monitoring.
Instrument an LLM app with OpenTelemetry spans for retrieval, model calls and tools, record GenAI semantic convention attributes like tokens and model, and handle prompt privacy.
Make vector search survive node and zone failures. Covers the HA model of Qdrant, Milvus, Weaviate, pgvector, OpenSearch and Azure AI Search, how to size replicas and shards, write consistency settings and what to test.
Choose a DR strategy for vector indexes by comparing rebuild time, snapshot restore and continuous replication. Includes how to estimate rebuild time from embedding throughput, snapshot commands for common engines, and keeping the embedding model version consistent.
Size memory for a vector index, choose M, ef_construction and ef_search, measure recall against exact search, and plan for deletes and re-embedding.
Separate retrieval failures from generation failures using traces, then fix them with better retrieval, stricter grounding prompts, citations, abstention and evaluation.
Why token-per-minute quotas trip before request limits do, how to retry with Retry-After and jitter, and when to spread load across regions or move to provisioned throughput.
What to monitor and version in a retrieval-augmented generation system, from ingestion freshness and embedding drift to retrieval recall, groundedness and prompt injection.
No posts in this category yet.