Home / Tracks / Advanced
Deep dives for experienced engineers: internals, tuning, scale and design trade-offs. 271 posts.
27 posts
How Azure CNI Overlay, flat Azure CNI with a dedicated pod subnet and kubenet differ in IP use, routing and limits; how to size CIDRs for growth; and how to plan egress so clusters do not hit SNAT or IP exhaustion later.
Control where pods land on AKS so a zone or node loss does not take a service down. Topology spread constraints, dedicated node pools with taints, pod anti-affinity, priority and preemption, and how they interact with the cluster autoscaler.
Use data collection rule transformations to drop noisy rows, remove or hash sensitive columns, and route data to cheaper tables before it is stored, with a safe test-and-rollout process.
Use Azure Virtual Network Manager network groups and security admin rules to enforce organisation-wide allow/deny rules across many VNets, understand how they combine with NSGs, and deploy them without cutting off production.
Use what-if output as a pipeline gate, manage resource lifecycles with deployment stacks so removed resources are actually deleted or detached, and protect stack-managed resources from portal edits with deny settings.
Pods in AKS see random 5-second delays or "Temporary failure in name resolution". Prove it's DNS, find whether it's ndots, conntrack races, CoreDNS load or Azure DNS limits, and fix each.
Connections through a pair of third-party firewalls in Azure hang or reset at random. Prove the return path takes a different NVA, then fix UDRs, gateway routes, HA ports and SNAT so each flow stays on one appliance.
Apps fail or slow down with HTTP 429 from Key Vault during peaks or deployments. Identify who is calling, compare with the service limits, then fix with caching, backoff and vault design.
High-volume jobs get 503 ServerBusy, 500 OperationTimedOut or slow blobs from Azure Storage. Tell account-level throttling from hot partitions, then fix naming, spreading, client retries and tier.
Log-based alerts miss incidents because records land minutes after they happened. Measure agent and pipeline latency with ingestion_time() and _TimeReceived, find the slow source, and write alert queries that don't skip late data.
Your UDRs send spoke traffic through the hub firewall, but connections to private endpoints never appear in its logs. Understand the /32 InterfaceEndpoint routes, enable route table policies, and keep return traffic symmetric.
A rolling gateway update overloaded a regional control plane and left some ExpressRoute gateways without connectivity across 33 regions. Customers could not prevent the fault itself, but redundant paths, failover tests and the right alerts decide how much it hurts.
Azure Policy works best as a product with a lifecycle, not a pile of portal assignments. Choose effects deliberately, roll out in audit-first rings, handle exceptions with expiring exemptions, manage definitions and assignments as code, and measure compliance in a way teams can act on.
A design guide for matching each workload to a disaster recovery pattern on Azure. Classify by business impact, translate RTO and RPO into a pattern, map each tier (compute, data, network, identity) to Azure services, and check the hidden dependencies that break recoveries.
A design guide for how container images get from a Dockerfile to production on Azure. Decide registry layout and SKU, import and curate base images instead of pulling from the internet, scan at build and in the registry, sign images and enforce trusted sources at deploy time, and keep the registry tidy.
AKS clusters often run at low real utilisation because pod requests are inflated and nodes never scale down. Measure requests against usage, fix requests with VPA recommendations, tune the cluster autoscaler profile, choose node sizes that pack well, and move tolerant workloads to spot node pools.
Firewall policies collect years of one-off rules, and every environment runs a Premium firewall around the clock. Use Policy Analytics and logs to find unused and overlapping rules, restructure collections with IP Groups, match the SKU to the features you use, and deallocate non-production firewalls out of hours.
Networking charges are spread across many small meters, so they grow unnoticed. Break the bill down by meter, find the flows behind the biggest ones with flow logs, and fix the usual causes - hairpinned traffic through hubs, chatty cross-region replication, oversized gateways and private endpoints nobody uses.
When and how to move a customer-managed hub-spoke network to Azure Virtual WAN. Compare the models, design hubs and routing intent, migrate spokes and branches in phases, and avoid the routing surprises that cause outages.
A practical path from on-premises vSphere to Azure VMware Solution (AVS). Size the private cloud, connect it with ExpressRoute Global Reach, deploy HCX, choose vMotion or bulk migration per wave, and plan what comes after the move.
A phased plan to put Storage, SQL, Key Vault and other PaaS services behind private endpoints. Build DNS first, add endpoints alongside public access, move clients, then disable public network access with policy.
Retire AD FS by moving Microsoft 365 and Azure sign-in to password hash sync or pass-through authentication. Migrate relying-party apps, test with staged rollout, convert the domain and decommission farm servers safely.
Recover AKS workloads after a cluster failure or region outage using GitOps for configuration, AKS Backup for persistent volumes and a warm or cold standby cluster in a second region.
Find out why a node went NotReady (resource pressure, kubelet, network or a VMSS problem) and recover it safely without losing workloads.
Design region-to-region DR for Azure VMs with Site Recovery, including network mapping, recovery plans with scripts, failover, re-protect and failback, and the RPO/RTO you can realistically promise.
Fix common replication health errors, then run a non-disruptive test failover into an isolated network and clean up afterwards.
Build a two-node SQL Server AG across availability zones using a multi-subnet design that removes the need for an Azure load balancer, with a cloud witness and tested failover.
31 posts
Replace dozens of per-project policies with a few attribute-based policies that compare principal tags with resource tags, protect the tags themselves, pass tags from IAM Identity Center, and test the result before rollout.
Publish an internal service to other VPCs and accounts with PrivateLink instead of peering. Build the NLB-backed endpoint service, approve consumers, set up private DNS with domain verification, and debug the usual connection failures.
Deploy baseline resources (roles, Config rules, alarms, logging) to every account and region with service-managed StackSets, roll out with concurrency and failure tolerance, handle drift and failed instances, and avoid the classic StackSet traps.
How SQS FIFO ordering really works, how to choose message group IDs so throughput scales, deduplication windows and IDs, high-throughput mode, Lambda batch behaviour, and why one poison message can block a whole group.
Design Step Functions workflows that survive throttling and partial failures. Retry with backoff and jitter, catch and compensate, make tasks idempotent, and run large fan-outs with Distributed Map tolerated failure thresholds and redrive.
Another account can see your shared bucket, snapshot or AMI but fails with AccessDenied or an instance that terminates at launch. Fix the key policy, the consumer's IAM, grants and the AWS managed key trap.
An admin role still gets AccessDenied. Read the policy type named in the error, walk the IAM evaluation order, decode encoded messages and use CloudTrail and the policy simulator to find the exact deny.
TransactWriteItems calls fail with TransactionCanceledException under load. Read the per-item cancellation reasons, tell conflicts from conditions and throttling, and fix hot items, retries and idempotency.
Retransmits, DNS timeouts or dropped connections on busy EC2 instances while CPU and bandwidth graphs look fine. Read the ENA allowance counters, find which limit you hit and fix it with sizing, connection tracking changes or caching.
Big jobs get 503 Slow Down from S3, or KMS ThrottlingException on encrypted buckets. Find the hot prefixes, understand how S3 scales per prefix, enable S3 Bucket Keys and tune SDK retries.
Traffic through an AWS Transit Gateway and a firewall VPC drops at random or not at all. Check associations and propagations, find blackhole routes, enable appliance mode and use Route Analyzer to prove the path.
EKS automatically upgrades control planes after extended support ends, without a notice, and leaves nodes and add-ons behind. Check which clusters are on 1.31 to 1.34, plan the upgrade path, and run it before the dates rather than reacting afterwards.
Put RDS Proxy in front of an RDS or Aurora database to absorb connection storms, fix unhealthy targets, and find and remove the session pinning that stops multiplexing.
With the Amazon VPC CNI every pod takes a VPC IP. When subnets or per-node ENI slots run out, pods hang with "failed to assign an IP address". Check subnet free IPs and max-pods, then use prefix delegation or secondary CIDRs.
Diagnose a failed Secrets Manager rotation step by step, clear a stuck AWSPENDING version safely, and make sure applications are still using a working password.
A design guide for backups across many AWS accounts. Define backup tiers, enforce them with Organizations backup policies and tags, protect recovery points with vault lock and cross-account or logically air-gapped copies, prove recoverability with automated restore testing and report compliance centrally.
A design guide for how container images reach production on AWS. Structure ECR repositories and access, curate base images through pull-through cache, scan with Amazon Inspector enhanced scanning, make tags immutable, sign images and verify signatures at deploy time on EKS and ECS.
AWS Organizations offers several kinds of guardrails that work at different points. Map each control objective to the right mechanism - SCPs for what principals may do, RCPs for who may access resources, declarative policies for service configuration, proactive hooks at deploy time and detective rules after the fact - then roll them out safely and handle exceptions.
Multi-AZ on paper often fails in practice because recovery depends on launching new capacity or calling control-plane APIs during the event. Design for static stability - pre-provisioned capacity, data planes that keep working, health checks that detect partial failures - and practise moving traffic away from an impaired zone.
A design guide for connecting data centres and offices to AWS. Choose between VPN and Direct Connect by bandwidth, latency consistency and cost, pick a resiliency model that matches the business impact, decide on gateways and routing, and plan encryption, monitoring and failover tests.
EKS clusters waste money when pods over-request, nodes never consolidate and spot is used with too few instance types. Measure requests vs usage, tune Karpenter NodePools for consolidation and disruption budgets, diversify spot across families and sizes, use Graviton where images support it, and protect critical workloads.
Data transfer charges hide inside EC2, RDS and other service lines. Query the Cost and Usage Report by usage type to see which kind of transfer costs most, trace it to resources and flows, and fix the common causes - chatty cross-AZ tiers, needless cross-region copies and egress that should come from a CDN.
Auto Scaling groups often run with a high fixed minimum because scale-out is too slow to trust. Make instances ready faster, use target tracking on the right metric, add predictive scaling for daily patterns, keep warm pools for slow-booting apps, and mix on-demand and spot across instance types.
Move HDFS data and Spark/Hive jobs from an on-premises Hadoop cluster to Amazon EMR and S3. Separate storage from compute, copy data with DistCp or DataSync, move the Hive metastore to Glue Data Catalog and run jobs on transient or serverless compute.
Move workloads from kops, kubeadm or Rancher clusters on EC2 to Amazon EKS. Build the target cluster with IaC, translate add-ons and IAM, autoscale nodes with Karpenter, migrate namespaces with blue/green and retire the old control plane.
Move databases you run yourself on EC2 into RDS or Aurora. Choose between native dump/restore, native replication and AWS DMS with CDC, prepare parameters and users, rehearse, cut over with a short write freeze and keep a rollback path.
Add a secondary region to an Aurora cluster, monitor replication lag, run a planned switchover with no data loss and an unplanned failover with minimal loss, and reconnect applications.
Use AWS Elastic Disaster Recovery to replicate EC2 or on-premises servers to a DR region, configure launch settings, run non-disruptive drills, perform recovery and fail back.
Fix nodes that launch but never appear in kubectl get nodes, by checking the node role mapping or access entries, endpoint reachability, bootstrap and the VPC CNI.
When EBS volumes won't attach, instances won't start or S3 and RDS reads fail with KMS errors, check the key policy, grants, key state and cross-account setup.
Find out who is calling, which policy layer denied the call and why, using STS, CloudTrail, the policy simulator and encoded-message decoding.
30 posts
Build an event-driven remediation loop. An Events rule catches a risky change, an OCI Function using resource principals fixes it and posts a notification. Includes IAM policies, a Python function, testing and the safety rails that stop automation fighting your engineers.
Choose between full clones, metadata clones, clones from backup and refreshable clones on Autonomous Database. Create and refresh them from the CLI, mask data for test, use refreshable clones for reporting offload, and know the limits that bite.
Logs stopped reaching the bucket, stream, function or SIEM. Read the connector's state and error metrics, fix the policy or target, then reactivate and prove data is flowing.
Go beyond "manage all-resources in compartment". Use request.principal, request.operation, target and tag-based conditions to scope access precisely, let teams own resources by tag, and avoid the conditions that silently grant nothing.
Route spoke-to-spoke, on-prem and internet traffic through an OCI Network Firewall in a hub VCN. DRG route tables and import distributions, VCN ingress route tables, symmetric return paths and the checks that prove traffic is really inspected.
Design OCI private DNS for many VCNs and on-prem. Share private views across VCN resolvers, create listening and forwarding endpoints, set forwarding rules in both directions, and debug resolution with dig from the right place.
Queries on Autonomous Database are slow or queue while CPU isn't obviously maxed. Read resmgr waits, check which service (consumer group) each session uses, then fix service choice, shares, concurrency and scaling.
The standby falls minutes or hours behind. Separate transport lag from apply lag, check archive gaps, redo rate versus network bandwidth, and standby I/O, then fix each with Data Guard Broker and OCI settings.
With FastConnect primary and IPSec VPN backup, some flows go out over one path and back over the other, and on-premises firewalls drop them. Learn how OCI and your router choose routes, and set BGP so both directions agree.
Spokes can't reach each other or on-premises, or traffic skips the hub firewall. Trace the DRG attachment's route table, its import distribution and the hub VCN's ingress route table, then fix them step by step.
After enabling source IP preservation on an OCI Network Load Balancer, some clients hang while others work. Understand how replies travel back, fix security rules and routes, and handle clients in the same subnet as the backends.
OCI shows the new VNIC or private IP as attached, but the OS has no address or replies leave on the wrong interface. Configure Oracle Linux and Ubuntu with oci-network-config, nmcli or netplan and policy routing.
New pods in OKE stay in ContainerCreating or Pending with IP allocation errors. Work out whether the pod subnet, the node's VNIC capacity or max-pods setting is the limit, and fix it without rebuilding the cluster.
Read the job log, cancel a stuck job safely, fix permission, conflict and "already exists" errors, and repair the stack state after a partial apply.
A design guide that maps workload tiers to disaster recovery patterns on OCI. Classify workloads by business impact, translate RTO and RPO into backup-restore, pilot light, warm standby or active-active, map compute, data, network and identity to OCI services, and orchestrate with Full Stack DR.
A design guide for how container images reach OKE and Container Instances. Structure OCIR repositories and access, curate base images, scan with the Vulnerability Scanning Service, sign images with Vault keys and enforce signature verification on OKE, and keep repositories clean with retention.
A design guide for connecting on-premises sites to OCI. Choose FastConnect or IPSec VPN by bandwidth and latency needs, pick a redundancy model matching business impact, design DRG routing and BGP preferences, decide on encryption over FastConnect and plan monitoring and failover tests.
Exadata Database Service bills enabled CPU by the hour, so leaving every VM cluster at peak capacity wastes money. Measure CPU per VM cluster and database, scale enabled CPUs online on a schedule, consolidate databases with Resource Manager and instance caging to share capacity safely, and review storage and licensing.
OKE clusters waste money through over-requested pods, fixed-size node pools and expensive shapes for light workloads. Right-size requests, enable the Cluster Autoscaler add-on, choose between managed and virtual nodes per workload, add Ampere and preemptible node pools, and decide where basic vs enhanced clusters make sense.
Restructure a tenancy where resources live in the root compartment or a couple of catch-all compartments. Design the target compartment tree, move resources safely, rebuild policies, and add tagging, quotas, budgets and Cloud Guard.
A physical migration of an on-premises Oracle database to an OCI VM DB system. Build the target, instantiate a standby from an RMAN backup or duplicate, sync with Data Guard over FastConnect, switch over and make OCI the primary.
Plan a datacentre exit to Oracle Cloud VMware Solution. Size the SDDC, plan VLANs and the VCN, connect over FastConnect, migrate with HCX in waves and use OCI-native services next to the SDDC afterwards.
Replace an ageing on-premises Exadata with Exadata Database Service on Dedicated Infrastructure in OCI or Exadata Cloud@Customer in your datacentre. Choose the deployment, size it, migrate databases with Data Guard and adapt operations to the cloud tooling.
Move Oracle databases to Autonomous Database using Zero Downtime Migration (ZDM) logical online migration. Assess with CPAT, prepare source and target, configure Data Pump and GoldenGate replication, rehearse and switch over.
How Oracle Database@Azure works, when it beats Oracle on Azure VMs or a cross-cloud interconnect, how to plan networking and identity, and how to migrate an on-premises Oracle database into it with Data Guard or ZDM.
Move Java EE applications running on WebLogic Server VMs to Kubernetes on OKE. Export the domain with WDT, build an image with WebLogic Image Tool, deploy with the WebLogic Kubernetes Operator and expose it through an OCI load balancer.
Create a cross-region standby for a Base Database Service DB system using the OCI-managed Data Guard association, check transport and apply lag, and perform switchover, failover and reinstate.
Orchestrate cross-region failover of compute, volumes, databases and load balancers with OCI Full Stack Disaster Recovery, using DR protection groups, generated plans, custom steps and prechecks.
Troubleshoot Oracle Container Engine for Kubernetes worker nodes that don't become Ready and pods that won't schedule, from security rules and capacity to pod networking IPs.
Deploy a 2-node RAC VM DB system, configure services and SCAN connections with Application Continuity, do rolling patching, and handle node failures without application outages.
47 posts
Replace copy-pasted tasks and sensor chains with dynamic task mapping and dataset (asset) triggered DAGs. Control fan-out with max_active_tis_per_dag and pools, handle partial failures, and avoid the mapping patterns that overload the scheduler.
How Iceberg partition transforms work, why queries do not need partition columns, how to evolve a table from daily to hourly or add bucketing without rewriting history, and how to check pruning and compact old layouts when you want to.
Move partitions between brokers without hurting producers and consumers. Generate rack-aware plans, throttle replication traffic, monitor progress, handle stuck reassignments, and rebalance leaders after adding or removing brokers.
Producers fail with RecordTooLargeException or MESSAGE_TOO_LARGE. Find which limit is hit, raise the right settings in the right order, or shrink the messages instead.
Read the Snowflake Query Profile to find where time goes, measure partition pruning, decide when a clustering key is worth its cost, and use search optimization for selective lookups instead of over-sizing warehouses.
Prove a slow Spark stage is caused by skew, not too few resources. Read the task-duration distribution, find the hot keys, and choose between AQE skew join, broadcast joins, salting, splitting hot keys and two-phase aggregation.
Tasks using deferrable operators sit in the deferred state for hours. Check the triggerer is running and healthy, find triggers that block the async loop, fix provider packages and scale triggerer capacity.
Copy activities sit in Queued for minutes on a self-hosted integration runtime and nightly loads miss their window. Read the IR metrics, find whether CPU, memory, concurrency or network is the limit, and fix it with tuning and scale-out.
Data written to a Fabric Lakehouse appears in Spark but not in the SQL analytics endpoint, or appears late. Understand the metadata sync, find tables it skipped and trigger a refresh as part of the pipeline.
Consumer groups keep rebalancing, lag grows and messages are processed twice. Find which timeout triggers it, then fix processing time, assignment strategy and pod restarts with static membership or KIP-848.
Exactly-once pipelines fail with fenced producers, or read_committed consumers stop making progress while the topic keeps growing. Diagnose with kafka-transactions.sh, abort hanging transactions safely and fix transactional.id and timeout design.
A Spark job pulling from SQL Server, Oracle, PostgreSQL or MySQL crawls in a single task. Fix it with partitioned reads, sensible bounds, explicit predicates and a larger fetch size.
A stateful streaming job gets slower every day until executors run out of memory. Read the state metrics in query progress, fix watermarks, deduplication and joins so old state is dropped, and move to RocksDB state storage.
Teams that upgraded the Airflow metadata database in place have hit different failures in staging and production. Clean the database, run the migration on a restored copy, fix DAGs with ruff's AIR rules, and only then schedule the real cutover with a tested rollback.
Some teams upgrading from Airflow 3.3.0 to 3.3.1 or 3.3.2 on Kubernetes report every DAG file timing out in the DAG processor, DAGs being deactivated and runs stuck in queued. The root cause isn't confirmed yet. Here's how to tell whether you're hit, what to collect, and how to roll back cleanly.
Kafka 4.0 removed ZooKeeper and old protocol versions, raised Java minimums and removed several configs. Before you upgrade, confirm KRaft mode on 3.3+ metadata, find clients older than 2.1, move brokers to Java 17, and clean configs that no longer exist.
The Schema Registry REST calls you need on call - list subjects, inspect versions and IDs, test compatibility before deploying, change modes and clean up safely after a 409 rejection.
Slow task scheduling, workers blocked by sensors and a busy metadata database are usually design and configuration problems. Make DAG files cheap to parse, set concurrency limits deliberately, use pools to protect shared systems, replace blocking sensors with deferrable operators and keep the metadata database lean.
Most Kafka performance problems are client settings, not broker size. Measure where time goes, tune producer batching, linger and compression, keep idempotence and acks safe, tune consumer fetch and poll settings, scale consumers to partitions and check the broker side last.
Kafka clusters grow disks because every topic keeps data longer than anyone needs, with more partitions than consumers can use. Measure size per topic, set retention and compaction per use case, move long retention to tiered storage, compress, and right-size partition counts.
Dedicated SQL pools bill per DWU-hour whether busy or idle. Pause and scale on a schedule, isolate loads from reporting with workload groups and classifiers, enable result set caching and materialized views for repeated queries, and keep columnstore and statistics healthy so a smaller DWU does the job.
Plan the move from a Synapse dedicated SQL pool to a Fabric Warehouse. Check T-SQL surface differences, use the Fabric migration assistant, move data through OneLake, re-point pipelines and Power BI, and size Fabric capacity.
Plan the move from CDH/HDP/CDP clusters to Databricks on Azure or AWS. Map every Hadoop component to its lakehouse equivalent, migrate data to Delta, convert Hive/Impala SQL and Spark jobs, replace Oozie and Ranger, and run both in parallel before switching.
Decide when streaming is worth it, then move a batch pipeline to Kafka and Structured Streaming into Delta tables. Covers exactly-once sinks, watermarks for late data, stateful aggregations, triggers and operating streams in production.
Replace database triggers and timestamp-based incremental extracts with log-based change data capture. Prepare MySQL, PostgreSQL or SQL Server, deploy Debezium on Kafka Connect, land changes in the lakehouse with MERGE and remove the old triggers.
Move a production Kafka cluster off ZooKeeper before ZooKeeper support disappears. Pre-checks, deploying KRaft controllers in migration mode, moving brokers, finalising, and the rollback window you get before finalisation.
Move Kafka off self-run VMs onto a managed service. Compare MSK, Confluent Cloud and the Event Hubs Kafka endpoint, replicate topics and offsets with MirrorMaker 2 or Cluster Linking, migrate consumers before producers and cut over safely.
A practical migration plan from an appliance data warehouse to Snowflake. Inventory and usage analysis, schema and SQL conversion, historical data transfer, rebuilding loads, testing with reconciliation, cutover of BI tools and decommissioning.
A practical method to speed up slow Databricks jobs and lower their cost. Start from the Spark UI and query profile, then fix the usual causes — shuffle partitions, skew, bad joins, small files, Python UDFs, spill and over-caching — and decide when Photon pays off.
A complete checklist for deploying Databricks the right way on Azure or AWS. Covers account and workspace design, networking and private connectivity, identity, Unity Catalog, storage, compute governance, security, CI/CD, monitoring, cost, and DR, with the decision for each item.
The cutover steps for one migration wave from a source Databricks workspace to the target tenant. Covers pre-checks, freezing the source, final data sync, enabling target jobs, switching consumers, validation, rollback and decommissioning the source.
Plan a move of Azure Databricks (or a Databricks account on AWS) from one tenant or account to another after a merger, divestiture or tenant consolidation. Covers why workspaces can't simply be moved, what has to be rebuilt versus copied, the target design, phases and the main risks.
Two jobs writing the same Delta table and one fails with ConcurrentAppendException or a sibling. Find the other writer, then fix it with predicates, row-level concurrency or scheduling.
Map and re-provision every identity when Databricks moves to another Entra tenant or Databricks account. Covers building the identity map, SCIM or automatic identity management, recreating service principals and secrets, fixing ownership and run-as identities, and handling guest accounts.
Move code and configuration to a new Databricks workspace in another tenant or account. Covers Git folders, export/import of notebooks, the Terraform exporter, generating Asset Bundles from existing jobs, cluster policies and warehouses, SQL queries and dashboards, and MLflow models and experiments.
Rebuild Unity Catalog in the target tenant or account with the same (or improved) structure. Covers metastore creation, access connectors and storage credentials, external locations, catalogs and schemas as code, translating grants to new groups, volumes, and Hive metastore leftovers.
Choose and run the right method to copy lakehouse data into the target tenant. Covers re-pointing to existing storage, copying Delta folders with azcopy or S3 replication and re-registering tables, Databricks-to-Databricks Delta Sharing with CTAS, incremental sync, and validation.
Run Airflow without single points of failure using multiple active schedulers, a highly available PostgreSQL metadata database, redundant webservers and workers on Kubernetes, and remote logs.
Design Databricks DR as two workspaces in two regions. Covers what has to be replicated (code, jobs, data, catalog metadata, secrets), how to sync each piece, and a failover runbook with realistic RTO and RPO.
Check connector status, source-side replication slots and binlogs, and the sink, to find where change-data-capture lag builds up and recover without losing changes.
Find which topics fill the broker disks, free space safely by lowering retention, recover an offline broker, and prevent it with quotas, alerts and tiered storage.
Measure lag per partition, tell slow processing from rebalance storms and hot partitions, and reset offsets safely when you need to.
Replicate Kafka topics and consumer group offsets to a DR cluster with MirrorMaker 2. Covers naming policies, offset translation, monitoring lag, and a consumer failover runbook.
Configure Kafka so that a broker or availability zone can fail without data loss or downtime, with the right replication settings, rack awareness, controller quorum, producer settings and monitoring.
Replicate Snowflake databases and account objects to another region or cloud with failover groups, refresh them on a schedule, redirect clients with connection URLs, and run a failover drill.
Find skewed tables and expensive shuffle and broadcast steps in a Synapse dedicated SQL pool, and fix them with the right distribution, statistics and columnstore health.
Use the run page, driver logs and Spark UI to tell driver OOM from executor OOM, find skewed partitions and repair only the tasks that failed.
26 posts
Build control flow inside Databricks Jobs instead of an external orchestrator. Pass values between tasks, branch with if/else condition tasks, fan out with for each tasks and concurrency limits, use run-if rules for cleanup, and define it all in an asset bundle.
Split Azure Databricks Terraform into account, workspace and data layers, configure providers with Entra ID auth, create storage credentials, external locations, catalogs and grants, and avoid the state and permission traps that make plans destructive.
Enable change data feed on Delta tables, read inserts, updates and deletes by version or as a stream, apply them downstream with MERGE, handle retention and schema changes, and know when CDF is the wrong tool.
Use expectations in Lakeflow Declarative Pipelines (formerly Delta Live Tables) to warn, drop or fail on bad data, route failing rows to a quarantine table instead of losing them, share rule sets across tables, and report quality trends from the event log.
Use shallow clones for fast, cheap test copies and deep clones for independent copies, refresh them incrementally, mask sensitive columns, keep them inside Unity Catalog permissions, and avoid the shallow-clone trap where VACUUM on the source breaks the clone.
After locking an ADLS Gen2 account to selected networks, Databricks gets 403 "This request is not authorized to perform this operation". Work out which compute path is blocked and allow each one correctly.
A Delta table written by Azure Databricks suddenly fails in Synapse serverless, Power BI, Trino or another Spark with protocol or unsupported feature errors. Find which feature changed the protocol, then downgrade it safely or give readers another path.
Big jobs retry stages over and over with FetchFailedException or ExecutorLostFailure. Work out whether executors died from memory, spot eviction or node loss, and fix partition sizing, memory, skew and spot settings.
Streaming upserts with foreachBatch and MERGE produce duplicates after failures, fail on duplicate keys within a batch, or slow down as the target grows. Make the batch idempotent, deduplicate correctly and give MERGE pruning to work with.
CDC pipelines built with AUTO CDC (formerly APPLY CHANGES INTO) keep stale values, create duplicate SCD2 rows or resurrect deleted keys. Fix sequence_by, delete handling, history tracking and full-refresh risks.
Creating an external table, volume or external location in Azure Databricks fails because the path overlaps another securable. Find what owns the path, understand the rules, and lay out ADLS containers so it can't happen again.
Several 2026 fixes in Databricks Runtime 18 change query results rather than throwing errors - NATURAL JOIN now matches columns case-insensitively, LEFT JOIN LATERAL with EXPLODE keeps rows it used to drop, and windowed min/max on BinaryType columns now returns per-frame values. Find the queries that could be affected and decide whether to accept the new results.
Since the 24 July 2026 maintenance update for Databricks Runtime 18 LTS and 18.2, legacy DStream jobs that put custom or third-party types in closures must list those classes in spark.streaming.checkpoint.allowedClasses before checkpoint recovery. Find the classes, set the config, and plan the move to Structured Streaming.
Copy-paste SQL against the Lakeflow Declarative Pipelines event log to find why an update failed, which expectations dropped rows, where backlog is building and what feeds what.
Writing row filters and column masks table by table doesn't scale. Define a classification scheme as governed tags, tag columns automatically and by review, attach attribute-based access control policies to tags at catalog or schema level, and monitor for untagged sensitive data.
Serverless compute runs in Databricks' account, not your VNet, so classic VNet injection rules don't apply. Use network connectivity configurations for stable egress identities and private endpoints to your resources, control outbound access with serverless network policies, and design firewall rules on storage and other services around them.
Databricks generates detailed audit events, but they only help if someone looks. Decide between system audit tables and diagnostic settings to Log Analytics/Sentinel, collect the event categories that matter, build detections for high-risk actions, and set retention that meets audit needs.
Unity Catalog allows one metastore per region per account, so multi-region platforms need a plan for where data lives, how it's shared and how governance stays consistent. Decide which regions host which data, share across metastores with Delta Sharing, manage permissions as code, and plan DR separately from day-to-day sharing.
Plan the exit from HDInsight clusters. Map Spark, Hive/LLAP, Kafka, HBase and Oozie workloads to Databricks and Azure services, reuse data already in ADLS, migrate the Hive metastore to Unity Catalog and decommission clusters.
Replace Entra ID credential passthrough and Hive table access control with Unity Catalog grants, external locations and row/column controls. Map ADLS ACLs to UC privileges, test with real users and switch clusters to UC access modes.
Harden an Azure Databricks deployment that started with the managed VNet and public access. Plan the new network, build a VNet-injected workspace with secure cluster connectivity and Private Link, migrate content and switch users.
Split an all-in-one workspace into separate dev, test and prod workspaces on one Unity Catalog metastore. Use workspace-catalog bindings, environment catalogs, service principals and bundles so code moves between environments and data access doesn't leak.
Move tables, permissions, jobs and code from the workspace Hive metastore to Unity Catalog using Databricks Labs UCX. Assessment, group migration, storage credentials and external locations, table upgrade, code changes and cutover.
A reusable design for a Delta lakehouse on Azure Databricks. Covers how to lay out Unity Catalog catalogs and schemas, what belongs in bronze, silver and gold, standard loading patterns (append, MERGE, SCD2, snapshot), table settings, data quality, compute per workload, and how to publish gold tables as governed data products.
Who holds the encryption keys, which security controls to switch on, how to find the slow step in a query plan, and how to publish and apply changes with Change Data Feed and AUTO CDC.
When a Bloom filter index really speeds up equality lookups on high-cardinality columns, how to size fpp and numItems, how to prove it works, and why liquid clustering usually comes first.
57 posts
Change big MySQL tables without downtime. Know which ALTERs are INSTANT, INPLACE or COPY, protect against metadata lock pile-ups, and use gh-ost or pt-online-schema-change with throttling, replica lag checks and safe cut-over.
Use SQL plan baselines to stop plan regressions after statistics changes, patches and upgrades. Load known-good plans from the cursor cache or AWR, fix one plan in an emergency, let the evolve task accept better plans, and manage baselines over time.
Find integer keys close to 2,147,483,647, buy time when inserts already fail, and convert the column to bigint with either a rewrite or an online add-backfill-swap.
When EXPLAIN shows estimated rows far from actual rows, PostgreSQL picks bad joins and scans. Find misestimates, raise per-column statistics targets, add extended statistics for correlated columns and functional dependencies, and know what the planner still cannot see.
How keys map to hash slots and shards, how to add or remove shards and move slots safely with redis-cli, why multi-key commands fail across slots, and how to find hot keys and big keys that make one shard the bottleneck.
Recognise parameter-sensitive plans, prove it with Query Store and plan cache data, and pick the right fix. Parameter Sensitive Plan optimization in SQL Server 2022, OPTIMIZE FOR, RECOMPILE, plan forcing and Query Store hints, with the trade-offs of each.
One shard holds most of the data or takes most of the writes. Read balancer status, find jumbo chunks and monotonic shard keys, then fix with balancer windows, clearing jumbo flags, refining or resharding the key.
A schema change hangs and every query on that table queues behind it. Find the transaction holding the metadata lock with performance_schema, clear the pile-up, and run future DDL with lock timeouts, INSTANT/INPLACE or online schema change tools.
CPU jumps and sessions pile up on library cache and cursor mutex waits. Find the SQL with thousands of child cursors, read the reasons in V$SQL_SHARED_CURSOR, and fix bind handling, hard parsing and hot objects.
Throughput collapses on the primary or a hot standby with LWLock waits on subtransaction SLRU buffers. Find the code that creates subtransactions, the long transaction that turns them into a problem, and fix both.
An AG secondary lags minutes or hours behind, reports on the readable secondary show old data, and failover RTO is at risk. Separate send queue from redo queue, find a blocked redo thread and fix I/O, workload and redo settings.
New connections time out and the server looks hung while CPU is low. Confirm worker thread exhaustion, connect through the Dedicated Admin Connection, find the blocking chain or parallelism storm, and fix it without blindly raising max worker threads.
Since the September 2026 SQL Server updates, querying sys.dm_exec_requests while a database is recovering (after a restart or failover) can raise an access violation and write a dump. Monitoring tools trigger it most. Choose between trace flag 4696, pausing those queries during recovery, or removing the update.
Route ApplicationIntent=ReadOnly connections through the AG listener to readable secondaries, load-balance them, test with sqlcmd and fix the usual reasons reads still land on the primary.
Redis is usually sub-millisecond. When p99 jumps, use SLOWLOG, the latency monitor and INFO to find the blocking command, fork, fsync or host problem, then fix it.
The shard key decides whether a sharded cluster spreads load evenly or piles it onto one shard. Evaluate candidates on cardinality, frequency and monotonic growth, make sure common queries include the key, prefer compound or hashed keys where needed, and know your resharding options before you commit.
SaaS-style applications must keep tenants isolated without making operations impossible. Compare database-per-tenant, schema-per-tenant and shared tables with a tenant key, enforce isolation with row-level security where tables are shared, and plan for noisy neighbours, per-tenant restores and growth.
A dependable file-transfer job. A PowerShell step uses the WinSCP .NET assembly to download new files, verify them, archive them on the remote side, record each file in a control table and fail cleanly with a clear message when anything goes wrong.
Copy SQL Server Agent history from msdb into a central reporting database with an incremental, watermark-based load before Agent purges it. Covers a simple star schema, the load procedure, collecting from many servers and the queries it makes possible.
Large, growing tables slow queries, backups and purges. Partition by date or another access key so queries prune and old data drops instantly, compress cold partitions, move history to cheaper tablespaces, and convert online where possible. Check licensing for Partitioning and Advanced Compression before using them.
Huge time-series and event tables make vacuum slow, purges painful and indexes bloated. Partition by time with declarative partitioning, automate partition creation, drop or detach old partitions instead of deleting rows, and migrate existing tables with minimal downtime.
Agent has no built-in dependency between jobs, so people chain them with sp_start_job or guess at schedule gaps. Instead, keep dependencies in a table and add a gate step that checks predecessors succeeded for today's batch. Each job can then wait, skip or fail, and the whole chain is visible in one query.
Oracle 19c is the last release that supports the non-CDB architecture, so every database moving to 23ai must become a pluggable database. Check compatibility, describe the non-CDB, plug it into a CDB with NOCOPY or COPY, run noncdb_to_pdb.sql, and fix services and backups afterwards.
Data Pump is too slow for multi-terabyte databases moving from big-endian UNIX to Linux. Cross-platform transportable tablespaces with incremental backups keep the target in step while the source stays online, so the outage is a short final incremental plus a metadata import.
Move a large self-managed or RDS MySQL database to Aurora MySQL with minimal downtime. From RDS, use an Aurora read replica. From EC2 or on-premises, restore a Percona XtraBackup from S3 and then replicate binlogs until cut-over.
Move an Oracle database to RDS for Oracle when you want a managed service but must stay on Oracle. Check what RDS doesn't allow, size and license the instance, load with Data Pump through S3, catch up with DMS CDC, then cut over.
Move an Oracle schema and its data to PostgreSQL (self-managed, Azure Flexible Server or RDS/Aurora). Estimate effort with the Ora2Pg report, convert schema and PL/SQL, load data, and handle the differences that cause most bugs. Then run a parallel validation before switching.
pg_upgrade needs an outage and an in-place change. Instead, build a new cluster on the new major version, replicate into it with logical replication, test it under real data, then switch over in minutes with an easy way back.
Use AutoUpgrade to analyse, fix up and upgrade 19c databases to 23ai, including converting non-CDBs into PDBs. Covers the config file, the analyze, fixups and deploy modes, restore points for fallback, and the checks to run after the upgrade.
SQL Server Agent records that a job changed but not who changed it or what changed. Use a database audit specification on msdb's job procedures to capture who did it, keep script snapshots to capture what changed, and know the limits of triggers on msdb tables.
The pg_wal directory keeps growing until the volume fills. Find what is holding WAL back, usually a stale replication slot or a failing archive_command, release it safely and cap it.
A practical map of the on-premises SQL Server features that block or complicate moving to Azure SQL Managed Instance or Azure SQL Database, with the replacement for each and example code.
Move databases from Amazon RDS for SQL Server to Azure SQL MI, for example after a consolidation decision or an acquisition. Covers native backups from RDS to S3, copying to Azure Blob Storage with azcopy, restoring with Log Replay Service, TDE and identity considerations, and cutover.
Use an Always On availability group to move Tier 1 SQL Server workloads to Azure with seconds of downtime. Covers network and cluster design across sites, adding an Azure VM replica (or a distributed AG), using it for DR first, and the planned failover that completes the migration.
Plan the two areas that delay most SQL Server moves to Azure. Covers mapping Windows logins and groups to Microsoft Entra ID, service accounts to managed identities, MI Windows authentication with Kerberos, private endpoints vs VNet-injected MI, DNS and firewall design.
SQL Server Agent publishes its own Windows performance counters for jobs, steps and alerts. Find them, collect them with Get-Counter and logman, and alert on failed jobs and success rate. Then go further and measure the CPU, reads and duration each running job step uses through DMVs.
Use AWS Database Migration Service to copy SQL Server data to RDS or EC2 and keep it in sync with change data capture until cutover. Covers source prerequisites (MS-CDC or replication), schema-first approach, task settings for LOBs, validation, monitoring latency and cutover.
Use the Managed Instance link, which is built on distributed availability groups, to replicate on-premises databases to Azure SQL MI in near real time, cut over with a planned failover in seconds, and keep the option to fail back with SQL Server 2022.
Use Babelfish for Aurora PostgreSQL to move SQL Server applications off SQL Server licences with fewer code changes. Covers Babelfish Compass assessment, migration modes, creating the cluster, moving schema and data, connecting apps over TDS, and testing gaps.
Size Azure SQL, RDS and cloud VMs from real data rather than server specs. Covers which counters to collect and for how long, how to read CPU, memory, IOPS, throughput and log rate, and how to map them to service tiers and instance families.
Rebuild a shared-storage SQL Server Failover Cluster Instance on EC2 using Amazon FSx for Windows File Server (SMB) or FSx for NetApp ONTAP (iSCSI) across two Availability Zones. Covers when FCI beats an AG, cluster networking on AWS, storage choice, and migration steps.
RDS Custom gives a managed SQL Server with OS and sysadmin access for apps that can't run on standard RDS. Covers when to choose it, prerequisites, the shared responsibility model, migrating with native backups, automation pause for customisations, and HA.
Check replica and database state with DMVs, interpret send and redo queues, and resume or reseed a database that is NOT SYNCHRONIZING or SUSPENDED.
Generate AWR and ASH reports for the right window, then read DB time, top wait events, load profile and top SQL to find the real bottleneck instead of guessing.
Build a three-node MySQL InnoDB Cluster with MySQL Shell, put MySQL Router in front of the applications, understand single-primary failover and quorum, and run status checks, switchovers and recovery.
Read SHOW REPLICA STATUS correctly, then fix the usual causes of lag, from single-threaded appliers and missing primary keys to large transactions.
Run Oracle Data Guard role changes safely with DGMGRL. Covers health checks, protection modes, a planned switchover, an emergency failover, reinstating the old primary with Flashback Database, and Fast-Start Failover.
Install Oracle Database 19c or 23ai as a single instance on Oracle Linux. Covers the preinstall RPM, storage layout, silent install, creating a CDB/PDB, memory settings, archivelog mode, RMAN backups and basic security.
Get real HA out of Oracle RAC. Covers database services with preferred and available instances, SCAN listeners, client failover (FAN, TAC and Application Continuity), rolling maintenance, health checks and common failure scenarios.
Use RMAN's Data Recovery Advisor and validate commands, restore a single lost datafile online, and perform full and point-in-time recovery of a database.
Protect PostgreSQL against site or region loss with a streaming standby in the DR region and a pgBackRest repository there too. Covers a Patroni standby cluster, promoting the standby, PITR from object storage, and failback.
Build a three-node PostgreSQL cluster with automatic failover using Patroni and etcd, route clients with HAProxy or a VIP, choose synchronous mode settings, and run switchovers and health checks.
Prepare, check, run and validate a PostgreSQL major version upgrade with pg_upgrade, including extensions, replicas, statistics and a rollback plan.
Choose and build a SQL Server DR option for a second site or region. Covers an asynchronous AG replica, distributed availability groups, and log shipping, with RPO/RTO, network needs and a forced-failover runbook.
Choose between an FCI and an AG for SQL Server high availability. Compares what each protects, storage, licensing and failover behaviour, then walks through building a two-replica synchronous AG with a listener and checking its health.
Identify which sessions and objects are filling tempdb, recognise PAGELATCH allocation contention, and configure tempdb files correctly.
Find tables with dead tuples, the long transactions that block vacuum, and transaction ID wraparound risk, then tune autovacuum for each table.
53 posts
Improve recall on vague, conversational and multi-part questions by transforming the query before retrieval. Implement standalone-question rewriting, multi-query with reciprocal rank fusion, HyDE and decomposition, then prove the gain with a retrieval evaluation set and keep latency in budget.
CPU-based HPA does not work for GPU inference. Scale vLLM or TGI replicas on waiting requests and KV cache usage with KEDA and Prometheus, pair it with the cluster autoscaler or Karpenter for GPU nodes, and hide multi-minute cold starts with warm capacity.
A step-by-step checklist for rolling a new NVIDIA driver across GPU node pools serving LLMs - compatibility checks, capacity headroom, GPU Operator upgrade policy, canary nodes, validation and rollback.
Make a RAG assistant return only what each user may read. Capture ACLs at ingestion, store them as filterable metadata, apply identity-based filters at query time, keep permissions fresh, and test for leaks before and after go-live.
Treat every MCP tool as an API that an untrusted prompt can call. Authenticate clients with OAuth for remote servers, scope tools per role, separate read and write tools, validate arguments, require confirmation for destructive actions, and log every call.
Understand how vLLM splits GPU memory between weights, activations and KV cache, read the startup log to see how many tokens fit, size max-model-len and max-num-seqs for your traffic, use FP8 KV cache and prefix caching, and spot preemption before users feel it.
An agent creates two tickets, sends the same email twice or repeats a refund. Trace where the duplicate came from (HTTP retry, model re-planning, framework replay) and make side-effecting tools idempotent with keys, dedup stores and confirmation steps.
Asking for 10 nearest neighbours with a tenant or date filter returns 2, or misses the best match. Understand how each engine applies filters to approximate search, then fix it with the right filter mode, iterative scans, partial indexes or partitioning.
Scaling out an LLM deployment is useless when each new pod spends most of its time downloading and loading 100+ GB of weights. Measure where the time goes, then fix it with caching, faster storage, streaming loaders, image design and probes.
After deleting or upserting millions of entities, Milvus row counts look wrong, object storage keeps growing, memory doesn't drop and searches get slower. Understand how Milvus deletes work, check segment and compaction state, and get storage and latency back under control.
A tensor-parallel model server hangs at startup or under load with NCCL watchdog timeouts. Turn on NCCL debug logging, check shared memory, PCIe/NVLink topology and P2P, pick the right network interface, and prove the fix with nccl-tests.
An HNSW build on 10 million embeddings runs all night, or slows to a crawl halfway with a "graph no longer fits into maintenance_work_mem" notice. Size memory for the graph, use parallel builds, watch progress, and plan loads so the build happens once.
A document store that re-embeds or deletes chunks every day ends up with an HNSW index twice the expected size, autovacuum running for hours on it and slower searches. Measure the bloat, reindex before vacuum, and change the write pattern that causes it.
Unfiltered Qdrant searches take a few milliseconds but the same search with a tenant or category filter takes hundreds, or returns fewer results than expected. Add the right payload indexes, mark tenant fields, and understand how Qdrant plans filtered searches.
A single Qdrant node is running out of memory or is a single point of failure. Turn it into a cluster, choose shard and replica counts you won't regret, balance data onto new nodes by moving and replicating shards, and handle a failed peer.
Tests that compare LLM outputs flap, and users get different answers to identical questions even with temperature 0. Understand batching, floating point and routing effects, use seeds and fingerprints where they exist, and design tests and caches that tolerate variation.
One base model with dozens of fine-tuned LoRA adapters saves GPUs, until adapters fail to load, requests slow down or memory runs out. Fix rank and module mismatches, size max-loras and CPU cache, and load adapters dynamically and safely.
In a three-node Weaviate cluster one node restarts and writes start failing, or different requests return different results. Understand replication factor and consistency levels, choose settings for reads and writes, and handle node failures, repairs and rolling upgrades.
For about six hours, Azure OpenAI and Foundry requests in Sweden Central failed intermittently. The incident review recommends retries with backoff, BCDR planning and granular monitoring. Put a second region behind API Management with a circuit breaker, keep retries bounded, and measure per-backend error rates.
Vectors from different embedding models can't be mixed, so when an embedding model retires (Cohere embed v3 on Foundry is deprecated in November 2026 and retires in February 2027) every document must be re-embedded. Build a second index in parallel, compare retrieval quality on a test set, then switch the query path in one step.
When Triton reports a model as UNAVAILABLE, the cause is usually the repository layout, a config.pbtxt mismatch, a missing backend or framework version, or not enough GPU memory. Read the server log and the repository index, then fix the layer that failed.
When every team builds its own model access, costs, keys and controls scatter. Offer a shared platform with a gateway, approved models, per-team quotas and keys or identities, usage-based showback, built-in guardrails and logging, and a clear path for teams to request new models or capacity.
Some failures move faster than an on-call engineer can respond. Pick the signals worth automating (harmful-output rate, blocked-content spikes, write-tool bursts, token rate, groundedness drop), set thresholds from baselines, and wire alerts to a switch with a human confirmation step.
When a key has leaked, an unknown caller is using your model, or the model must be withdrawn, you need to stop every call to the endpoint, not just your app's calls. Here are the identity, key, network and deployment levers on Azure OpenAI and Amazon Bedrock, in order of speed and blast radius.
For open-weight models served with vLLM, TGI or Triton on Kubernetes, you own the off switch. Here are four levers, from blocking external traffic to stopping the GPUs, with commands for each, what keeps running, and how to bring the model back safely.
Fine-tuning jobs often leave GPUs idle while waiting for data or wasting memory on full precision. Profile utilisation first, use bf16 mixed precision, size batches with gradient accumulation, enable gradient checkpointing and parameter-efficient methods when memory is tight, fix the data loader, and checkpoint so spot or preempted capacity is usable.
Provisioned throughput is paid whether used or not, so low utilisation wastes money and saturation causes 429s. Measure utilisation by hour, size PTUs for the steady baseline, send peaks to pay-as-you-go deployments through spillover or a gateway, shift batch work off-peak, and review sizing as traffic changes.
Small models, embedding services and dev notebooks often use a fraction of a GPU while holding the whole device. Measure real GPU use, partition supported GPUs with MIG for isolation, use time-slicing where isolation isn't needed, pack workloads with node pools and autoscaling, and keep large models on dedicated GPUs.
Vector indexes grow with document count times dimensions times bytes per value, and memory is the main cost. Use shorter embeddings where the model supports it, quantize vectors to int8 or binary, rescore top candidates with full-precision vectors, and measure recall so savings don't quietly hurt answer quality.
Speculative decoding uses a cheap draft method to propose several tokens that the main model verifies in one step, cutting inter-token latency when proposals are often accepted. Pick a draft method, measure acceptance rate and latency at your real concurrency, and keep it only where it helps.
A self-hosted model's cost per token depends on how many requests each GPU serves at once without breaking latency targets. Tune GPU memory use and batch limits, enable prefix caching and chunked prefill, choose tensor parallelism only when needed, and load-test against time-to-first-token and inter-token latency SLOs.
Users can get answers from HR, finance or legal documents they have no access to. Exclude the source from retrieval at query time in minutes, purge it from the index and caches, check what was exposed, and fix permission trimming before re-adding it.
Often you want the agent to keep answering questions but stop it from using one risky tool. Build a tool registry with per-tool enable flags and a read-only mode, enforce it in both the agent and the MCP server, and give the model a clear message when a tool is off.
Small and medium RAG workloads often run a separate vector database next to a PostgreSQL that already holds the metadata. Moving vectors into pgvector removes a system to operate and enables joins and transactional updates. Check the fit, design the table and HNSW index, bulk load, verify recall and switch reads.
Teams with a working MLOps platform for classic models don't need a second platform for LLM apps. Reuse the registry, pipelines and monitoring, and add what's missing - prompt versioning, evaluation datasets and judges, tracing, and token cost tracking - with a release gate that works for both.
An incident-summary script that pastes logs into one prompt hits limits fast. Turn it into a bounded agent that fetches what it needs through MCP tools. Start read-only, cap steps and tokens, log every tool call, require approval for actions, and compare results against the old workflow.
When a high-volume task needs a long prompt full of rules and examples on a large model, a small model fine-tuned with LoRA can match quality at lower latency and cost. Decide if fine-tuning is the right fix, build a training set from logged outputs, train with PEFT, evaluate against the prompt baseline and serve adapters with vLLM.
Models served from a GPU VM with a Flask app and a cron restart are fragile and hard to scale. Move them to managed online endpoints for health checks, autoscaling, blue/green deployments and built-in monitoring. Package the model, define the deployment, test, shift traffic gradually and retire the VMs.
Serving open-weight models at FP16/BF16 uses the most GPU memory per request. Quantizing weights (and optionally the KV cache) to FP8 or INT4 can fit a model on fewer GPUs or serve more concurrent users. Pick the scheme your GPUs support, quantize or use a pre-quantized checkpoint, evaluate quality, load test and roll out behind the gateway.
Self-hosting open-weight models can cut cost at high volume and meet strict data or latency needs, but it adds GPU operations. Decide with numbers, pick a model with an evaluation, size GPUs, serve with vLLM behind an OpenAI-compatible API, and migrate workloads one at a time.
Put an AI gateway in front of every model endpoint. Covers authentication with managed identity, per-team token limits, usage metrics for chargeback, logging, backend pools for failover, and semantic caching, using Azure API Management's GenAI policies.
Lay out a secure Azure AI Foundry foundation. Covers hub-and-spoke placement, private endpoints and DNS for the AI services, storage, search and Key Vault, managed identities and RBAC, policy guardrails, and the shared versus per-app split.
Azure AI Search has no built-in cross-region failover. Plan DR with a second search service in another region, defined as code and kept in sync by indexers or dual-push, then switch app traffic. Includes sync options, a failover runbook and cost notes.
Plan capacity for GenAI before it becomes an outage. Covers estimating tokens per minute from user numbers, sizing pay-as-you-go quota versus provisioned throughput, sizing self-hosted GPUs from memory and throughput, cloud GPU quotas and reservations, and monitoring headroom.
Spread LLM traffic across several Azure OpenAI deployments and regions with APIM backend pools. Covers priority-based spillover from PTU to pay-as-you-go, weighted balancing, circuit breakers on 429 and 5xx errors, retries, and testing failover.
Run vLLM model servers on Kubernetes without single points of failure. Covers multiple replicas across zones, readiness probes that wait for the model to load, PodDisruptionBudgets, autoscaling on queue depth, fast model loading, and rolling updates that keep capacity.
Why vectors from different embedding models can't be mixed, and a blue-green runbook to re-embed, evaluate and switch a production vector index safely.
Layered controls against direct and indirect prompt injection, covering least-privilege tools, human approval, content isolation, output handling, detection and testing.
Design RAG ingestion and query paths that keep working through failures. Covers event-driven ingestion with queues and dead-letter handling, idempotent chunk IDs, embedding API backoff, index versioning with aliases, and timeouts and fallbacks on the query side.
Size GPU memory for weights and KV cache, choose vLLM flags for throughput and latency, and watch queue depth, cache usage and time-to-first-token.
Make vector search survive node and zone failures. Covers the HA model of Qdrant, Milvus, Weaviate, pgvector, OpenSearch and Azure AI Search, how to size replicas and shards, write consistency settings and what to test.
Choose a DR strategy for vector indexes by comparing rebuild time, snapshot restore and continuous replication. Includes how to estimate rebuild time from embedding throughput, snapshot commands for common engines, and keeping the embedding model version consistent.
Size memory for a vector index, choose M, ef_construction and ef_search, measure recall against exact search, and plan for deletes and re-embedding.