Home / Databricks on Azure
Azure Databricks design, security, ingestion, storage, tables and pipelines with Data Factory. 113 posts.
Build control flow inside Databricks Jobs instead of an external orchestrator. Pass values between tasks, branch with if/else condition tasks, fan out with for each tasks and concurrency limits, use run-if rules for cleanup, and define it all in an asset bundle.
Split Azure Databricks Terraform into account, workspace and data layers, configure providers with Entra ID auth, create storage credentials, external locations, catalogs and grants, and avoid the state and permission traps that make plans destructive.
Databricks spend shows as untagged in Azure cost analysis or chargeback reports. Find which tag path failed (pool, workspace, conflict, restart, serverless) and fix it.
A guided first hour in an Azure Databricks workspace. What each sidebar area is for, which compute to pick (serverless, all-purpose, job, SQL warehouse), creating a small cluster with auto-termination, and a first notebook that reads and writes a table.
Use system.access.table_lineage and column_lineage to find every job, pipeline, dashboard and user that reads a table or column before you change it, and what lineage misses.
Turn a working notebook into a reliable scheduled job. Pick job compute, set a schedule and timezone, pass parameters, configure retries and timeouts, get alerts on failure, and run it as a service principal.
The notebook features you use every day. Language magics (%sql, %python, %sh, %pip), widgets for parameters, the most useful dbutils commands for files, secrets and notebooks, and how %run differs from dbutils.notebook.run.
Create or pick a SQL warehouse, write and save queries in the SQL editor, read query history and the query profile when something is slow, build a simple AI/BI dashboard and schedule it, with sizing tips that keep costs down.
Enable change data feed on Delta tables, read inserts, updates and deletes by version or as a stream, apply them downstream with MERGE, handle retention and schema changes, and know when CDF is the wrong tool.
Use expectations in Lakeflow Declarative Pipelines (formerly Delta Live Tables) to warn, drop or fail on bad data, route failing rows to a quarantine table instead of losing them, share rule sets across tables, and report quality trends from the event log.
Use shallow clones for fast, cheap test copies and deep clones for independent copies, refresh them incrementally, mask sensitive columns, keep them inside Unity Catalog permissions, and avoid the shallow-clone trap where VACUUM on the source breaks the clone.
A misconfigured job or notebook used the production catalog instead of dev, overwriting or appending test data to live tables. Stop writers, find every table it touched from audit and Delta history, restore versions or remove the bad rows, and add catalog isolation so dev compute cannot reach prod.
After an IP access list update, users and integrations get "Source IP address is blocked by Databricks IP ACL". Regain access from an allowed address or through the account console, fix or disable the list, and roll changes out with a tested allow list next time.
The workspace URL times out or shows a blank page for everyone, while jobs may still run. Check name resolution for the workspace and browser-authentication endpoints, fix private DNS zone links and records, and restore front-end connectivity without opening public access permanently.
Overnight jobs fail with INVALID_PARAMETER_VALUE or "validation failed for ... policy" errors after a platform team tightened a cluster policy. Find the failing rule, roll the policy back or relax it, repair runs, and introduce a policy change process that tests existing jobs first.
Queries on managed and external tables fail across all workspaces with 403 AuthorizationPermissionMismatch or "Failed to get credentials". Check the access connector, its managed identity and storage role assignments, restore them, and update the storage credential if the identity changed.
The Unity Catalog object model explained for newcomers. The three-level namespace, managed vs external tables, volumes for files, and the handful of GRANT statements you need to share data safely with a team.
After locking an ADLS Gen2 account to selected networks, Databricks gets 403 "This request is not authorized to perform this operation". Work out which compute path is blocked and allow each one correctly.
Find out whether the driver or the workers ran out of local disk, which files filled it (shuffle spill, temp files, disk cache, logs) and how to fix it without just buying bigger VMs.
Set up Databricks job notifications that page the right people on failure, slow runs and streaming lag, without alert noise from retries, skipped or canceled runs.
A Delta table written by Azure Databricks suddenly fails in Synapse serverless, Power BI, Trino or another Spark with protocol or unsupported feature errors. Find which feature changed the protocol, then downgrade it safely or give readers another path.
Big jobs retry stages over and over with FetchFailedException or ExecutorLostFailure. Work out whether executors died from memory, spot eviction or node loss, and fix partition sizing, memory, skew and spot settings.
Streaming upserts with foreachBatch and MERGE produce duplicates after failures, fail on duplicate keys within a batch, or slow down as the target grows. Make the batch idempotent, deduplicate correctly and give MERGE pruning to work with.
CDC pipelines built with AUTO CDC (formerly APPLY CHANGES INTO) keep stale values, create duplicate SCD2 rows or resurrect deleted keys. Fix sequence_by, delete handling, history tracking and full-refresh risks.
Creating an external table, volume or external location in Azure Databricks fails because the path overlaps another securable. Find what owns the path, understand the rules, and lay out ADLS containers so it can't happen again.
Enable workspace IP access lists in Azure Databricks with the CLI or REST API, allow office, VPN and CI/CD ranges, and fix "blocked by Databricks IP ACL" errors.
Someone left, their account was deactivated and scheduled jobs, pipelines and BI refreshes started failing. Find everything tied to that identity, move it to service principals and close the gap.
Several 2026 fixes in Databricks Runtime 18 change query results rather than throwing errors - NATURAL JOIN now matches columns case-insensitively, LEFT JOIN LATERAL with EXPLODE keeps rows it used to drop, and windowed min/max on BinaryType columns now returns per-frame values. Find the queries that could be affected and decide whether to accept the new results.
A maintenance update for Databricks Runtime 17.3 LTS, 18 LTS and 18.2 blocks some RESTORE and CLONE commands that change a table's partitioning when the source and target share files. Understand why, then use a full rewrite instead when you really need to change partitioning.
Since the 24 July 2026 maintenance update for Databricks Runtime 18 LTS and 18.2, legacy DStream jobs that put custom or third-party types in closures must list those classes in spark.streaming.checkpoint.allowedClasses before checkpoint recovery. Find the classes, set the config, and plan the move to Structured Streaming.
Copy-paste SQL against the Lakeflow Declarative Pipelines event log to find why an update failed, which expectations dropped rows, where backlog is building and what feeds what.
Databricks Runtime 18 LTS lists a known issue where the Avro fast reader slowly exhausts executor heap in long-running jobs, ending in OutOfMemoryError GC overhead limit exceeded. Confirm the pattern, then disable the fast reader on driver and executors and restart the cluster.
Since 29 June 2026, Databricks Runtime 17.3 LTS, 18 LTS and 18.2 reject metric views whose window measures reference other window measures. Rewrite the measure to reference base measures, and only use the opt-out flag as a temporary bridge.
Add row-level security and column masking to Unity Catalog tables with SQL UDFs and group membership, check which tables are protected, and avoid the common performance and compute traps.
A job times out connecting to SQL Server, an on-prem host or an external API. Test from the cluster, then work through DNS, NSG, UDR/firewall and the target's own firewall.
The Spark driver stops responding when it runs out of memory or spends its time in garbage collection, usually from collect(), toPandas(), large broadcasts or too many notebooks on one cluster. Find which one, then fix the code or resize the driver.
The Databricks CLI commands an on-call engineer actually uses on Azure - sign in, find and stop clusters, rerun failed jobs, move notebooks and files, and check Unity Catalog grants.
Compute (cluster) policies control what users can create - node types, worker counts, auto-termination, runtime and tags. Write a few policies per persona, assign them to groups, and remove unrestricted cluster creation.
The system.billing tables show every DBU your account used. Join usage with list prices to see spend by SKU, workspace, job, cluster, SQL warehouse, user and custom tag, and spot day-over-day spikes.
Delta rejects writes whose schema doesn't match the table. Decide whether the change is expected, then use mergeSchema for new columns, overwriteSchema for deliberate rewrites, or schema evolution in MERGE, and leave type changes to an explicit migration.
Unity Catalog needs privileges at every level - catalog, schema and object. Copy-ready SQL for granting read, write and create access to groups, checking effective grants, changing owners and fixing the most common permission errors.
A Databricks-specific review based on the lakehouse well-architected pillars, with evidence you can pull from system tables and settings rather than opinions. Use it yearly or before major changes, rate findings and keep a short owned action list.
Each Databricks compute type has a job it does best. Map workloads to serverless, classic job compute, all-purpose clusters, SQL warehouses or pipelines compute, decide when classic features still matter, and enforce the choice with policies so production doesn't end up on interactive clusters.
Writing row filters and column masks table by table doesn't scale. Define a classification scheme as governed tags, tag columns automatically and by review, attach attribute-based access control policies to tags at catalog or schema level, and monitor for untagged sensitive data.
Teams get into trouble when notebooks are edited in place in production or several people work on the same branch in one Git folder. Use personal Git folders on feature branches, keep code in files and wheels where sensible, choose a simple branching model, and deploy only through CI with Asset Bundles.
Databricks supports several ways to build transformations, and teams often mix them without a plan. Compare declarative pipelines, imperative jobs with notebooks or Python files, and dbt on SQL warehouses by skills, features and operations, then set a default per use case.
Unity Catalog permissions are only as clean as the identities behind them. Sync users and groups from Entra ID automatically, grant to account-level groups not individuals, give each pipeline its own service principal, keep admin roles few and audited, and plan joiner/mover/leaver handling end to end.
Both Databricks and Azure Machine Learning can train, register and serve models. Decide by where the data and feature pipelines live, who the users are, serving needs and existing MLOps tooling, and avoid splitting one model's lifecycle across both platforms without a clear handover.
Serverless compute runs in Databricks' account, not your VNet, so classic VNet injection rules don't apply. Use network connectivity configurations for stable egress identities and private endpoints to your resources, control outbound access with serverless network policies, and design firewall rules on storage and other services around them.
The Power BI storage mode decides dashboard speed, freshness and Databricks SQL cost. Choose import, DirectQuery or composite models per dataset, use SQL warehouses sized for the pattern, authenticate with SSO or service principals and private connectivity, and design gold tables for BI.
Lakehouse Federation lets you query external databases without copying data; ingestion brings data into Delta for performance and history. Decide per source by query volume, latency, source load and governance, use federation for exploration and low-volume access, and ingest for heavy, repeated or historical analytics.
When documents and tables already live in the lakehouse, the retrieval index can sit next to them or in Azure AI Search. Compare sync from Delta, hybrid search features, governance, application integration and operations, and choose by where the application runs and what search features it needs.
Natural-language analytics is only as good as the data and instructions behind it. Start with curated gold tables and clear metadata, scope each Genie space to one domain with example queries and instructions, rely on Unity Catalog permissions, test with real questions, and review usage and wrong answers regularly.
Databricks generates detailed audit events, but they only help if someone looks. Decide between system audit tables and diagnostic settings to Log Analytics/Sentinel, collect the event categories that matter, build detections for high-risk actions, and set retention that meets audit needs.
Teams often export files to share data, creating uncontrolled copies. Choose Databricks-to-Databricks or open Delta Sharing for live read access, Clean Rooms when neither side may see the other's raw data, Marketplace for wider distribution, and copying only when the recipient truly needs it, with governance and audit for each.
Unity Catalog allows one metastore per region per account, so multi-region platforms need a plan for where data lives, how it's shared and how governance stays consistent. Decide which regions host which data, share across metastores with Delta Sharing, manage permissions as code, and plan DR separately from day-to-day sharing.
Once usage is steady, commitments can lower the unit price of both DBUs and the Azure VMs under classic clusters. Measure the stable baseline, commit only to that, account for the move to serverless, and track utilisation monthly. No prices here; check current Azure pricing and your agreement.
Spot VMs cut the VM part of Databricks cost for workloads that can tolerate losing workers. Keep the driver on-demand, use spot with fallback to on-demand for workers, choose which jobs qualify, and watch eviction rates and run-time impact so savings don't turn into missed SLAs.
The VM type under a classic cluster decides how much memory per core, local disk for caching and shuffle, and price per hour you get. Match the family to the workload shape using Spark UI evidence, prefer fewer larger nodes for shuffle-heavy work, use local-SSD types for disk cache, and re-test after each change.
Always-on streams are often the largest steady cost in a workspace. Check whether each stream really needs to run continuously, use availableNow on a schedule where minutes of latency are fine, tune trigger intervals and batch sizes, use RocksDB for large state, and run several light streams on one cluster.
Instance pools keep ready VMs so clusters start in less time, but idle pool VMs are billed by Azure. Size min idle and idle timeouts to your schedule, preload runtimes, separate pools for drivers and workers, and decide whether serverless now does the job better.
Databricks has two very different caches. The disk cache keeps remote Parquet/Delta files on local SSDs automatically; Spark cache stores DataFrames in memory on request. Use the disk cache by choosing VM types with local SSD, use df.cache() sparingly, and remove cache calls that cause memory pressure or stale results.
Dashboards that aggregate large fact tables on every visit waste warehouse time. Find the repeated heavy queries, pre-compute them with materialized views refreshed on a schedule or incrementally, use streaming tables for continuously arriving data, and let result caching and Power BI design handle the rest.
MERGE is often the most expensive step of an incremental pipeline. Make it touch fewer files with pruning predicates and clustering on the join key, enable deletion vectors to avoid rewriting whole files, deduplicate the source first, and read the operation metrics to see what each run really rewrote.
Declarative pipelines (formerly Delta Live Tables) are easy to build and easy to overspend on. Run triggered instead of continuous where latency allows, choose the lowest product edition that has the features you use, prefer serverless or tuned classic compute, refresh only what changed, and keep development mode out of production.
Row-at-a-time Python UDFs move every value between the JVM and Python and block Photon. Rewrite them with built-in functions where possible, use vectorized pandas UDFs or Arrow-optimized UDFs where Python is needed, and keep driver-side pandas work and collect() out of production jobs.
Serverless compute removes cluster start time and tuning, but its cost depends on how it's used. Pick performance-optimized or standard mode per job, use budget policies to attribute spend, watch for long-running interactive notebooks, and compare cost per run with classic job compute before moving everything.
Besides DBUs and VMs, a Databricks platform drives storage transactions, private endpoint data processing, NAT gateway and cross-region charges. Find them in Azure Cost Management, cut storage operations with compaction and fewer listings, keep compute and storage in one region, and check what each network path costs.
Auto Loader works out of the box, but large landing zones make directory listing slow and costly. Switch to file notification or managed file events for big folders, cap batch sizes, avoid repeated schema inference, and keep landing paths well partitioned so discovery stays fast.
Autoscaling helps uneven workloads but can also waste money or slow jobs. Set minimum and maximum workers from observed usage, understand how scale-down works for jobs versus interactive clusters, use fixed-size clusters for steady batch jobs, and check node utilisation in system tables.
Photon charges more DBUs per hour but can finish SQL and DataFrame work much faster. Whether it saves money depends on the job. Check how much of a job Photon can run, compare cost per run with and without it, and keep it off for jobs dominated by Python UDFs, RDD code or tiny data.
Fix an Auto Loader ingestion that stopped picking up files or fails on every restart. Diagnose schema evolution stops, broken or moved checkpoints, Event Grid notification setup and backlog, and recover without double-loading files.
Fix clusters that terminate during start because they can't reach the Databricks control plane (secure cluster connectivity relay) or Azure dependencies after a firewall, UDR or DNS change. Check required egress, Private Link DNS and NSG rules.
Fix job and cluster launch failures in a VNet-injected Azure Databricks workspace when the host or container subnet has no free IPs. Confirm it from cluster events, free capacity now and resize or redesign the subnets properly.
Recover when job clusters can't get VMs from Azure: quota exceeded for a VM family, regional capacity stockouts, or spot evictions. Read the termination reason, switch node types or capacity type fast, and request quota before the next peak.
Two runs of the same job overlapped, causing duplicate rows or ConcurrentAppendException. Find where the extra run came from (ADF retry, manual run, schedule overlap), clean the data and enforce single-run behaviour with queueing and idempotent writes.
Restore ADF, CI/CD and external tools that call Azure Databricks after an Entra ID client secret, Databricks OAuth secret or PAT expired. Identify which credential, rotate it, update every consumer and move to managed identity or federation.
Fix job and cluster failures caused by library installation: egress to PyPI or Maven blocked, a new dependency version breaking the job, a yanked package or a private feed outage. Pin, mirror and package dependencies so a public repo can't take down production.
Respond to a Databricks PAT found in Git, a notebook output or a log. Revoke the token, find what it was used for in audit system tables, check data access and exfiltration paths, and move people and automation to OAuth.
Treat a sudden Azure Databricks cost jump as an incident. Use system.billing tables to find the workspace, SKU, cluster or job behind it, stop the waste safely and put budgets, policies and auto-termination in place.
Restore BI when a Databricks SQL warehouse fails to start or queues every query. Check warehouse state and events, quota and capacity for classic/pro, scaling limits, runaway queries and Power BI connection storms.
Catch up a Structured Streaming job reading from Azure Event Hubs when input rate exceeds processing rate. Read stream progress, fix partition parallelism, batch sizing, slow sinks and state growth, and catch up safely before retention deletes events.
Fix Delta tables that throw FileNotFoundException because data files referenced by the transaction log were removed by an aggressive VACUUM, a lifecycle rule or a manual delete. Repair the table, recover files if possible and set safe retention.
Fix clusters and jobs that terminate with an init script failure. Find the script's log, check common causes (moved location, blocked downloads, apt locks, bad line endings), hotfix the script and move scripts to Unity Catalog volumes or workspace files.
Regain administration of an Azure Databricks account and Unity Catalog metastore after the only admin left, an admin group was deleted in Entra ID or SCIM removed members. Use Entra Global Administrator bootstrap rights, restore groups and set up resilient admin ownership.
Recover a dropped managed table in Unity Catalog with UNDROP, re-register external tables, restore overwritten data with Delta time travel, and stop cleanup scripts from dropping production objects again.
Move transformation logic from ADF mapping data flows into Databricks, keeping ADF for orchestration if you like. Translate data flow transformations to Spark SQL or Lakeflow declarative pipelines, compare outputs and cost, and switch.
Plan the exit from HDInsight clusters. Map Spark, Hive/LLAP, Kafka, HBase and Oozie workloads to Databricks and Azure services, reuse data already in ADLS, migrate the Hive metastore to Unity Catalog and decommission clusters.
Replace a chain of notebooks that read, clean and write tables with a declarative pipeline. Define bronze/silver/gold tables, move checks into expectations, handle CDC with AUTO CDC / APPLY CHANGES, and run it incrementally.
Replace Entra ID credential passthrough and Hive table access control with Unity Catalog grants, external locations and row/column controls. Map ADLS ACLs to UC privileges, test with real users and switch clusters to UC access modes.
Harden an Azure Databricks deployment that started with the managed VNet and public access. Plan the new network, build a VNet-injected workspace with secure cluster connectivity and Private Link, migrate content and switch users.
Move jobs created in the UI and notebooks edited in production to version-controlled Databricks Asset Bundles. Generate bundles from existing jobs, add dev/test/prod targets, test in CI and deploy with a service principal.
Split an all-in-one workspace into separate dev, test and prod workspaces on one Unity Catalog metastore. Use workspace-catalog bindings, environment catalogs, service principals and bundles so code moves between environments and data access doesn't leak.
Stop running scheduled production jobs on shared interactive clusters. Move them to job clusters or serverless jobs, handle libraries and init scripts, set policies and timeouts, and compare cost and reliability.
Move users, CI/CD, ADF and external tools off long-lived PATs. Use OAuth U2M for people, service principals with OAuth M2M or Entra federation for pipelines, ADF managed identity, then restrict and expire PATs.
Serve reporting marts straight from the lakehouse instead of copying data into Synapse dedicated pools or Azure SQL. Convert T-SQL objects, size SQL warehouses, repoint Power BI and retire the copy pipelines.
Replace dbutils.fs.mount points and files stored in the DBFS root with governed Unity Catalog external locations and volumes. Inventory mounts, create the UC objects, rewrite paths in code and remove the mounts.
Fix jobs that run pandas on the driver and die with out-of-memory errors as data grows. Recognise driver-bound code, convert common pandas operations to PySpark, use pandas API on Spark or applyInPandas where it helps, and validate results.
Move Spark workloads from Azure Synapse Spark pools to Azure Databricks. Map pools to clusters, Synapse lake database tables to Unity Catalog, mssparkutils to dbutils, linked services to external locations, and pipelines to Workflows or ADF.
Move tables, permissions, jobs and code from the workspace Hive metastore to Unity Catalog using Databricks Labs UCX. Assessment, group migration, storage credentials and external locations, table upgrade, code changes and cutover.
Plan a fleet-wide move from an ageing Databricks Runtime LTS to the current one. Inventory runtime versions, read the breaking changes, test with shadow runs, handle Python and library versions, and roll out by wave.
Find which layer is blocking access, whether that's Unity Catalog grants, the access connector's Azure role, the storage firewall, serverless NCC or the compute access mode, then fix it with the right command.
Create Key Vault-backed and Databricks-backed secret scopes, grant READ/WRITE/MANAGE, read secrets from notebooks, SQL and cluster config, and fix the usual "secret does not exist" and 403 errors.
A reusable design for a Delta lakehouse on Azure Databricks. Covers how to lay out Unity Catalog catalogs and schemas, what belongs in bronze, silver and gold, standard loading patterns (append, MERGE, SCD2, snapshot), table settings, data quality, compute per workload, and how to publish gold tables as governed data products.
Who holds the encryption keys, which security controls to switch on, how to find the slow step in a query plan, and how to publish and apply changes with Change Data Feed and AUTO CDC.
Driver disk, DBFS, workspace files or Unity Catalog volumes? Which path works from Spark, Python, %sh and the CLI, how to move files between them, and how to retire DBFS mounts.
Build a production Auto Loader pipeline step by step: schema inference and evolution, rescued records, bronze to gold with availableNow triggers, file notification mode, monitoring and fixes for common failures.
When a Bloom filter index really speeds up equality lookups on high-cardinality columns, how to size fpp and numItems, how to prove it works, and why liquid clustering usually comes first.
Data Factory ingests and orchestrates, Databricks transforms bronze to gold. Covers managed-identity linking, parameters, idempotent MERGE, monitoring and CI/CD for both tools.
Control exactly who sees which data: workspace entitlements, object permissions, Unity Catalog grants, row filters, column masks and dynamic views, with test and audit queries.
Turn shared helper code into a versioned Python wheel. Build it with pyproject.toml, test it locally, and install it on Databricks from a Unity Catalog volume or a private package feed. Run it as a wheel job task, and automate build and deploy with Databricks Asset Bundles.
Pick the storage account, tier and redundancy for a Databricks lakehouse, connect it through access connectors and Unity Catalog with no keys, lock it down with private endpoints, and avoid settings that break Delta.
Where Delta data and metadata live, what DROP really deletes, how managed locations work, and how to convert Parquet and upgrade Hive metastore tables to Unity Catalog.
No posts in this category yet.