Home / Data Platform / Databricks & Spark
Databricks and Apache Spark: jobs, memory, shuffle, streaming and performance. 30 posts.
Prove a slow Spark stage is caused by skew, not too few resources. Read the task-duration distribution, find the hot keys, and choose between AQE skew join, broadcast joins, salting, splitting hot keys and two-phase aggregation.
A Spark job pulling from SQL Server, Oracle, PostgreSQL or MySQL crawls in a single task. Fix it with partitioned reads, sensible bounds, explicit predicates and a larger fetch size.
A stateful streaming job gets slower every day until executors run out of memory. Read the state metrics in query progress, fix watermarks, deduplication and joins so old state is dropped, and move to RocksDB state storage.
Get a failing production Spark job through tonight's run. Read the failure in the Spark UI, separate driver OOM, executor OOM, container kills and shuffle fetch failures, apply the quick fix and plan the real one.
Plan the move from CDH/HDP/CDP clusters to Databricks on Azure or AWS. Map every Hadoop component to its lakehouse equivalent, migrate data to Delta, convert Hive/Impala SQL and Spark jobs, replace Oozie and Ranger, and run both in parallel before switching.
Design a small set of cluster policies per persona that enforce auto-termination, approved node types and runtimes, tags, spot usage and Unity Catalog access modes. Includes policy JSON, rollout steps and how to measure compliance.
Use system.billing tables to see where DBUs go by workspace, SKU, job, warehouse and team, then apply the levers that usually cut 20–40% — job compute instead of all-purpose, auto-termination, right-sizing, Photon where it pays, serverless, spot, and tags with budgets.
A practical method to speed up slow Databricks jobs and lower their cost. Start from the Spark UI and query profile, then fix the usual causes — shuffle partitions, skew, bad joins, small files, Python UDFs, spill and over-caching — and decide when Photon pays off.
Build a complete inventory of a source Databricks workspace before a tenant or account migration. Covers jobs, clusters, notebooks, Unity Catalog and Hive tables, grants, mounts and hard-coded paths, owners, and active usage from system tables, with scripts you can run today.
Build day-2 monitoring for a Databricks platform from system tables. Covers job failures and SLA misses, long-running and stuck jobs, cluster start failures, warehouse queueing, security events from the audit log, data freshness, and how to route alerts.
The checks to pass before a new Databricks pipeline, job or data product goes into production. Covers code and deployment, compute, data quality, security and access, reliability, monitoring and alerting, cost, documentation and handover to operations.
A complete checklist for deploying Databricks the right way on Azure or AWS. Covers account and workspace design, networking and private connectivity, identity, Unity Catalog, storage, compute governance, security, CI/CD, monitoring, cost, and DR, with the decision for each item.
Tune Databricks SQL warehouses for BI and analytics. Covers serverless vs pro vs classic, sizing by query complexity, scaling by concurrency, auto-stop, query history analysis, result and disk caching, and fixing slow dashboards.
The cutover steps for one migration wave from a source Databricks workspace to the target tenant. Covers pre-checks, freezing the source, final data sync, enabling target jobs, switching consumers, validation, rollback and decommissioning the source.
Plan a move of Azure Databricks (or a Databricks account on AWS) from one tenant or account to another after a merger, divestiture or tenant consolidation. Covers why workspaces can't simply be moved, what has to be rebuilt versus copied, the target design, phases and the main risks.
Replace hand-tuned partitioning, Z-ORDER and scheduled OPTIMIZE/VACUUM jobs with liquid clustering and predictive optimization. Covers choosing clustering keys, converting existing tables, automatic clustering, enabling predictive optimization by catalog, and checking what it did.
Map and re-provision every identity when Databricks moves to another Entra tenant or Databricks account. Covers building the identity map, SCIM or automatic identity management, recreating service principals and secrets, fixing ownership and run-as identities, and handling guest accounts.
Move code and configuration to a new Databricks workspace in another tenant or account. Covers Git folders, export/import of notebooks, the Terraform exporter, generating Asset Bundles from existing jobs, cluster policies and warehouses, SQL queries and dashboards, and MLflow models and experiments.
Rebuild Unity Catalog in the target tenant or account with the same (or improved) structure. Covers metastore creation, access connectors and storage credentials, external locations, catalogs and schemas as code, translating grants to new groups, volumes, and Hive metastore leftovers.
Choose and run the right method to copy lakehouse data into the target tenant. Covers re-pointing to existing storage, copying Delta folders with azcopy or S3 replication and re-registering tables, Databricks-to-Databricks Delta Sharing with CTAS, incremental sync, and validation.
A stream won't restart after a code change, or you need a new checkpoint. Decide whether to keep, repair or replace it, and restart from the right offset without gaps or duplicates.
Map Databricks termination reason codes to cloud quota, capacity, networking and init-script problems, and fix the cluster so scheduled jobs run again.
Design Databricks DR as two workspaces in two regions. Covers what has to be replicated (code, jobs, data, catalog metadata, secrets), how to sync each piece, and a failover runbook with realistic RTO and RPO.
Set up one metastore per region, workspaces per environment, catalogs bound to workspaces, storage credentials and external locations, groups from your IdP and cluster policies.
Where to look in the Spark UI (jobs, stages, task metrics, SQL plans) to find skew, spills, small files and bad joins, with the fix for each pattern.
Make Databricks jobs survive VM loss, spot evictions, cloud capacity issues and partial failures, with job retries, availability settings, pools, idempotent writes and streaming checkpoints.
When a single-node cluster is the right choice, how to configure it, how it differs from multi-node job clusters and serverless compute, and policies that stop small workloads from burning large budgets.
Find tables made of thousands of tiny Parquet files, compact them with OPTIMIZE or rewrite procedures, and stop streaming and partitioning choices from creating them again.
Fix Unity Catalog access errors by understanding the USE CATALOG / USE SCHEMA / SELECT chain, external locations, ownership and compute access modes.
Use the run page, driver logs and Spark UI to tell driver OOM from executor OOM, find skewed partitions and repair only the tasks that failed.
No posts in this category yet.