Home / Data Platform
Databricks, Spark, lakehouse, Data Factory, Fabric, Power BI, warehouses, Kafka, Airflow and dbt. 161 posts.
Replace copy-pasted tasks and sensor chains with dynamic task mapping and dataset (asset) triggered DAGs. Control fan-out with max_active_tis_per_dag and pools, handle partial failures, and avoid the mapping patterns that overload the scheduler.
Write a small, correct Airflow DAG with the TaskFlow API, set a schedule without the classic start_date and catchup surprises, pass data between tasks, add retries, and test it with airflow dags test before deploying.
How Iceberg partition transforms work, why queries do not need partition columns, how to evolve a table from daily to hourly or add bucketing without rewriting history, and how to check pruning and compact old layouts when you want to.
What each common data file format is good and bad at, why Parquet is the default for analytics, where Avro and JSON still fit, and how to inspect and convert files with Python, Spark and DuckDB.
Set up a dbt Core project, declare sources, build staging and mart models with ref(), add built-in tests, choose materializations, and generate documentation, with the commands you will run every day.
A first hands-on tour of Delta Lake. Create a Delta table, append and upsert data with MERGE, read the transaction history, travel back to an older version and restore it, with SQL and PySpark side by side.
The core Kafka concepts in plain terms. Create a topic, produce and consume messages, see how partitions and keys affect ordering, and read consumer group lag, all with the standard Kafka command-line tools.
Move partitions between brokers without hurting producers and consumers. Generate rack-aware plans, throttle replication traffic, monitor progress, handle stuck reassignments, and rebalance leaders after adding or removing brokers.
Producers fail with RecordTooLargeException or MESSAGE_TOO_LARGE. Find which limit is hit, raise the right settings in the right order, or shrink the messages instead.
An UPDATE without the right WHERE clause or a DELETE wiped production data. Find the query ID, clone the table as it was just before that statement, compare, and restore only the damaged rows (or swap the whole table) before the Time Travel window closes.
After a producer release, consumers fail with SerializationException or schema-not-found errors and lag explodes. Identify the bad schema version, stop the producer, get consumers past the bad records, restore compatibility settings and prevent unchecked schema changes.
Someone ran kafka-topics --delete on a live topic. Producers fail or auto-create an empty topic, consumers reset, and data is gone from the brokers. Freeze auto-creation, recreate the topic with the right config, replay from mirrors, sinks or sources, and fix consumer offsets.
After activating or editing an account-level network policy, logins fail with "IP is not allowed to access Snowflake" for users, BI tools and pipelines. Find a path back in, unset or fix the policy, then roll network rules out safely with user-level testing.
A release changed shared code, a requirement or a provider version, and now many DAGs show import errors, vanish from the UI or never get scheduled. Find the broken import, roll back or hotfix, catch up missed runs safely, and add the CI checks that stop it recurring.
Find out why a Snowflake dynamic table is stale - auto-suspension, upstream failures, lost change tracking, skipped refreshes or a FULL refresh mode - and fix it safely.
Read the Snowflake Query Profile to find where time goes, measure partition pruning, decide when a clustering key is worth its cost, and use search optimization for selective lookups instead of over-sizing warehouses.
Prove a slow Spark stage is caused by skew, not too few resources. Read the task-duration distribution, find the hot keys, and choose between AQE skew join, broadcast joins, salting, splitting hot keys and two-phase aggregation.
Tasks using deferrable operators sit in the deferred state for hours. Check the triggerer is running and healthy, find triggers that block the async loop, fix provider packages and scale triggerer capacity.
Copy activities sit in Queued for minutes on a self-hosted integration runtime and nightly loads miss their window. Read the IR metrics, find whether CPU, memory, concurrency or network is the limit, and fix it with tuning and scale-out.
Data written to a Fabric Lakehouse appears in Spark but not in the SQL analytics endpoint, or appears late. Understand the metadata sync, find tables it skipped and trigger a refresh as part of the pipeline.
The Kafka Connect REST calls on-call engineers use most. Find failed tasks, restart only what failed, pause or stop connectors, change log levels and reset offsets safely.
Consumer groups keep rebalancing, lag grows and messages are processed twice. Find which timeout triggers it, then fix processing time, assignment strategy and pod restarts with static membership or KIP-848.
Exactly-once pipelines fail with fenced producers, or read_committed consumers stop making progress while the topic keeps growing. Diagnose with kafka-transactions.sh, abort hanging transactions safely and fix transactional.id and timeout design.
A Spark job pulling from SQL Server, Oracle, PostgreSQL or MySQL crawls in a single task. Fix it with partitioned reads, sensible bounds, explicit predicates and a larger fetch size.
A stateful streaming job gets slower every day until executors run out of memory. Read the state metrics in query progress, fix watermarks, deduplication and joins so old state is dropped, and move to RocksDB state storage.
Teams that upgraded the Airflow metadata database in place have hit different failures in staging and production. Clean the database, run the migration on a restored copy, fix DAGs with ruff's AIR rules, and only then schedule the real cutover with a tested rollback.
In Airflow 3, operators and tasks can't open database sessions to the Airflow metadata DB. Find the code that does, decide what it was for, and replace it with the Task SDK context or the official REST API client with a token, rather than a direct SQL hook.
Some teams upgrading from Airflow 3.3.0 to 3.3.1 or 3.3.2 on Kubernetes report every DAG file timing out in the DAG processor, DAGs being deactivated and runs stuck in queued. The root cause isn't confirmed yet. Here's how to tell whether you're hit, what to collect, and how to roll back cleanly.
Two patch releases in August and September 2026 changed behaviour you can trip over - DataFrame XComs under pandas 3, masked team-scoped secrets, backfill API responses and which credential wins when a request has both a cookie and a token. Check each one before rolling out.
The Airflow CLI commands, environment variable formats and secrets backend settings for managing connections and variables, plus how to move them between environments.
Kafka 4.0 removed ZooKeeper and old protocol versions, raised Java minimums and removed several configs. Before you upgrade, confirm KRaft mode on 3.3+ metadata, find clients older than 2.1, move brokers to Java 17, and clean configs that no longer exist.
Producers or consumers fail with Kafka authorization exceptions. Find the principal the broker really sees, read the authorizer log, and add the minimal ACLs with kafka-acls.sh.
The Schema Registry REST calls you need on call - list subjects, inspect versions and IDs, test compatibility before deploying, change modes and clean up safely after a 409 rejection.
A scheduled Snowflake task stopped loading data. Read SHOW TASKS and TASK_HISTORY, find the suspended, skipped or failing run, fix privileges or a stale stream, and restart the graph safely.
When a report shows stale or missing data, work backwards from the report through the semantic model, warehouse tables, pipelines and source. Find the first layer where the data stops, fix it, backfill and tell users.
Tasks that sit in scheduled or queued are waiting for something, usually a concurrency limit, a full pool, missing workers or a broken executor. Check each limit in order with the Airflow CLI and metadata.
Iceberg tables slow down and grow in storage without maintenance. Schedule rewrite_data_files for small files, expire_snapshots for old versions, rewrite_manifests for planning speed and remove_orphan_files for leftovers.
Mapping data flows run on managed Spark, so failures look like Spark failures. Find the error code in the activity output, then fix the cluster size, joins/broadcast, partitioning or sink settings.
dbt snapshots record how source rows change over time as SCD type 2 history. Pick the timestamp or check strategy, set a stable unique key, handle hard deletes, and avoid the mistakes that corrupt history.
The Kafka command-line tools on-call engineers use most, grouped by task. Check cluster health, manage topics and configs, inspect and reset consumer groups, and move partitions safely.
Files land in the bucket but never reach the table. Read SYSTEM$PIPE_STATUS and COPY_HISTORY, fix the pipe, notification or file error, then reload the gap without duplicates.
Real-time pipelines cost more to build and run, and many requests for "real time" don't need it. Turn freshness requests into explicit latency requirements, map them to batch, micro-batch or streaming, account for the operational cost of each, and use hybrid designs where only part of the data must be fast.
The orchestrator runs every pipeline, so its choice shapes operations for years. Compare code-first and visual tools, platform-native and cross-platform schedulers, decide by what you orchestrate and who builds pipelines, and avoid running several orchestrators that trigger each other in loops.
A neutral design guide for picking the core analytics platform. Start from workloads, skills and existing estate rather than feature lists, compare the platforms on the questions that decide operations and cost, run a proof of value with your own data, and plan for coexistence because most organisations end up with more than one.
The streaming backbone is hard to change once producers depend on it. Compare self-managed and managed Kafka with cloud-native services on protocol compatibility, operations effort, ecosystem and network fit, and decide by who will run it and how portable it must be.
Each source system has a best way to get data out. Classify sources by type, change rate and the load you may put on them, pick log-based CDC, incremental or full extracts, API pulls, file drops or event streams accordingly, and design for deletes, schema changes, replays and source protection.
Shared warehouses, clusters and capacities make it hard to see who drives cost. Choose allocation keys per platform, tag or separate compute by team, split shared costs fairly, start with showback before any chargeback, and use the numbers to drive optimisation rather than arguments.
A data contract is a written, versioned agreement on what a data product provides. Define the schema, semantics, quality and freshness guarantees, owners and change process, enforce what you can in CI and at runtime, and introduce contracts first where breakage hurts most.
Data quality work fails when it is a pile of tests nobody acts on. Place checks at the right layer, mix rule-based tests with anomaly monitoring, decide which failures block a pipeline and which only warn, route alerts to owners, and measure quality with a few clear indicators.
Open table formats give the lake ACID tables, time travel and schema evolution, but engines support them differently. Decide by which engines must read and write the data, the catalog you'll use and the operational features you need, and use interoperability options instead of copying data between formats.
Keeping everything forever raises cost and privacy risk; deleting too early breaks audits. Build a retention schedule per data class with legal and business input, map it to each layer, implement deletion that also covers table history and backups, support legal holds and erasure requests, and prove it with evidence.
Data platforms need separate environments like any software, but data makes it harder. Choose how environments are isolated, decide what data each one gets (synthetic, masked, sampled or read-only production), keep code promotion automated and identical, and stop sandboxes from becoming unmanaged copies of production.
A structured review across reliability, security, cost, performance and operations finds the risks that build up in a data platform over time. Scope it, gather evidence before the workshop, use a question set tailored to data workloads, rate risks, and turn findings into a short, owned improvement plan.
When "revenue" means three things in three dashboards, trust in data drops. Decide where metric definitions live (BI semantic models, a headless semantic layer or platform metric views), how many BI tools to support, and how certified datasets, ownership and change control keep numbers consistent.
Users don't care that a job succeeded; they care that the data is fresh and correct by the time they need it. Define data SLOs per critical dataset, collect pipeline, platform and data signals in one place, alert on SLO risk rather than every task failure, and review SLO performance with consumers.
A dbt project that grows without structure becomes slow to build and hard to change. Use clear staging, intermediate and mart layers with naming rules, decide between a monolith and several projects with cross-project references, set ownership and access per group, and keep CI and environments consistent.
Slow task scheduling, workers blocked by sensors and a busy metadata database are usually design and configuration problems. Make DAG files cheap to parse, set concurrency limits deliberately, use pools to protect shared systems, replace blocking sensors with deferrable operators and keep the metadata database lean.
ADF bills for orchestration, data movement, pipeline activity time and data flow compute separately, and each has its own waste pattern. Use the consumption view to find expensive pipelines, cut activity counts in loops, right-size DIUs, set data flow TTL and core counts sensibly, and review managed VNet and self-hosted IR costs.
Serverless SQL engines bill by data scanned, so file layout decides cost. Find the most expensive queries, convert to partitioned Parquet, make queries prune partitions and columns, materialise repeated results with CTAS/CETAS and put per-query and per-day limits in place.
Data lakes grow because every layer keeps everything forever - raw files, bronze copies, old table versions and test extracts. Measure size by zone and dataset, agree retention per zone, convert raw formats to compressed columnar files, tier cold data, vacuum table history and remove orphaned paths.
dbt projects get slower and more expensive as they grow because everything is rebuilt as tables on every run and CI rebuilds the world. Find the slowest models, make big ones incremental with the right strategy, pick materialisations deliberately, use slim CI with state comparison and tune threads and warehouse sizes.
Slow Power BI reports usually come from a model that's too big or badly shaped, not from capacity. Measure with Performance Analyzer and VertiPaq Analyzer, move to a star schema, cut columns and cardinality, fix relationships, rewrite the slowest DAX, and use aggregations or Direct Lake where they fit.
Long full refreshes strain sources, gateways and capacities. Make Power Query fold to the source, configure incremental refresh with the right partitions, tune refresh parallelism with the enhanced refresh API, size gateways, and spread schedules away from peaks.
Connectors running one task, tiny batches and verbose JSON converters move far less data than the cluster can handle. Measure per-connector throughput and lag, raise tasks to match partitions, tune connector and client batch settings, pick efficient converters, use DLQs instead of stopping on bad records, and size the worker cluster.
Most Kafka performance problems are client settings, not broker size. Measure where time goes, tune producer batching, linger and compression, keep idempotence and acks safe, tune consumer fetch and poll settings, scale consumers to partitions and check the broker side last.
Kafka clusters grow disks because every topic keeps data longer than anyone needs, with more partitions than consumers can use. Measure size per topic, set retention and compaction per use case, move long retention to tiered storage, compress, and right-size partition counts.
Fabric capacity sizing is about understanding capacity units (CU) over time, not peak minutes. Read the Capacity Metrics app correctly, find the items and operations that consume most, optimise them, separate workloads across capacities, set surge protection, pause non-production capacities and only then choose the SKU and commitment.
Event Hubs namespaces are often sized once for a launch and never revisited, or throttle because partitions were chosen badly. Measure ingress, egress and events per second, choose the tier, set throughput units with auto-inflate and a plan to scale down, pick partition counts for consumer parallelism, and review Capture and retention costs.
Snowflake storage includes much more than active data. Time Travel, Fail-safe, data retained for clones and churn from frequent rewrites can multiply the bill. Measure each component per table, set retention per data class, use transient tables for rebuildable data, review automatic clustering spend and clean up stages.
Warehouses are where most Snowflake credits go. Separate workloads into purpose-built warehouses, size them by query behaviour rather than habit, set auto-suspend tight, use multi-cluster for concurrency instead of bigger sizes, try query acceleration for outlier scans, and guard everything with resource monitors.
Dedicated SQL pools bill per DWU-hour whether busy or idle. Pause and scale on a schedule, isolate loads from reporting with workload groups and classifiers, enable result set caching and materialized views for repeated queries, and keep columnstore and statistics healthy so a smaller DWU does the job.
Recover an incremental dbt model that lost years of history because someone ran --full-refresh against a source that only keeps recent data. Restore from warehouse time travel, protect the model with full_refresh=false and fix the run process.
Restore an Airflow deployment where no tasks are scheduled because the scheduler is crashed, hung or blocked by its metadata database. Check heartbeats, database health, connections and locks, clean up old metadata safely and bring the backlog through.
Find out why scheduled ADF pipelines stopped running with no failures to alert on. Check trigger runtime state after CI/CD deployments, restart triggers, backfill missed tumbling windows and add a 'no runs' alert.
Get a failing production Spark job through tonight's run. Read the failure in the Spark UI, separate driver OOM, executor OOM, container kills and shuffle fetch failures, apply the quick fix and plan the real one.
When the morning data is late or missing, find which upstream step stalled, communicate, and backfill the missed intervals in Airflow in the right order without overloading sources or creating duplicates.
Recover a Debezium CDC connector that fails because the source purged the binlog or WAL it needed, or its offsets were lost. Choose between incremental snapshots, a filtered re-snapshot or a new connector, and keep consumers consistent.
Revenue doubled overnight because a failed load was re-run on top of a partial one. Find the duplicates, remove them safely in Delta Lake, Snowflake or SQL, and redesign the load with MERGE or partition overwrite so re-runs are always safe.
Fix Event Hubs consumers that keep disconnecting each other, stall on partitions or restart from the wrong position. Check consumer groups, partition ownership in the checkpoint store, client versions and retention.
Restore ADF and Synapse pipelines that fail together because a linked service secret, SAS token, service principal secret or SHIR credential expired. Find the shared credential, rotate it through Key Vault and move to managed identity.
Get a Kafka broker or KRaft controller quorum back after a crash. Read the startup errors, handle long log recovery, corrupted index or segment files, wrong cluster IDs and lost controller quorum without making data loss worse.
Recover when a consumer group suddenly starts from the beginning (duplicates) or from the end (missed data). Find why offsets were lost, stop consumers, reset to the right position by time or saved offsets, and protect committed offsets.
Restore a Kafka cluster where partitions are offline or under-replicated after one or more brokers failed. Find affected partitions, bring brokers back, decide on unclean leader election only as a last resort, and rebalance safely.
Recover Snowflake data after a DROP, TRUNCATE, bad UPDATE or a pipeline that overwrote a table. Use UNDROP, Time Travel AT/BEFORE queries and zero-copy clones, then swap safely and understand Fail-safe limits.
Clear a Synapse dedicated SQL pool where queries pile up in the queue. Find what holds concurrency slots, kill or isolate runaway loads, use workload groups and classifiers to protect BI, and scale if you must.
Recover when a source system renames, retypes or drops columns and pipelines fail or, worse, load nulls silently. Detect the change, quarantine bad batches, adapt the load safely and put schema checks and contracts in place.
Stop editing pipelines directly in the production factory. Connect the dev factory to Git, build ARM templates automatically with the npm package, deploy to test and prod with parameter overrides, and handle triggers and secrets correctly.
Plan the move from a Synapse dedicated SQL pool to a Fabric Warehouse. Check T-SQL surface differences, use the Fabric migration assistant, move data through OneLake, re-point pipelines and Power BI, and size Fabric capacity.
Plan the move from CDH/HDP/CDP clusters to Databricks on Azure or AWS. Map every Hadoop component to its lakehouse equivalent, migrate data to Delta, convert Hive/Impala SQL and Spark jobs, replace Oozie and Ranger, and run both in parallel before switching.
A practical, tool-level path from one overloaded central data team to domain teams that own data products. Define what a data product is, set up the catalog and permissions per domain, give domains a self-serve platform and keep shared governance.
Decide when streaming is worth it, then move a batch pipeline to Kafka and Structured Streaming into Delta tables. Covers exactly-once sinks, watermarks for late data, stateful aggregations, triggers and operating streams in production.
Replace cron jobs that poll SFTP folders once a night with ingestion that reacts to each file as it lands. Use managed SFTP into object storage, storage events, Auto Loader or Snowpipe, file validation and a quarantine path.
Replace database triggers and timestamp-based incremental extracts with log-based change data capture. Prepare MySQL, PostgreSQL or SQL Server, deploy Debezium on Kafka Connect, land changes in the lakehouse with MERGE and remove the old triggers.
Replace mapping-heavy ETL tools with SQL-first ELT in dbt. Inventory mappings, separate extraction from transformation, rebuild logic as layered dbt models with tests, run old and new in parallel and retire the ETL servers.
Move a production Kafka cluster off ZooKeeper before ZooKeeper support disappears. Pre-checks, deploying KRaft controllers in migration mode, moving brokers, finalising, and the rollback window you get before finalisation.
Upgrade folders of Parquet files registered as Hive tables to a transactional table format. Choose Delta or Iceberg (or both via UniForm), convert in place or rewrite, and change writers and readers without breaking consumers.
Move workspaces from Power BI Premium per capacity to Fabric F SKUs. Map P to F sizes, create the Fabric capacity in Azure, reassign workspaces with no content changes, check region and features, and manage cost with pause and reservations.
Move DAGs off a hand-built Airflow server onto managed Airflow. Clean up DAGs and dependencies, move connections and variables to a secrets backend, test on the target version, migrate history only if needed and switch DAG by DAG.
Move Kafka off self-run VMs onto a managed service. Compare MSK, Confluent Cloud and the Event Hubs Kafka endpoint, replicate topics and offsets with MirrorMaker 2 or Cluster Linking, migrate consumers before producers and cut over safely.
Move SSAS Tabular or Azure Analysis Services models to Power BI semantic models on Premium/Fabric capacity. Check compatibility, deploy through the XMLA endpoint, rebind data sources and security, redirect reports and retire the servers.
A practical migration plan from an appliance data warehouse to Snowflake. Inventory and usage analysis, schema and SQL conversion, historical data transfer, rebuilding loads, testing with reconciliation, cutover of BI tools and decommissioning.
The dbt selectors, graph operators and CLI flags you need on support duty to rebuild one model, its downstream, only what changed, or only what failed.
Split a Snowflake credit spike by service, warehouse, query and serverless feature with ACCOUNT_USAGE views, stop the burn, and add guardrails so it doesn't repeat.
Design a small set of cluster policies per persona that enforce auto-termination, approved node types and runtimes, tags, spot usage and Unity Catalog access modes. Includes policy JSON, rollout steps and how to measure compliance.
Use system.billing tables to see where DBUs go by workspace, SKU, job, warehouse and team, then apply the levers that usually cut 20–40% — job compute instead of all-purpose, auto-termination, right-sizing, Photon where it pays, serverless, spot, and tags with budgets.
A practical method to speed up slow Databricks jobs and lower their cost. Start from the Spark UI and query profile, then fix the usual causes — shuffle partitions, skew, bad joins, small files, Python UDFs, spill and over-caching — and decide when Photon pays off.
Build a complete inventory of a source Databricks workspace before a tenant or account migration. Covers jobs, clusters, notebooks, Unity Catalog and Hive tables, grants, mounts and hard-coded paths, owners, and active usage from system tables, with scripts you can run today.
Build day-2 monitoring for a Databricks platform from system tables. Covers job failures and SLA misses, long-running and stuck jobs, cluster start failures, warehouse queueing, security events from the audit log, data freshness, and how to route alerts.
The checks to pass before a new Databricks pipeline, job or data product goes into production. Covers code and deployment, compute, data quality, security and access, reliability, monitoring and alerting, cost, documentation and handover to operations.
A complete checklist for deploying Databricks the right way on Azure or AWS. Covers account and workspace design, networking and private connectivity, identity, Unity Catalog, storage, compute governance, security, CI/CD, monitoring, cost, and DR, with the decision for each item.
Tune Databricks SQL warehouses for BI and analytics. Covers serverless vs pro vs classic, sizing by query complexity, scaling by concurrency, auto-stop, query history analysis, result and disk caching, and fixing slow dashboards.
The cutover steps for one migration wave from a source Databricks workspace to the target tenant. Covers pre-checks, freezing the source, final data sync, enabling target jobs, switching consumers, validation, rollback and decommissioning the source.
Plan a move of Azure Databricks (or a Databricks account on AWS) from one tenant or account to another after a merger, divestiture or tenant consolidation. Covers why workspaces can't simply be moved, what has to be rebuilt versus copied, the target design, phases and the main risks.
Two jobs writing the same Delta table and one fails with ConcurrentAppendException or a sibling. Find the other writer, then fix it with predicates, row-level concurrency or scheduling.
Replace hand-tuned partitioning, Z-ORDER and scheduled OPTIMIZE/VACUUM jobs with liquid clustering and predictive optimization. Covers choosing clustering keys, converting existing tables, automatic clustering, enabling predictive optimization by catalog, and checking what it did.
Map and re-provision every identity when Databricks moves to another Entra tenant or Databricks account. Covers building the identity map, SCIM or automatic identity management, recreating service principals and secrets, fixing ownership and run-as identities, and handling guest accounts.
Move code and configuration to a new Databricks workspace in another tenant or account. Covers Git folders, export/import of notebooks, the Terraform exporter, generating Asset Bundles from existing jobs, cluster policies and warehouses, SQL queries and dashboards, and MLflow models and experiments.
Rebuild Unity Catalog in the target tenant or account with the same (or improved) structure. Covers metastore creation, access connectors and storage credentials, external locations, catalogs and schemas as code, translating grants to new groups, volumes, and Hive metastore leftovers.
Choose and run the right method to copy lakehouse data into the target tenant. Covers re-pointing to existing storage, copying Delta folders with azcopy or S3 replication and re-registering tables, Databricks-to-Databricks Delta Sharing with CTAS, incremental sync, and validation.
A stream won't restart after a code change, or you need a new checkpoint. Decide whether to keep, repair or replace it, and restart from the right offset without gaps or duplicates.
Go from DAG import errors and schedule logic to scheduler health, pools, concurrency limits and executor workers when Airflow tasks don't start.
Run Airflow without single points of failure using multiple active schedulers, a highly available PostgreSQL metadata database, redundant webservers and workers on Kubernetes, and remote logs.
Read Airflow task logs and worker events to tell code failures from killed workers, zombie tasks and timeouts, and set retries, timeouts and resources so jobs recover on their own.
Use the copy activity's duration breakdown to find the bottleneck (queue, source read, sink write or the self-hosted IR), then apply the matching tuning setting.
Prepare a secondary Data Factory in another region from Git and CI/CD, cover linked services, secrets, integration runtimes and triggers, and run failover without duplicate pipeline runs.
Install and harden a self-hosted integration runtime for on-premises and private sources, configure networking, credentials, monitoring and updates, and understand its single-node limits.
Diagnose ServerBusy and quota-exceeded errors in Event Hubs, size throughput or processing units, fix partition hot spots and consumer lag.
Separate dev, test and prod, put pipelines, notebooks, SQL models and infrastructure in Git, run automated tests on pull requests and promote releases with parameterised configs.
The minimum governance setup that makes a data platform supportable, covering scanning and cataloguing, ownership, sensitive data classification, lineage, access request workflow and audit.
A practical set of freshness, volume, schema, uniqueness and business-rule checks for pipelines, with SQL and dbt examples and guidance on blocking vs warning.
Map Databricks termination reason codes to cloud quota, capacity, networking and init-script problems, and fix the cluster so scheduled jobs run again.
Design Databricks DR as two workspaces in two regions. Covers what has to be replicated (code, jobs, data, catalog metadata, secrets), how to sync each piece, and a failover runbook with realistic RTO and RPO.
Set up one metastore per region, workspaces per environment, catalogs bound to workspaces, storage credentials and external locations, groups from your IdP and cluster policies.
Separate compilation, connection, database and test failures in dbt, use the compiled SQL and logs to debug, and rerun only what failed.
Check connector status, source-side replication slots and binlogs, and the sink, to find where change-data-capture lag builds up and recover without losing changes.
Recover from bad writes, accidental deletes and storage loss in a Delta lakehouse. Covers time travel and RESTORE, what VACUUM does to your recovery window, deep clone backups, and storage soft delete and versioning.
Keep Delta tables fast and storage under control by compacting small files, choosing between Z-ORDER and liquid clustering, and running VACUUM safely.
Add nodes to a self-hosted integration runtime for HA and scale-out, configure node-to-node communication, plan patching and updates, and monitor node health.
Build a gateway cluster for Power BI and Fabric so refreshes and DirectQuery survive a gateway machine failure, with load balancing, consistent drivers, recovery keys and a safe update routine.
Find which topics fill the broker disks, free space safely by lowering retention, recover an offline broker, and prevent it with quotas, alerts and tiered storage.
Measure lag per partition, tell slow processing from rebalance storms and hot partitions, and reset offsets safely when you need to.
Replicate Kafka topics and consumer group offsets to a DR cluster with MirrorMaker 2. Covers naming policies, offset translation, monitoring lag, and a consumer failover runbook.
Configure Kafka so that a broker or availability zone can fail without data loss or downtime, with the right replication settings, rack awareness, controller quorum, producer settings and monitoring.
Define bronze, silver and gold layers, the storage account/bucket layout per environment, table naming, file formats and ownership rules before the first pipeline lands data.
Understand Fabric's smoothing, overage and throttling stages, find the items burning capacity units in the Capacity Metrics app, and fix it by tuning, rescheduling or scaling.
Plan Fabric capacities per environment or workload, structure workspaces and domains, lock down tenant settings, and use Git integration with deployment pipelines for dev, test and prod.
Read the refresh history error, then fix gateway connectivity, expired credentials, mismatched data source definitions, timeouts and capacity limits.
Where to look in the Spark UI (jobs, stages, task metrics, SQL plans) to find skew, spills, small files and bad joins, with the fix for each pattern.
Make Databricks jobs survive VM loss, spot evictions, cloud capacity issues and partial failures, with job retries, availability settings, pools, idempotent writes and streaming checkpoints.
When a single-node cluster is the right choice, how to configure it, how it differs from multi-node job clusters and serverless compute, and policies that stop small workloads from burning large budgets.
Run a single combined broker/controller Kafka node with KRaft for development and test, set replication factors that work on one node, and list the settings you must change before production.
Work through the active role, the USAGE chain on database, schema and warehouse, future grants and ownership when Snowflake users can't see objects.
The first-week setup for a production Snowflake account, covering role hierarchy, warehouse sizing and timeouts, SSO and MFA, network policies, resource monitors, Time Travel and audit.
Replicate Snowflake databases and account objects to another region or cloud with failover groups, refresh them on a schedule, redirect clients with connection URLs, and run a failover drill.
Use query history and the query profile to tell warehouse queuing from spilling and poor partition pruning, then fix it with multi-cluster scaling, sizing or clustering.
Run Airflow on a single VM for a small team with LocalExecutor, an external PostgreSQL metadata database, persistent logs, secrets, backups and monitoring, and know when to move to a managed or HA setup.
Find skewed tables and expensive shuffle and broadcast steps in a Synapse dedicated SQL pool, and fix them with the right distribution, statistics and columnstore health.
Find tables made of thousands of tiny Parquet files, compact them with OPTIMIZE or rewrite procedures, and stop streaming and partitioning choices from creating them again.
Fix Unity Catalog access errors by understanding the USE CATALOG / USE SCHEMA / SELECT chain, external locations, ownership and compute access modes.
Use the run page, driver logs and Spark UI to tell driver OOM from executor OOM, find skewed partitions and repair only the tasks that failed.
Find failed runs across factories with KQL, read activity error codes, fix self-hosted integration runtime problems and rerun from the point of failure.
No posts in this category yet.