Optimization: Lakeflow Declarative Pipelines cost - triggered vs continuous, serverless, product edition and refresh scope

#Biggest levers

#Triggered on a schedule

# Asset Bundle excerpt
resources:
  pipelines:
    sales_pipeline:
      name: sales_pipeline
      continuous: false
      serverless: true
      catalog: prod
      schema: sales
  jobs:
    sales_pipeline_schedule:
      name: sales_pipeline_schedule
      schedule:
        quartz_cron_expression: "0 0/30 * * * ?"
        timezone_id: UTC
      tasks:
        - task_key: run
          pipeline_task:
            pipeline_id: ${resources.pipelines.sales_pipeline.id}

Use continuous only when consumers need data within a minute or two.

#Edition (classic compute)

If no pipeline uses expectations, Advanced is wasted. Check current edition definitions in the docs; serverless pipelines are priced differently.

#Refresh only what changed

• Avoid "full refresh all" as a routine fix; use selective refresh of the affected tables.
• Materialized views refresh incrementally when sources support it; check the pipeline event log for whether an update was incremental or full.

-- Pipeline event log: planning info for materialized views
SELECT timestamp, details:planning_information
FROM event_log(TABLE(prod.sales.daily_sales_mv))
WHERE event_type = 'planning_information'
ORDER BY timestamp DESC LIMIT 10;

#Classic compute tuning

• Enhanced autoscaling with sensible min/max.
• Right VM family (see the VM family post).
• Separate maintenance cluster settings are usually fine as defaults.

#Measure cost per pipeline

SELECT usage_metadata.dlt_pipeline_id, sku_name, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_metadata.dlt_pipeline_id IS NOT NULL
  AND usage_date >= current_date() - 30
GROUP BY ALL ORDER BY dbus DESC;

Warning

Pipelines left in development mode in production keep their clusters alive after updates. Deploy through bundles with production mode for prod targets.

#Checklist

• Continuous only where latency requires it
• Edition matches features used
• Serverless vs classic compared per pipeline
• Full refreshes avoided; incremental refresh verified
• Production mode in prod
• Cost per pipeline reported