Azure Data Factory first pipeline: linked service, datasets, copy activity and how to read a failure

#The goal

You need to copy a CSV file from one storage folder to another as Parquet, and run it on a schedule. It's a small job that shows you every part of Azure Data Factory (ADF).

#Step 1: create the factory and give it access

az extension add --name datafactory
az datafactory create --resource-group rg-data --factory-name adf-ops-demo --location centralindia

The factory has a system-assigned managed identity, so no storage key is needed. Give it a data role:

ADF_PRINCIPAL=$(az datafactory show -g rg-data --factory-name adf-ops-demo --query identity.principalId -o tsv)
ST_ID=$(az storage account show -g rg-data -n stdataops01 --query id -o tsv)

az role assignment create \
  --assignee-object-id "$ADF_PRINCIPAL" \
  --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope "$ST_ID"

If you don't see an identity, turn on the system-assigned identity in the factory's settings.

#Step 2: create the linked service

Save as ls_adls.json:

{
  "type": "AzureBlobFS",
  "typeProperties": {
    "url": "https://stdataops01.dfs.core.windows.net"
  }
}

az datafactory linked-service create -g rg-data --factory-name adf-ops-demo \
  --linked-service-name ls_adls --properties @ls_adls.json

With only a URL, it signs in with the factory's managed identity.

#Step 3: create the two datasets

ds_orders_csv.json:

{
  "type": "DelimitedText",
  "linkedServiceName": { "referenceName": "ls_adls", "type": "LinkedServiceReference" },
  "typeProperties": {
    "location": { "type": "AzureBlobFSLocation", "fileSystem": "landing", "folderPath": "orders" },
    "columnDelimiter": ",",
    "firstRowAsHeader": true
  }
}

ds_orders_parquet.json:

{
  "type": "Parquet",
  "linkedServiceName": { "referenceName": "ls_adls", "type": "LinkedServiceReference" },
  "typeProperties": {
    "location": { "type": "AzureBlobFSLocation", "fileSystem": "bronze", "folderPath": "orders" }
  }
}

az datafactory dataset create -g rg-data --factory-name adf-ops-demo \
  --dataset-name ds_orders_csv --properties @ds_orders_csv.json
az datafactory dataset create -g rg-data --factory-name adf-ops-demo \
  --dataset-name ds_orders_parquet --properties @ds_orders_parquet.json

#Step 4: create the pipeline

pipeline.json:

{
  "activities": [
    {
      "name": "CopyOrders",
      "type": "Copy",
      "policy": { "timeout": "0.01:00:00", "retry": 2, "retryIntervalInSeconds": 60 },
      "inputs": [{ "referenceName": "ds_orders_csv", "type": "DatasetReference" }],
      "outputs": [{ "referenceName": "ds_orders_parquet", "type": "DatasetReference" }],
      "typeProperties": {
        "source": {
          "type": "DelimitedTextSource",
          "storeSettings": { "type": "AzureBlobFSReadSettings", "recursive": true, "wildcardFileName": "*.csv" }
        },
        "sink": {
          "type": "ParquetSink",
          "storeSettings": { "type": "AzureBlobFSWriteSettings" }
        }
      }
    }
  ]
}

az datafactory pipeline create -g rg-data --factory-name adf-ops-demo \
  --name pl_copy_orders --pipeline @pipeline.json

Tip

The default activity timeout is long. Set your own, like the one hour above, so a stuck copy fails and alerts you rather than running for days.

The @file syntax can behave differently in PowerShell, where @ has a special meaning. Quote it as "@pipeline.json", or run these commands in Bash or Cloud Shell.

#Step 5: run it and check

RUN_ID=$(az datafactory pipeline create-run -g rg-data --factory-name adf-ops-demo \
  --name pl_copy_orders --query runId -o tsv)

az datafactory pipeline-run show -g rg-data --factory-name adf-ops-demo \
  --run-id "$RUN_ID" --query "{status:status,ms:durationInMs,msg:message}"

Look at the activity's details, including rows and data copied:

az datafactory activity-run query-by-pipeline-run -g rg-data --factory-name adf-ops-demo \
  --run-id "$RUN_ID" \
  --last-updated-after 2026-11-06T00:00:00Z \
  --last-updated-before 2026-11-07T00:00:00Z \
  --query "value[].{name:activityName,status:status,read:output.rowsRead,written:output.rowsCopied,error:error.message}"

Check the output files:

az storage fs file list --account-name stdataops01 --file-system bronze --path orders --auth-mode login -o table

#Step 6: read a failure

For bigger problems, the pipeline failure runbook has the next steps.

#Step 7: add a daily schedule

trigger.json:

{
  "type": "ScheduleTrigger",
  "pipelines": [
    { "pipelineReference": { "referenceName": "pl_copy_orders", "type": "PipelineReference" } }
  ],
  "typeProperties": {
    "recurrence": {
      "frequency": "Day",
      "interval": 1,
      "startTime": "2026-11-07T01:00:00Z",
      "timeZone": "UTC"
    }
  }
}

az datafactory trigger create -g rg-data --factory-name adf-ops-demo \
  --name tr_daily --properties @trigger.json
az datafactory trigger start -g rg-data --factory-name adf-ops-demo --name tr_daily

A trigger does nothing until it's started. This is the usual reason a new schedule doesn't run.

Warning

Rerunning this pipeline copies all the source files again. Before you run it on production data, plan how to handle reruns, for example by moving processed files or writing to a dated folder.

#Prevent it next time

• Connect the factory to Git, instead of publishing by hand.
• Add an alert for failed pipeline runs in Azure Monitor.
• Use parameters for folder names, so one pipeline can serve many tables.
• Cost and speed come next: see copy activity tuning.