Symptoms
- Cluster events show termination with reason
INIT_SCRIPT_FAILURE. - All jobs using a shared cluster policy fail at start; interactive clusters won't come up.
- Started after a change to the script, to network egress rules, or to where the script is stored.
1. Get the script's output
Init script logs go to the cluster log delivery location if configured, under <log-path>/<cluster-id>/init_scripts/. Configure log delivery on a test cluster if it's missing:
"cluster_log_conf": { "volumes": { "destination": "/Volumes/ops/logs/cluster-logs" } }
Each script run writes .stdout.log and .stderr.log. The last lines of stderr usually tell you exactly what broke.
2. Common causes
| Error in stderr | Cause | Fix |
|---|---|---|
No such file or directory for the script itself | Script location changed or DBFS-based init scripts no longer supported | Store in a UC volume or workspace files; update the cluster/policy reference |
Could not resolve host, timeouts on pip/apt/curl | Egress blocked (firewall, UDR, NSG change) | Allow the repos, or use an internal mirror |
Could not get lock /var/lib/dpkg/lock | apt running in parallel with system processes | Retry loop or wait for the lock |
$'\r': command not found | Windows line endings | Convert to LF |
| Permission denied reading the script | Cluster identity can't read the volume/workspace file | Grant READ VOLUME / workspace file permissions |
| Package version conflict | Unpinned install pulled a new version | Pin versions |
3. Hotfix
Make the script defensive and fast:
#!/bin/bash
set -euo pipefail
exec > >(tee -a /tmp/init.log) 2>&1
for i in {1..10}; do
if sudo apt-get update -y && sudo apt-get install -y --no-install-recommends libpq-dev=14.*; then break; fi
echo "apt busy or failed, retry $i"; sleep 15
done
/databricks/python/bin/pip install --no-cache-dir "psycopg2==2.9.9" "pyodbc==5.1.0"
Convert line endings if needed: sed -i 's/\r$//' init.sh.
If the business needs jobs running now, temporarily remove the init script from the job cluster spec (or the policy) for jobs that don't depend on it, and run dependent ones after the fix.
4. Move scripts to a supported location
databricks fs cp ./init.sh dbfs:/Volumes/ops/scripts/init/init.sh --overwrite
Cluster spec:
"init_scripts": [ { "volumes": { "destination": "/Volumes/ops/scripts/init/init.sh" } } ]
For shared access mode clusters, the script must be on the metastore's allowlist of init scripts (Catalog Explorer → metastore → Allowed JARs/Init Scripts).
Tip
Most init scripts only install Python packages. Use cluster libraries, job environment dependencies or a requirements file in a volume instead — they're cached and don't fail the whole cluster on a transient error.
5. Test before rollout
Keep a small test cluster using the same policy. Any change to an init script goes there first, with logs checked, before production policies pick it up.
Prevent it next time
- Scripts in Git, deployed to volumes by CI, with versioned paths (
init-v12.sh). - Pin package versions; mirror external repos internally.
- Cluster log delivery enabled on all production clusters.