Home / Databases / PostgreSQL

DatabasesRunbookAdvanced

Sev1: PostgreSQL won't start after a crash or reboot — reading the log and fixing the common causes

After a host crash, failover or patch reboot, PostgreSQL refuses to start. Read the startup log, fix permissions, stale PID files, config and memory errors, and know when to promote a standby instead.

Deepak Bhardwaj3 min read

Applies toPostgreSQL 12-17 on Linux (systemd, Debian/Ubuntu and RHEL packaging)

Symptoms

  • systemctl start postgresql fails; applications get connection refused.
  • No standby has been promoted yet.

1. Read the real error

sudo systemctl status postgresql@16-main --no-pager
sudo journalctl -u postgresql@16-main -n 100 --no-pager
sudo tail -100 /var/log/postgresql/postgresql-16-main.log   # Debian/Ubuntu
sudo tail -100 /var/lib/pgsql/16/data/log/postgresql-*.log  # RHEL family
Message Cause Go to
lock file "postmaster.pid" already exists Stale PID file or a second instance Step 2
data directory ... has wrong ownership / invalid permissions Restore or copy changed owner/mode Step 3
could not create shared memory segment / could not map anonymous shared memory Memory settings vs host RAM Step 4
configuration file ... contains errors Bad edit in postgresql.conf or pg_hba.conf Step 5
No space left on device Disk full Disk-full runbook
could not locate a valid checkpoint record / invalid primary checkpoint record WAL damaged or missing Step 6

2. Stale postmaster.pid

Make sure no postgres process is running first:

ps -ef | grep [p]ostgres

Only if nothing is running, move the stale file aside:

sudo -u postgres mv /var/lib/postgresql/16/main/postmaster.pid /tmp/postmaster.pid.old

3. Ownership and permissions

sudo chown -R postgres:postgres /var/lib/postgresql/16/main
sudo chmod 700 /var/lib/postgresql/16/main

On RHEL with SELinux, a restored directory may have the wrong context:

sudo restorecon -Rv /var/lib/pgsql

4. Shared memory

After moving to a smaller VM or enabling huge pages, shared_buffers may no longer fit:

grep -E "shared_buffers|huge_pages" /etc/postgresql/16/main/postgresql.conf
grep -i huge /proc/meminfo

Set huge_pages = try (not on) or reduce shared_buffers to about 25% of RAM, then start.

5. Configuration errors

sudo -u postgres /usr/lib/postgresql/16/bin/postgres -D /var/lib/postgresql/16/main \
  -c config_file=/etc/postgresql/16/main/postgresql.conf -C shared_buffers

The -C check parses the config and prints the value or the parse error. Revert the last change (check postgresql.auto.conf too, written by ALTER SYSTEM).

6. Damaged or missing WAL

This is the dangerous case. Options in order of preference:

  1. Promote a standby if you have one that is current:
sudo -u postgres psql -h standby1 -c "SELECT pg_promote();"

  1. Restore from backup with point-in-time recovery (pgBackRest, Barman, cloud snapshots plus WAL archive).

  2. Last resort: pg_resetwal. It lets the server start but can leave the database inconsistent (lost or half-applied transactions, broken indexes).

Caution

Before pg_resetwal, copy the whole data directory (cp -a or a volume snapshot). Afterwards, immediately pg_dumpall the data, restore it into a fresh cluster, and check application data. Never keep running production on a reset cluster.

sudo systemctl stop postgresql@16-main
sudo -u postgres cp -a /var/lib/postgresql/16/main /backup/main-before-reset
sudo -u postgres /usr/lib/postgresql/16/bin/pg_resetwal -n /var/lib/postgresql/16/main   # dry run

Get a second senior DBA to agree before running it without -n.

7. After it starts

  • Check SELECT pg_is_in_recovery(); returns what you expect.
  • Run amcheck on critical indexes if the crash was hardware-related.
  • Confirm replication to standbys resumes.

Prevent it next time

  • Keep at least one streaming standby and a tested PITR backup.
  • Use data_checksums on new clusters to detect corruption early.
  • Validate config changes with -C or pg_ctl reload before restarting.
  • Put the data directory on storage with power-loss protection; never disable fsync.

Related fixes