Transformation: replace cron on EC2 with EventBridge Scheduler, Lambda and Step Functions

#The problem

One EC2 instance runs dozens of cron jobs. It's a single point of failure, nobody knows all the jobs, failures are silent, and it holds long-lived credentials.

#1. Inventory

for u in $(cut -f1 -d: /etc/passwd); do sudo crontab -l -u $u 2>/dev/null | sed "s/^/$u: /"; done | grep -v '^.*: #'
ls /etc/cron.d /etc/cron.daily /etc/cron.hourly
systemctl list-timers --all

For each job record: schedule, command, runtime, what it reads/writes, owner, and whether it's still needed.

#2. Pick a target

#3. Create a schedule

aws scheduler create-schedule --name nightly-report \
  --schedule-expression "cron(0 2 * * ? *)" \
  --schedule-expression-timezone "Europe/London" \
  --flexible-time-window Mode=OFF \
  --target '{
    "Arn": "arn:aws:lambda:eu-west-1:111122223333:function:nightly-report",
    "RoleArn": "arn:aws:iam::111122223333:role/scheduler-invoke",
    "RetryPolicy": {"MaximumRetryAttempts": 3, "MaximumEventAgeInSeconds": 3600},
    "DeadLetterConfig": {"Arn": "arn:aws:sqs:eu-west-1:111122223333:scheduler-dlq"}
  }'

EventBridge cron has six fields (with year) and uses ? for day-of-month or day-of-week — it's not a copy-paste of Linux cron. Time zones are supported, so DST is handled for you.

Universal target example — stop dev instances every evening without any Lambda:

aws scheduler create-schedule --name stop-dev-ec2 \
  --schedule-expression "cron(0 19 ? * MON-FRI *)" --schedule-expression-timezone "Europe/London" \
  --flexible-time-window Mode=OFF \
  --target '{"Arn":"arn:aws:scheduler:::aws-sdk:ec2:stopInstances","RoleArn":"arn:aws:iam::111122223333:role/scheduler-ec2","Input":"{\"InstanceIds\":[\"i-0abc\",\"i-0def\"]}"}'

#4. Multi-step jobs with Step Functions

A nightly ETL that was extract.sh && transform.sh && load.sh becomes a state machine with per-step retries and a failure notification:

{
  "StartAt": "Extract",
  "States": {
    "Extract":   { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "extract"},
                   "Retry": [{"ErrorEquals": ["States.ALL"], "MaxAttempts": 2, "BackoffRate": 2}], "Next": "Transform",
                   "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Notify"}] },
    "Transform": { "Type": "Task", "Resource": "arn:aws:states:::ecs:runTask.sync", "Parameters": {"Cluster": "jobs", "TaskDefinition": "transform", "LaunchType": "FARGATE"}, "Next": "Load",
                   "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "Notify"}] },
    "Load":      { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Parameters": {"FunctionName": "load"}, "End": true },
    "Notify":    { "Type": "Task", "Resource": "arn:aws:states:::sns:publish", "Parameters": {"TopicArn": "arn:aws:sns:eu-west-1:111122223333:ops", "Message": "Nightly ETL failed"}, "Next": "Fail" },
    "Fail":      { "Type": "Fail" }
  }
}

(The ECS step also needs a NetworkConfiguration for Fargate.)

#5. Observability

• Alarms on Lambda Errors, Step Functions ExecutionsFailed, and messages in the scheduler DLQ.
• Logs in CloudWatch with retention set.

#6. Cut over

Run old and new in parallel for one cycle with the new job in dry-run or writing to a test location; compare outputs; then comment out the crontab line. Decommission the instance once the crontab is empty.

#Checklist

• Every cron job inventoried; dead jobs deleted
• Target chosen per job; IAM role per schedule
• Retries, DLQ and alarms configured
• Time zones set explicitly
• Cron server retired