Rethinking Nightly Batch Failure Alerts as an Operations Entry Point

When a nightly batch fails, posting a red alert to Slack does not make operations any easier.

The notification itself is necessary; the real trouble starts afterward. Who will look at it? Can I retry? Should I fix the input data? Do I need to notify the owner? If this decision is made ad hoc every time, more alerts does not mean lighter operations.

When I organized nightly batch failure alerts, I broke them down into four stages.

detection
  -> triage the cause
  -> decide whether to retry
  -> hand off to a human

This breakdown changes what the notification needs to contain. An exit code alone is not enough. Being able to see the target date, input files, record count diffs, last successful run time, retry command, and contact information together makes morning decisions much easier.

You do not need to build automatic recovery right away. In fact, simply clarifying when a retry is safe is a good enough starting point. It helps to design notifications not as announcements of failure, but as interfaces for the next human decision.

I think this same mindset applies when bringing AI agents into operations. Even when an agent detects something, you do not need to automate the entire response. Deciding in advance how far the agent should judge and where to hand off to a person makes the whole system more robust in the end.