Alertmanager decided it simply couldn’t route alerts to your notification receivers because its routing rules were either missing, malformed, or pointed to receivers that didn’t exist.
This is the classic "Alertmanager isn’t sending alerts" problem, and it almost always boils down to its configuration. The two main culprits are the route block, which dictates where alerts go, and receivers, which define how they get sent.
Here’s a systematic breakdown of what’s likely broken and how to fix it:
1. The Root Route is Missing or Incorrect
Alertmanager requires a top-level route block in its configuration. This is the entry point for all alerts. If it’s absent or misconfigured, nothing gets routed.
Diagnosis:
Check your alertmanager.yml file. Look for a top-level route: key.
# This is what you SHOULD see at the top
route:
receiver: 'default-receiver' # Or whatever your default is
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
Cause:
The route block is completely missing from alertmanager.yml.
Fix:
Add a basic route block to your alertmanager.yml file.
route:
receiver: 'default-receiver'
group_by: ['alertname']
Why it works: This establishes the initial point of entry for all alerts. Without it, Alertmanager has no idea where to begin processing them.
Cause:
The receiver specified in the root route does not exist in the receivers section of your configuration.
Diagnosis:
Run amtool config routes on your Alertmanager instance (if amtool is available) or manually inspect your alertmanager.yml. Ensure the receiver name in the root route matches exactly one of the receiver names defined in the receivers list.
# alertmanager.yml
route:
receiver: 'this-receiver-does-not-exist' # Problematic!
# ... other route settings
receivers:
- name: 'actual-receiver-name'
webhook_configs:
- url: 'http://localhost:5001/'
Fix:
Correct the receiver name in the root route to match an existing receiver.
route:
receiver: 'actual-receiver-name' # Corrected!
# ... other route settings
Why it works: Alertmanager needs a valid destination to send alerts. If the specified receiver isn’t defined, it can’t fulfill the routing request.
2. Routing Rules are Malformed or Conflicting
Alertmanager uses a hierarchical routing tree. Alerts are matched against rules from top to bottom. If a rule is syntactically incorrect or logically flawed, it can break the routing for alerts that should match it.
Cause:
A match or match_re label selector in a routing rule is invalid YAML or uses incorrect syntax.
Diagnosis:
Carefully review your alertmanager.yml for any route blocks with match or match_re keys. Look for missing quotes, incorrect operators, or invalid regular expressions.
# alertmanager.yml
route:
receiver: 'default'
routes:
- receiver: 'critical-alerts'
match:
severity: 'critical' # Correct
match_re:
service: '^(database|cache)$' # Correct
- receiver: 'warning-alerts'
match:
severity: 'warning' # Correct
match_re:
service: 'web-server' # This might be fine, but check logic
- receiver: 'misconfigured-alerts'
match: severity = 'info' # Incorrect syntax! Should be severity: 'info'
Fix:
Correct the syntax within your match and match_re selectors. Ensure they are valid YAML and follow the expected label-selector format.
# alertmanager.yml
route:
receiver: 'default'
routes:
- receiver: 'critical-alerts'
match:
severity: 'critical'
match_re:
service: '^(database|cache)$'
- receiver: 'warning-alerts'
match:
severity: 'warning'
match_re:
service: 'web-server'
- receiver: 'misconfigured-alerts'
match:
severity: 'info' # Corrected syntax
Why it works: Alertmanager parses these selectors to determine which alerts belong to a specific route. Incorrect syntax prevents this parsing, making the rule unusable.
Cause:
A routing rule has no receiver defined.
Diagnosis:
In your alertmanager.yml, examine each route block within routes:. Every such block must have a receiver key, unless it’s intended to be an intermediate node that only contains further nested routes.
# alertmanager.yml
route:
receiver: 'default'
routes:
- receiver: 'critical-alerts'
match:
severity: 'critical'
- # PROBLEM: This route has no receiver and no further routes!
match:
severity: 'warning'
Fix: Either assign a receiver to the problematic route or remove it if it’s not serving a purpose.
# alertmanager.yml
route:
receiver: 'default'
routes:
- receiver: 'critical-alerts'
match:
severity: 'critical'
- receiver: 'warning-receiver' # Added receiver
match:
severity: 'warning'
Why it works: Alertmanager needs a concrete destination for every alert that matches a route. A route without a specified receiver (and without further nested routes) is a dead end.
3. Receivers are Not Properly Defined
Even if Alertmanager can route an alert to a receiver name, if that receiver isn’t configured correctly, the alert won’t be sent.
Cause:
A receiver is defined but has no notification configuration (e.g., email_configs, webhook_configs, slack_configs).
Diagnosis:
Check your alertmanager.yml for receiver definitions. Ensure each receiver has at least one configuration block for sending notifications.
# alertmanager.yml
receivers:
- name: 'my-webhook-receiver'
webhook_configs: # This is good
- url: 'http://example.com/hook'
- name: 'my-email-receiver'
email_configs: # This is good
- to: 'alerts@example.com'
- name: 'broken-receiver'
# NO notification config here!
Fix: Add the appropriate notification configuration block to the receiver.
# alertmanager.yml
receivers:
- name: 'my-webhook-receiver'
webhook_configs:
- url: 'http://example.com/hook'
- name: 'my-email-receiver'
email_configs:
- to: 'alerts@example.com'
- name: 'broken-receiver'
webhook_configs: # Added configuration
- url: 'http://example.com/default-hook'
Why it works:
The receiver definition tells Alertmanager what to call a group of notification settings. The *_configs blocks tell it how to actually send the notifications. Without the latter, the receiver is just a name with no action.
Cause: The notification configuration within a receiver is incorrect (e.g., invalid URL, missing API key, incorrect Slack channel).
Diagnosis:
This is harder to diagnose directly from alertmanager.yml alone. You’ll often see specific errors in the Alertmanager logs indicating connection failures or authentication problems for a particular receiver. Double-check the syntax and values for url, api_url, token, channel, to, etc., for the relevant receiver.
Fix: Correct the specific parameters within the notification configuration. For webhooks, ensure the URL is reachable and the method (POST by default) is expected. For Slack, verify bot tokens and channel IDs.
Why it works: Alertmanager attempts to communicate with external services to deliver alerts. Incorrect credentials or endpoints prevent these communications from succeeding.
4. Silences Interfering with Routes
While silences don’t break the routing mechanism itself, they can prevent alerts from being delivered to receivers if the silence matches the alert and the receiver.
Cause: A silence is active that matches the alert’s labels and is configured to inhibit notifications for the target receiver.
Diagnosis: Check your active silences in Alertmanager’s UI or via the API. Look for silences that match the labels of the alerts you expect to receive.
# Example using amtool to list silences
amtool --config.file=alertmanager.yml silence list
Fix:
If the silence is no longer needed, expire it or delete it. If it’s still necessary, ensure its inhibit_labels and target_match (or target_match_re) are correctly configured to only suppress alerts when intended.
Why it works: Silences are designed to temporarily mute alerts. If configured broadly, they can inadvertently suppress alerts that should still be seen.
5. Configuration Reload Failure
Alertmanager needs to reload its configuration after alertmanager.yml changes. If this reload fails, it will continue using the old, potentially broken, configuration.
Cause:
Alertmanager failed to parse the updated alertmanager.yml and did not reload successfully.
Diagnosis: Check the Alertmanager logs for errors related to configuration parsing or reloading. You can also query the Alertmanager API to check the current configuration version or status.
# Get current config
curl -s http://localhost:9093/-/config | jq .
# Check metrics for reloads (look for config_reload_success/failure)
curl -s http://localhost:9093/-/metrics | grep config_reload
Fix:
Correct any YAML syntax errors or invalid configurations in alertmanager.yml that prevented the reload. Then, trigger a reload by sending a SIGHUP signal to the Alertmanager process or by making a POST request to its /-/reload endpoint.
# Trigger reload via API
curl -X POST http://localhost:9093/-/reload
Why it works: Alertmanager only applies configuration changes after a successful reload. Fixing the configuration and triggering the reload ensures the new rules are active.
After fixing these, the next error you’ll likely encounter is that your alerts are being grouped too aggressively, leading to fewer, larger notifications than you’d expect.