Prometheus Alertmanager Silencing alerts via UI does not prevent first occurence of alert to trigger

Can the ‘Silences’ feature on the Alertmanager UI be used to silence new alerts before they are sent to a receiver? This would be useful while performing an upgrade in an environment. My alerts contain a label 'env' (to identify a given environment) and I can add a silence rule via the UI. The problem is that it will only silence AFTER the first occurrence of an alert in the 'env'/environment. In the meantime, an alert has been sent to the Alertmanager receiver.I have also tried changing values in the alertmanager.conf for group_interval from ...Read more

prometheus alertmanager - How to fix 'err="yaml: unmarshal errors:\n line 21: field routes not found in type config.plain"'

I'm trying to set up Alertmanager to send to 2 different receivers based on the value of a label in the Prometheus rule. For example, if responsible_team label equals web, send to the web team receiver. If reponsible_team label equals database, send t o the database team receiver.I've followed Alertmanager's documentation and they say you should set up multiple routes with a match then labelName and labelValue underneath it.route: group_by: ['alertname','entity'] group_wait: 30s group_interval: 5m repeat_interval: 15m receiver: defaultrout...Read more

prometheus alertmanager - How to fix 'err="yaml: unmarshal errors:\n line 21: field routes not found in type config.plain"'

I'm trying to set up Alertmanager to send to 2 different receivers based on the value of a label in the Prometheus rule. For example, if responsible_team label equals web, send to the web team receiver. If reponsible_team label equals database, send t o the database team receiver.I've followed Alertmanager's documentation and they say you should set up multiple routes with a match then labelName and labelValue underneath it.route: group_by: ['alertname','entity'] group_wait: 30s group_interval: 5m repeat_interval: 15m receiver: defaultrout...Read more