Monitor metrics when Prometheus not available.

[Makdaam]

Monitor component should emit a metric that the prometheus metrics are not collectable instead of a flood of error log messages.

/priority high

[jim-minter]

keeping this open to implement preaggregates + dashboard

[Makdaam]

Dashboard configured, preaggregates are working.
Also configured an example monitor to alert us when 3 errors of the same type occur in 5min (I noticed single errors can happen due to request timeouts). See https://dev.azure.com/msazure/AzureRedHatOpenShift/_wiki/wikis/ARO.wiki/53765/WIP-Alerting

[Makdaam]

After looking at the alert history it seems very noisy. especially around emitPrometheusAlerts-fm erroring. Next step investigate the main cause of the emitPrometheusAlerts-fm issues.

[Makdaam]

Checked and the errors which are shown are not actionable. I'm leaving the alert in "Silent" state in Geneva.

I'll revisit the topic around 11th of June.