Datadog Provider: Add Support for Querying Monitor Count and Status

[michelleheh]

We currently use argo rollouts, and we found that using the current set of supported Datadog Metrics is very rigid for developers to iterate on.

I want to propose a new feature where instead of querying specify DataDog APM, metrics or logs, there can be a query for Datadog Monitors instead. Once the query is committed to the AnalysisTemplate, developer can iterate on rollback trigger by just modifying the monitors.

AS a developer
I CAN specify a DataDog monitor query that gets all the monitors that are currently alerting
SO that canary rolls back when any monitor is alerting

For example, it can look something like

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: monitor-alert-check
spec:
  metrics:
    - name: alerting-canary-monitors
      # How often to check Datadog Monitors API
      interval: 1m 
      
      # The check fails if the result (number of alerting monitors) is greater than 0
      successCondition: result == 0 
      failureLimit: 1
      
      provider:
        datadog:
          # This is the proposed, hypothetical field for querying Datadog Monitors
          # The provider would interpret this as a search query for the Monitors API.
          monitorQuery: 'status:alert tag:"canary:true" service: my-service' 
          
          # The hypothetical Datadog provider would return the count of monitors 
          # matching the monitorQuery as the 'result'.

See DataDog API doc on getting monitors with available query params: https://docs.datadoghq.com/api/latest/monitors/#get-all-monitors

[mwarkentin]

I think this issue is related: #4059

[kostis-codefresh]

From @zachaller in the CNCF slack

"With all the complexity and rich feature set that datadog has in querying it might make sense to start a new datadog plugin that is more feature rich than the built-in"

Check https://argo-rollouts.readthedocs.io/en/stable/analysis/plugins/#metric-plugins