Skip to content

Conversation

@piochelepiotr
Copy link
Contributor

@piochelepiotr piochelepiotr commented Dec 5, 2025

What does this PR do?

Adds support for AWS MSK IAM authentication to the Kafka Consumer integration using SASL/OAUTHBEARER mechanism.

Motivation

Customers using Amazon MSK clusters with IAM authentication were unable to monitor their Kafka infrastructure with the Datadog Agent. Previously, they had to use SASL/SCRAM or mTLS authentication, which requires manual credential management.

This change enables monitoring AWS MSK clusters using IAM roles, providing:

  • No credential management (uses AWS IAM roles automatically)
  • Fine-grained access control via IAM policies
  • Automatic token rotation

Configuration:
instances:

  • kafka_connect_str: "broker:9098"
    security_protocol: SASL_SSL
    sasl_mechanism: OAUTHBEARER
    sasl_oauth_token_provider:
    method: aws_msk_iamImplementation:
  • Uses aws-msk-iam-sasl-signer-python to generate AWS IAM tokens
  • OAuth callback provides tokens to confluent-kafka client
  • Fully backwards compatible with existing OIDC configurations
  • Tested end-to-end on real AWS MSK cluster

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

@codecov
Copy link

codecov bot commented Dec 5, 2025

Codecov Report

❌ Patch coverage is 74.13793% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.02%. Comparing base (8c6c226) to head (d51d7a5).
⚠️ Report is 13 commits behind head on master.

Additional details and impacted files
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

"""OAuth callback that generates AWS MSK IAM authentication tokens."""
try:
# Get AWS region from config or detect from environment
region = boto3.session.Session().region_name or 'us-east-1'
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be validated in config.py somehow instead? Defaulting to us-east-1 seems like it could cause subtle failures that are a headache to debug

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants