diff --git a/TOC-tidb-cloud-essential.md b/TOC-tidb-cloud-essential.md index cdeb3c7ca755d..d53eb7ec44d85 100644 --- a/TOC-tidb-cloud-essential.md +++ b/TOC-tidb-cloud-essential.md @@ -160,6 +160,7 @@ - [Connect to Amazon RDS](/tidb-cloud/serverless-private-link-connection-to-aws-rds.md) - [Connect to Alibaba Cloud RDS](/tidb-cloud/serverless-private-link-connection-to-alicloud-rds.md) - [Connect to Confluent Cloud on AWS](/tidb-cloud/serverless-private-link-connection-to-aws-confluent.md) + - [Connect to Amazon MSK Provisioned](/tidb-cloud/serverless-private-link-connection-to-amazon-msk.md) - [Connect to Self-Hosted Kafka on Alibaba Cloud](/tidb-cloud/serverless-private-link-connection-to-self-hosted-kafka-in-alicloud.md) - [Connect to Self-Hosted Kafka on AWS](/tidb-cloud/serverless-private-link-connection-to-self-hosted-kafka-in-aws.md) - Audit Management diff --git a/tidb-cloud/essential-changefeed-sink-to-kafka.md b/tidb-cloud/essential-changefeed-sink-to-kafka.md index 4326ec396012e..297ab350619ce 100644 --- a/tidb-cloud/essential-changefeed-sink-to-kafka.md +++ b/tidb-cloud/essential-changefeed-sink-to-kafka.md @@ -33,13 +33,14 @@ Ensure that your {{{ .essential }}} cluster can connect to the Apache Kafka serv Private link connections leverage **Private Link** technologies from cloud providers to enable resources in your VPC to connect to services in other VPCs using private IP addresses, as if those services were hosted directly within your VPC. -{{{ .essential }}} currently supports Private Link connections only for self-hosted Kafka and Confluent Cloud Dedicated clusters. It does not support direct integration with MSK or other Kafka SaaS services. +{{{ .essential }}} currently supports Private Link connections only for self-hosted Kafka, Confluent Cloud Dedicated clusters, and Amazon MSK Provisioned. It does not support direct integration with other Kafka SaaS services. To set up a Private Link connection based on your Kafka deployment and cloud provider, see the following guides: - [Connect to Confluent Cloud on AWS via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-aws-confluent.md) - [Connect to AWS Self-Hosted Kafka via Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-self-hosted-kafka-in-aws.md) - [Connect to Alibaba Cloud Self-Hosted Kafka via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-self-hosted-kafka-in-alicloud.md) +- [Connect to Amazon MSK Provisioned via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-amazon-msk.md) @@ -90,7 +91,7 @@ The steps vary depending on the connectivity method you select. 1. In **Connectivity Method**, select **Private Link**. 2. In **Private Link Connection**, select the private link connection that you created in the [Network](#network) section. Make sure the Availability Zones of the private link connection match those of the Kafka deployment. -3. Fill in the **Bootstrap Port** that you obtained from the [Network](#network) section. +3. Fill in the **Bootstrap Port** that you obtained from the [Network](#network) section. If you are using the Amazon MSK Provisioned private link connection, you can skip this field. 4. Select an **Authentication** option according to your Kafka authentication configuration. - If your Kafka does not require authentication, keep the default option **Disable**. diff --git a/tidb-cloud/serverless-private-link-connection-to-amazon-msk.md b/tidb-cloud/serverless-private-link-connection-to-amazon-msk.md new file mode 100644 index 0000000000000..3672c46330bb0 --- /dev/null +++ b/tidb-cloud/serverless-private-link-connection-to-amazon-msk.md @@ -0,0 +1,157 @@ +--- +title: Connect to Amazon MSK Provisioned via a Private Link Connection +summary: Learn how to connect to an Amazon MSK Provisioned cluster using an Amazon MSK Provisioned private link connection. +--- + +# Connect to Amazon MSK Provisioned via a Private Link Connection + +This document describes how to connect a {{{ .essential }}} cluster to an [Amazon MSK Provisioned](https://docs.aws.amazon.com/msk/latest/developerguide/msk-provisioned.html) cluster using an [Amazon MSK Provisioned private link connection](/tidb-cloud/serverless-private-link-connection.md#create-an-amazon-msk-provisioned-private-link-connection). + +## Prerequisites for {{{ .essential }}} {#prerequisites-for-essential} + +- Your {{{ .essential }}} cluster is hosted on AWS and is active. Retrieve and save the following for later use: + + - AWS Account ID + - Availability zones (AZ) + +To view the AWS account ID and availability zones: + +1. In the [TiDB Cloud console](https://tidbcloud.com), go to the cluster overview page of your TiDB cluster, and then click **Settings** > **Networking** in the left navigation pane. +2. In the **Private Link Connection For Dataflow** area, click **Create Private Link Connection**. +3. In the dialog, note the AWS account ID and availability zones. + +## Prerequisites for the Amazon MSK Provisioned cluster + +Before you begin, ensure the following for your Amazon MSK Provisioned cluster: + +- **Region and AZ**: Your Amazon MSK Provisioned cluster is in the same AWS region as your {{{ .essential }}} cluster, and the availability zones of the MSK cluster are the same as your TiDB Cloud cluster. +- **Authentication**: [SASL/SCRAM authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html) is required for the MSK cluster. +- **Broker type**: Do not use the `t4.small` broker type. It does not support private link. + +For more requirements, see [Amazon MSK multi-VPC private connectivity in a single Region](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html#mvpc-requirements). + + If you do not have an Amazon MSK Provisioned cluster, [create one](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster.html) in the same region and the same availability zone as your {{{ .essential }}} cluster, and then [set up SASL/SCRAM authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-tutorial.html) for the created cluster. + +- **Secret name**: the secret name must start with `AmazonMSK_`. +- **Encryption**: do not use the default encryption key. Create a new custom AWS KMS key for your secret. + +## Step 1. Set up Kafka ACLs for TiDB Cloud access + +You must set up Kafka ACLs so that TiDB Cloud can access your Amazon MSK Provisioned cluster. You can use SASL/SCRAM authentication (recommended) or IAM authentication to set up ACLs. + + +
+ +Use this method to create ACLs in the same VPC as your MSK cluster using SASL/SCRAM authentication. + +1. Create an EC2 instance (Linux) in the VPC where your MSK cluster is located and SSH to it. + +2. Download Kafka and OpenJDK: + + ```shell + wget https://archive.apache.org/dist/kafka/3.7.1/kafka_2.13-3.7.1.tgz + tar -zxf kafka_2.13-3.7.1.tgz + wget https://download.java.net/java/GA/jdk22.0.2/c9ecb94cd31b495da20a27d4581645e8/9/GPL/openjdk-22.0.2_linux-x64_bin.tar.gz + tar -zxf openjdk-22.0.2_linux-x64_bin.tar.gz + ``` + +3. Set the environment. Replace the path with your actual path. + + ```shell + export PATH=$PATH:/home/ec2-user/jdk-22.0.2/bin + ``` + +4. Create a file named `scram-client.properties` with the following content. Replace `username` and `pswd` with your SASL/SCRAM credentials: + + ```properties + security.protocol=SASL_SSL + sasl.mechanism=SCRAM-SHA-512 + sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ + username="username" \ + password="pswd"; + ``` + +5. Create the ACLs. Replace `bootstrap-server` with your MSK bootstrap server address and port (for example, `b-2.xxxxx.c18.kafka.us-east-1.amazonaws.com:9096`), and replace the path to Kafka if needed: + + ```shell + /home/ec2-user/kafka_2.13-3.7.1/bin/kafka-acls.sh --bootstrap-server --command-config scram-client.properties --add --allow-principal User: --operation All --topic '*' + /home/ec2-user/kafka_2.13-3.7.1/bin/kafka-acls.sh --bootstrap-server --command-config scram-client.properties --add --allow-principal User: --operation All --group '*' + /home/ec2-user/kafka_2.13-3.7.1/bin/kafka-acls.sh --bootstrap-server --command-config scram-client.properties --add --allow-principal User: --operation All --cluster '*' + ``` + + The principal `User:` is the SASL/SCRAM user that TiDB Cloud uses to access your MSK cluster. Use the username you configured for TiDB Cloud in your MSK ACLs. + +
+ +
+ +As an alternative to SASL/SCRAM, you can create ACLs in the same VPC as your MSK cluster using IAM authentication. The IAM user or role must have **Amazon MSK** and **Apache Kafka APIs for MSK** permissions. + +1. Create an EC2 instance (Linux) in the VPC where your MSK cluster is located and SSH to it. + +2. Download Kafka, OpenJDK, and the AWS MSK IAM auth JAR: + + ```shell + wget https://archive.apache.org/dist/kafka/3.7.1/kafka_2.13-3.7.1.tgz + tar -zxf kafka_2.13-3.7.1.tgz + wget https://download.java.net/java/GA/jdk22.0.2/c9ecb94cd31b495da20a27d4581645e8/9/GPL/openjdk-22.0.2_linux-x64_bin.tar.gz + tar -zxf openjdk-22.0.2_linux-x64_bin.tar.gz + wget https://github.com/aws/aws-msk-iam-auth/releases/download/v2.3.5/aws-msk-iam-auth-2.3.5-all.jar + ``` + +3. Configure the environment. Replace paths and credentials with your own values. + + ```shell + export PATH=$PATH:/home/ec2-user/jdk-22.0.2/bin + export CLASSPATH=/home/ec2-user/aws-msk-iam-auth-2.3.5-all.jar + export AWS_ACCESS_KEY_ID= + export AWS_SECRET_ACCESS_KEY= + ``` + +4. Create a file named `iam-client.properties` with the following content: + + ```properties + security.protocol=SASL_SSL + sasl.mechanism=AWS_MSK_IAM + sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; + sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler + ``` + +5. Create the ACLs. Replace `bootstrap-server` with your MSK bootstrap server address and port (for example, `b-1.xxxxx.c18.kafka.us-east-1.amazonaws.com:9098`), and replace the path to Kafka if needed: + + ```shell + /home/ec2-user/kafka_2.13-3.7.1/bin/kafka-acls.sh --bootstrap-server --command-config iam-client.properties --add --allow-principal User: --operation All --topic '*' + /home/ec2-user/kafka_2.13-3.7.1/bin/kafka-acls.sh --bootstrap-server --command-config iam-client.properties --add --allow-principal User: --operation All --group '*' + /home/ec2-user/kafka_2.13-3.7.1/bin/kafka-acls.sh --bootstrap-server --command-config iam-client.properties --add --allow-principal User: --operation All --cluster '*' + ``` + + The principal `User:` is the SASL/SCRAM user that TiDB Cloud uses to access your MSK cluster. Use the username you configured for TiDB Cloud in your MSK ACLs. + +
+ + +## Step 2. Configure the MSK cluster + +Update the following cluster configuration properties: + +- Set `auto.create.topics.enable=true`. +- Add `allow.everyone.if.no.acl.found=false` (required for SASL/SCRAM). +- Keep other properties unchanged or adjust them as needed. + +Apply the changes and wait for the cluster status to change from **Updating** to **Active**. + +## Step 3. Attach the cluster policy + +[Attach the cluster policy](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cluster-owner-action-policy.html) to allow TiDB Cloud to connect to your MSK cluster. Use the TiDB Cloud AWS account ID you obtained in [Prerequisites](#prerequisites-for-essential). + +## Step 4. Turn on multi-VPC connectivity + +After the cluster is active, [turn on multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cluster-owner-action-turn-on.html) for the MSK cluster. Multi-VPC connectivity is required for AWS PrivateLink. To connect from TiDB Cloud, you must enable SASL/SCRAM authentication. + +Wait for the cluster status to change from **Updating** to **Active** again. + +## Step 5. Create an Amazon MSK Provisioned private link connection in TiDB Cloud + +Create the private link connection in TiDB Cloud using the `ARN` of your MSK cluster. + +For more information, see [Create an Amazon MSK Provisioned private link connection](/tidb-cloud/serverless-private-link-connection.md#create-an-amazon-msk-provisioned-private-link-connection). \ No newline at end of file diff --git a/tidb-cloud/serverless-private-link-connection.md b/tidb-cloud/serverless-private-link-connection.md index 61d43b530d8b5..51f99dc2938f5 100644 --- a/tidb-cloud/serverless-private-link-connection.md +++ b/tidb-cloud/serverless-private-link-connection.md @@ -19,6 +19,10 @@ This type of private link connection enables TiDB Cloud clusters on **AWS** to c The private link connection can access various AWS services, such as RDS instances and Kafka services, by associating them with the endpoint service. +### Amazon MSK Provisioned + +This type of private link connection enables TiDB Cloud clusters on **AWS** to connect to your [Amazon MSK Provisioned](https://docs.aws.amazon.com/msk/latest/developerguide/msk-provisioned.html) with private link. + ### Alibaba Cloud Endpoint Service This type of private link connection enables TiDB Cloud clusters on **Alibaba Cloud** to connect to your [Alibaba Cloud endpoint service](https://www.alibabacloud.com/help/en/privatelink/share-your-service/#51976edba8no7) powered by Alibaba Cloud PrivateLink. @@ -81,6 +85,30 @@ To create a private link connection using the TiDB Cloud CLI: +## Create an Amazon MSK Provisioned private link connection + +You can create an Amazon MSK Provisioned private link connection using the TiDB Cloud console. + +Before creating the Amazon MSK Provisioned private link connection, ensure that your Amazon MSK Provisioned cluster turns on multi-VPC connectivity. See [Connect to Amazon MSK Provisioned via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-amazon-msk.md) for more details. + +1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/project/clusters) page of your project. + + > **Tip:** + > + > You can use the combo box in the upper-left corner to switch between organizations, projects, and clusters. + +2. Click the name of your target cluster to go to its overview page, and then click **Settings** > **Networking** in the left navigation pane. + +3. In the **Private Link Connection For Dataflow** area, click **Create Private Link Connection**. + +4. In the **Create Private Link Connection** dialog, enter the required information: + + - **Private Link Connection Name**: enter a name for the private link connection. + - **Connection Type**: select **Amazon MSK Provisioned**. If this option is not displayed, ensure that your cluster is created on AWS. + - **MSK Cluster ARN**: enter the ARN of your Amazon MSK Provisioned cluster, for example, `arn:aws:kafka:us-east-1:385595570414:cluster//xxxx`. + +5. Click **Create**. + ## Create an Alibaba Cloud Endpoint Service private link connection You can create an Alibaba Cloud Endpoint Service private link connection using the TiDB Cloud console or the TiDB Cloud CLI. @@ -147,6 +175,7 @@ Different private link connection types support attaching different domain types |--------------------------------|-------------------------------------------| | AWS Endpoint Service | | | Alibaba Cloud Endpoint Service | TiDB Cloud managed (`alicloud.tidbcloud.com`) | +| Amazon MSK Provisioned | Domain attachment is not supported. | If your domain is not included in this table, contact [TiDB Cloud Support](/tidb-cloud/tidb-cloud-support.md) to request support. @@ -281,6 +310,7 @@ ticloud serverless private-link-connection delete -c --private-link - [Connect to Confluent Cloud via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-aws-confluent.md) - [Connect to Amazon RDS via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-aws-rds.md) +- [Connect to Amazon MSK Provisioned via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-amazon-msk.md) - [Connect to Alibaba Cloud ApsaraDB RDS for MySQL via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-alicloud-rds.md) - [Connect to AWS Self-Hosted Kafka via Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-self-hosted-kafka-in-aws.md) - [Connect to Alibaba Cloud Self-Hosted Kafka via a Private Link Connection](/tidb-cloud/serverless-private-link-connection-to-self-hosted-kafka-in-alicloud.md)