Skip to main content
CockroachDB can stream change data out to Apache Kafka using OAuth authentication. OAuth 2.0 authentication uses credentials managed by a third-party provider (IdP) to authenticate with Kafka instead of requiring you to provide your Kafka cluster credentials directly in a statement. Your provider’s authentication server will issue a temporary token, giving you flexibility to apply access rules on the credentials that your IdP provides. In this tutorial, you will set up OAuth authentication for your Kafka changefeed sink using Okta. You will create an OAuth 2.0 endpoint in Okta that will provide a temporary credential token that the changefeed will use to connect to your Kafka cluster. An overview of the workflow involves:
  1. Creating an Okta application integration and authorization server scope.
  2. Configuring the Kafka cluster.
  3. Streaming the changefeed to the Kafka cluster.

​
Before you begin

Before starting this tutorial, you will need:
  • A CockroachDB cluster. You can use a CockroachDB Cloud or CockroachDB self-hosted cluster.
    • If you are using CockroachDB Basic, Standard, or Advanced, see the guide. For CockroachDB self-hosted clusters, see the page.
  • An Okta Developer account.
  • (Optional) A Kafka cluster. This tutorial includes Kafka cluster setup. Note that this tutorial was tested with Kafka version 2.8.2. The Kafka cluster configuration may vary with different versions, but the Okta setup and changefeed creation will be the same.
  • The CHANGEFEED privilege in order to create and manage changefeed jobs. Refer to for more details.
This tutorial uses the Cockroach Labs workload as an example database.

​
Step 1. Create an Okta application and scope

In this step, you will create an application integration in your Okta developer account.
  1. Log in to your Okta Developer account and navigate to Applications in the sidebar.
  2. In the dropdown, click Applications and then Create App Integration.
  3. On the Create a new app integration menu, select API services for OAuth 2.0 access tokens. Click Next.
  4. Add an App integration name and then Save.

​
Step 2. Configure an Okta authorization server with a scope

Once you have created the application integration, you’ll configure an authorization server with a scope. The authorization server produces the OAuth 2.0 tokens the changefeed will use to connect to the Kafka sink. Scopes contained within an authorization server provide a way to limit access that operations have to the OAuth 2.0 access tokens.
  1. Navigate to Security in the sidebar and then API. From here, you can add a new authorization server, or use the default authorization server. This tutorial will use the default server.
  2. Click on the name of your authorization server and then select the Scopes tab.
  3. Select Add Scope:
    • Add kafka as the name for your scope and include an optional description.
    • For User Consent, leave the default Implicit selected.
    • Check the box to set this as the Default Scope.
    • Optionally check Include in public metadata if you wish to include this scope in public metadata.
  4. Click Create.

​
Step 3. Create a Kafka sink

If you do not already have a Kafka cluster, create one with the instructions in this step. Note that the following instructions install Kafka v2.8.2 on a Linux machine:
  1. Update the latest version of packages on your machine:
  2. Install the default-jre package, which is a requirement for Kafka:
  3. Download Kafka v2.8.2:
  4. Extract the Kafka download:

​
Step 4. Update Kafka configuration

In this step, you will update configuration files in your Kafka cluster to set up the SASL mechanism for handling OAuth tokens.
  1. Move to the following directory:
  2. Open the server.properties file to edit:
    Add the following to the end of the server.properties file:
    This defines:
    • The SASL mechanism as OAUTHBEARER.
    • The ports that Kafka brokers listen for client and inter-broker communication. In this tutorial, you’ll be writing to the cluster through port :9093. The configuration also lists :9092 for a non-credentialed listener that can read from the cluster without needing to authenticate in the same way as the writer.
    • The login callback handler that will receive the OAuth token.
    • The server callback handler that will verify the token.
  3. Create a file in the config directory called kafka_server_jaas.conf:
    Add the following contents:
    The OAuthBearerLoginModule will ask the OauthAuthenticateLoginCallbackHandler (configured in the server.properties file) to return an OAuthBearerToken.

​
Step 5. Prepare Kafka OAuth libraries

The SASL callback handlers added in Step 4 are part of a separate library that you will need to install.
  1. Clone the library:
  2. Install Apache Maven, a tool for building Java-based projects:
  3. Move to the kafka-oauth directory:
  4. Run the following to compile and package the project:

​
Step 6. Add OAuth environment variables

In this step, you’ll export the environment variables that contain your OAuth client credentials. You will need the following information from your Okta developer account:
  • Your Okta domain, for example dev-12345678.okta.com
  • The absolute paths to your:
    • kafka-oauth directory
    • kafka_2.13-2.8.2 directory (or your Kafka version)
  • Your client ID and client secret from your Okta application (created in Step 1). You will need to build the value for the variable as {client ID}:{client secret} and base64 encode the whole value.
Export the following environment variables, or add them to your configuration file (e.g., .bash_profile or .bashrc) replacing the values in curly braces with those mentioned in the preceding list:

​
Step 7. Start your Kafka server

In this step, you will start your Kafka server, create a topic, and run your consumer:
  1. Move to the following directory:
  2. Start the Zookeeper server in one terminal:
  3. Start the Kafka server in another terminal:
  4. In another separate terminal window, create a topic to consume:
    To consume messages for this topic, run:

​
Step 8. Create a changefeed

In this step, you will create a changefeed authenticating with Okta. The Kafka URI must follow this format:
Note the following:
  • You can include your client ID directly (without any encoding).
  • You must base64 encode your client secret.
  • You must URL encode your Okta developer token URL, e.g., https://dev-12345678.okta.com/oauth2/default/v1/token will encode to https%3A%2F%2Fdev-12345678.okta.com%2Foauth2%2Fdefault%2Fv1%2Ftoken.
Since this is a long URI to run, you can create an to represent this URI:
Create a changefeed that will emit messages to the topic consumer:
In the terminal set up to consume messages, you will receive your changefeed output:

​
See also