Change Data Capture from CockroachDB to ConfluentCloud

Here's a simple tutorial for sending data from CockroachDB directly to Confluent Cloud using CockroachDB Change Data Capture, which is typically referred to as a Changefeed.

Setup Confluent Cloud

• Create Free 30 Day ConfluentCloud Cluster
• Add Confluent Cloud CLI

Setup Your Kafka Cluster

Get Kafka Resource ID

The ID list here for your Kafka cluster will be needed in the steps below

ccloud kafka cluster list

Create API Keys

The API Key and API Secret are needed for creating the CockroachDB Changefeed

ccloud api-key create --resource <RESOURCE ID>

Get Kafka End Point

The end point is needed to connect the Changefeed to Kafka

ccloud kafka cluster describe <RESOURCE ID>

Create Topic

ccloud kafka topic create demo_t --partitions 6

Start a Kafka Consumer to Verify Your Change Data Feed

ccloud kafka topic consume demo_t

Setup CockroachDB

Open a new terminal window and leave the Kafka consumer one open for later

Create CockroachDB Table

cockroach sql ...

create table t (k int default unique_rowid() primary key, v string);

Create Changefeed

When creating the changefeed, notice that you'll use 'kafka://' instead of using https:// or SASL_SSL://. Also, be sure to include your API Key and Secret in the Changefeed.

CREATE CHANGEFEED FOR TABLE t INTO 'kafka://<CONFLUENT CLOUD URL>:9092?sasl_enabled=true&sasl_password=<API SECRET>&sasl_user=<API KEY>&tls_enabled=true&topic_prefix=demo_' WITH updated, key_in_value, format = json;

Insert Some Rows

insert into t (v) values ('one');
insert into t (v) values ('two');
insert into t (v) values ('three');

Verify data is showing up in your consumer app

view raw cdc_cockroach_to_confluent.md hosted with ❤ by GitHub