Redpanda Connector¶

Use the Redpanda Connector to ingest data from your existing Redpanda cluster and load it into Tinybird so that you can quickly turn it into high-concurrency, low-latency REST APIs.

The Redpanda Connector is fully managed and requires no additional tooling. Connect Tinybird to your Redpanda cluster, choose a topic, and Tinybird automatically begins consuming messages from Redpanda.

The Redpanda Connector is:

• Easy to use. Connect to your Redpanda cluster in seconds. Choose your topics, define your schema, and ingest millions of events per second into a fully managed OLAP.
• SQL-based. Using nothing but SQL, query your Redpanda data and enrich it with dimensions from your database, warehouse, or files.
• Secure. Use Auth tokens to control access to API endpoints. Implement access policies as you need. Support for row-level security.

Using the UI¶

To connect Tinybird to your Redpanda cluster, select the + icon next to the data project section, select Data Source, and select Redpanda from the list of available Data Sources.

Enter the following details:

• Connection name: A name for the Redpanda connection in Tinybird.
• Bootstrap Server: The comma-separated list of bootstrap servers (including port numbers).
• Key: The Key component of the Redpanda API Key.
• Secret: The Secret component of the Redpanda API Key.
• Decode Avro messages with schema registry: Optionally, you can enable Schema Registry support to decode Avro messages. Enter the Schema Registry URL, username, and password when prompted.

Once you have entered the details, select Connect. This creates the connection between Tinybird and Redpanda. A list of your existing topics appears, where you can select the topic to consume from. Tinybird creates a Group ID that specifies the name of the consumer group this consumer belongs to. You can customize the Group ID, but ensure that your Group ID has read permissions to the topic.

Once you have chosen a topic, you can select the starting offset to consume from. You can choose to consume from the latest offset or the earliest offset. If you choose the earliest offset, Tinybird consumes all messages from the beginning of the topic. If you choose the latest offset, Tinybird consumes only messages produced after the connection is created. Select the offset, and select Next.

Tinybird then consumes a sample of messages from the topic and displays the schema. You can adjust the schema and Data Source settings as needed, then select Create Data Source to create the Data Source.

Tinybird begins consuming messages from the topic and loading them into the Data Source.

Using .datasource files¶

If you are managing your Tinybird resources in files, there are several settings available to configure the Redpanda Connector in .datasource files.

See the datafiles docs for more information.

The following is an example of Kafka .datasource file for an already existing connection:

SCHEMA >
  `__value` String,
  `__topic` LowCardinality(String),
  `__partition` Int16,
  `__offset` Int64,
  `__timestamp` DateTime,
  `__key` String
  `__headers` Map(String,String)

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

# Connection is already available. If you
# need to create one, add the required fields
# on an include file with the details.
KAFKA_CONNECTION_NAME my_connection_name
KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id
KAFKA_STORE_HEADERS true

Columns of the Data Source¶

When you connect a Kafka producer to Tinybird, Tinybird consumes optional metadata columns from that Kafka record and writes them to the Data Source.

The following fields represent the raw data received from Kafka:

• __value: A String representing the entire unparsed Kafka record inserted.
• __topic: The Kafka topic that the message belongs to.
• __partition: The Kafka partition the message belongs to.
• __offset: The Kafka offset of the message.
• __timestamp: The timestamp stored in the Kafka message received by Tinybird.
• __key: The Kafka message key.
• __headers: Headers parsed from the incoming topic messages. See Using custom Kafka headers for advanced message processing.

Using INCLUDE to store connection settings¶

To avoid configuring the same connection settings across many files, or to prevent leaking sensitive information, you can store connection details in an external file and use INCLUDE to import them into one or more .datasource files.

You can find more information about INCLUDE in the Advanced Templates documentation.

As an example, you may have two Redpanda .datasource files, which re-use the same Redpanda connection. You can create an INCLUDE file that stores the Redpanda connection details.

The Tinybird project may use the following structure:

ecommerce_data_project/
├── datasources/
│   └── connections/
│       └── my_connector_name.incl
│   └── my_kafka_datasource.datasource
│   └── another_datasource.datasource
├── endpoints/
├── pipes/

Where the file my_connector_name.incl has the following content:

KAFKA_CONNECTION_NAME my_connection_name
KAFKA_BOOTSTRAP_SERVERS my_server:9092
KAFKA_KEY my_username
KAFKA_SECRET my_password

And the Redpanda .datasource files look like the following:

SCHEMA >
    `value` String,
    `topic` LowCardinality(String),
    `partition` Int16,
    `offset` Int64,
    `timestamp` DateTime,
    `key` String

ENGINE "MergeTree"
ENGINE_PARTITION_KEY "toYYYYMM(timestamp)"
ENGINE_SORTING_KEY "timestamp"

INCLUDE "connections/my_connection_name.incl"

KAFKA_TOPIC my_topic
KAFKA_GROUP_ID my_group_id

Redpanda logs¶

SELECT *
FROM tinybird.datasources_ops_log
WHERE datasource_id = 't_1234'
  AND event_type = 'append-kafka'
  AND timestamp > now() - INTERVAL 1 day
ORDER BY timestamp DESC

SELECT *
FROM tinybird.kafka_ops_log
WHERE datasource_id = 't_1234'
  AND timestamp > now() - interval 1 day
  AND msg_type IN ['info', 'warning', 'error']