Stream from RudderStack

Learn two different methods to send events from RudderStack to Tinybird.

To better understand the behavior of their customers, companies need to unify timestamped data coming from a wide variety of products and platforms. Typical events to track would be 'sign up', 'login', 'page view' or 'item purchased'. A customer data platform can be used to capture complete customer data like this from wherever your customers interact with your brand. It defines events, collects them from different platforms and products, and routes them to where they need to be consumed.

RudderStack is an open source customer data pipeline tool. It collects, processes, and routes data from your websites, apps, cloud tools, and data warehouse. By using Tinybird's event ingestion endpoint for high-frequency ingestion as a Webhook in RudderStack, you can stream customer data in real time to Data Sources.

Option 1: A separate Data Source for each event type

This is the preferred approach. It sends each type of event to a corresponding Data Source.

The advantages of this method are:

  • Your data is well organized from the start.
  • Different event types can have different attributes (columns in their Data Source).
  • Whenever new attributes are added to an event type, a prompt asks you to add new columns.
  • New event types get a new Data Source.

Start by generating a Token in the UI to allow RudderStack to write to Tinybird.

Create a Tinybird Token

Go to the Workspace in Tinybird where you want to receive data and select Tokens in the side panel. Create a new Token by selecting Create Token.

Give your Token a descriptive name. In the section DATA SOURCES SCOPES select Data Sources management to give your Token permission to create Data Sources. Select Save changes.

Create a RudderStack destination

In RudderStack, Select Destinations in the side panel and then New destination.

Select Webhook:

  1. Give the destination a descriptive name.
  2. Connect your sources, you can test with the Rudderstack Sample HTTP Source.
  3. Input the following Connection Settings:
  • Webhook URL: <https://<your_host>/v0/events>
  • URL Method: POST
  • Headers Key: Authorization
  • Headers Value: Bearer TINYBIRD_AUTH_TOKEN

On the next page, select Create new transformation.

You can code a function in the box to apply to events when this transformation is active using the following example snippet. In this function, you can dynamically append the target Data Source to the target URL of the Webhook. Give your transformation a descriptive name and a helpful description.

Transformation code
export function transformEvent(event, metadata){
    event.appendPath="?name=rudderstack_"+event.event.toLowerCase().replace(/[\s\.]/g, '_')
    return event;
}

This example snippet uses the prefix *rudderstack\_* followed by the name of the event in lower case, with its words separated by an underscore. For instance, a "Product purchased" event would go to a Data Source named rudderstack_product_purchased.

Save the transformation. Your destination has been created successfully.

Test ingestion

In Rudderstack, select Sources > Rudderstack Sample HTTP > Live events > Send test event and paste the provided curl command into your terminal. The event appears on the screen and is sent to Tinybird.

Option 2: All events in the same Data Source

This alternative approach sends all events into a single Data Source and then splits them using Tinybird. By pre-configuring the Data Source, any events that RudderStack sends are ingested with the full JSON object as a String in a single column. This is useful when you have complex JSON objects, but using JSONExtract to parse data from the JSON object after ingestion affects performance.

Tinybird detects new columns from parsing the data and asks if you want to save them. You can adjust the inferred data types before saving any new columns. Pipes can be used to filter the Data Source by different events.

Preconfigure a Data Source

Create a new file in your local workspace, named rudderstack_events.datasource, for example, to configure the empty Data Source.

Data Source schema
SCHEMA >
'value' String 'json:$'

ENGINE "MergeTree"
ENGINE_SORTING_KEY "value"

Deploy the changes using tb push.

This pre-configured Data Source is only required if you need a column containing the full JSON object as a String. Otherwise, skip this step and let Tinybird infer the columns and data types when you send the first event. You can then select which columns to save and adjust their data types. Create the Token as in method 1.

Create a Tinybird Token for the Data Source

Go to the Workspace in Tinybird where you want to receive data and select Tokens in the side panel. Create a new Token by selecting Create Token.

Give your Token a descriptive name. In the section DATA SOURCES SCOPES, select Add Data Source scope, select the name of the Data Source that you created, and mark the Append checkbox. Select Save changes.

Create a RudderStack destination

In RudderStack, Select Destinations in the side panel and then New destination.

Select Webhook:

  1. Give the destination a descriptive name.
  2. Connect your sources, you can test with the Rudderstack Sample HTTP Source.
  3. Input the following Connection Settings:
  • Webhook URL: <https://<your_host>/v0/events?name=rudderstack_events>
  • URL Method: POST
  • Headers Key: Authorization
  • Headers Value: Bearer TINYBIRD_AUTH_TOKEN

Select No transformation needed and save. Your destination has been created successfully.

Test ingestion

Select Sources > Rudderstack Sample HTTP > Live events > Send test event and paste the provided curl command into your terminal. The event appears on the screen and is sent to Tinybird.

The value column contains the full JSON object. You also have the option to parse the data into columns. When viewing the new columns, you can select which ones to save and adjust their data types.

Whenever new columns are detected in the stream of events, Tinybird asks if you want to save them.

Updated