Common use cases¶

The following uses cases illustrate how Tinybird CLI solve common situations using available commands.

Download Pipes and data sources from your account¶

There are two ways you can start working with the CLI. You can either start a new data project from scratch, or if you already have some data and API Endpoints in your Tinybird account, pull it to your local disk to continue working from there.

For this second option, use the --match flag to filter Pipes or data sources containing the string passed as parameter.

For instance, to pull all the files named project:

tb pull --match project
[D] writing project.datasource(demo)
[D] writing project_geoindex.datasource(demo)
[D] writing project_geoindex_pipe.pipe(demo)
[D] writing project_agg.pipe(demo)
[D] writing project_agg_API_endpoint_request_log_pipe_3379.pipe(demo)
[D] writing project_exploration.pipe(demo)
[D] writing project_moving_avg.pipe(demo)

The pull command doesn't preserve the directory structure, so all your datafiles are downloaded to your current directory.

Once the files are pulled, you can diff or push the changes to your source control repository and continue working from the command line.

Push the entire data project¶

tb push --push-deps

Push a Pipe with all its dependencies¶

tb push pipes/mypipe.pipe --push-deps

Adding a new column to a Data Source¶

Data Source schemas are mostly immutable, but you have the possibility to append new columns at the end of an existing Data Source with an Engine from the MergeTree Family or Null Engine.

If you want to change columns, add columns in other positions, or modify the engine, you must first create a new version of the Data Source with the modified schema. Then ingest the data and finally point the Pipes to this new Endpoint. To force a Pipe replacement use the --force flag when pushing it.

Append new columns to an existing Data Source¶

As an example, imagine you have the following Data Source defined, and it has been already pushed to Tinybird:

SCHEMA >
    `test` Int16,
    `local_date` Date,
    `test3` Int64

If you want to append a new column, you must change the *.datasource file to add the new column new_column. You can append as many columns as you need at the same time:

SCHEMA >
    `test` Int16,
    `local_date` Date,
    `test3` Int64,
    `new_column` Int64

Here are 2 methods for appending a column:

1. Add the new column to the SELECT statement of the materializing pipe:

   • Use the command: tb push pipes/<pipe_name>.pipe --push-deps --force --override-datasource
   • This automatically updates both the Pipe and the Data Source schema.

2. Alternatively, you can add the column to the schema of the Materialized View and pipe manually:

   • Use the command: tb push datasources/<datasource_name>.datasource --force
   • Note: This doesn't automatically add the column to the SELECT statement of the materializing Pipe.
   • Add the new column to the SELECT statement of the materializing pipe
   • Push the changes with: tb push pipes/<pipe_name>.pipe --force

Important notes:

• The new column isn't backfilled with data for pre-existing rows. They contain 0 for numeric values, '' for strings, or default values from the schema, if defined.
• Append new data to the landing data source to populate the new column.
• You can only add columns that are not part of the Sorting Key in the target Materialized View.
• Run tb dependencies to ensure you understand all connected resources.

Create a new version of the Data Source to make additional add/change column operations¶

To create a new version of a Data Source, create a separate datafile with a different name. You can choose a helpful naming convention such as adding a _version suffix, for example my_ds_1.datasource.

Debug mode¶

When you work with Pipes that use several versions of different data sources, you might need to double check which version of which Data Source the Pipe is pointing at before you push it to your Tinybird account.

To do so, use the --dry-run --debug flags like this:

tb push my_pipe.pipe --dry-run --debug

After you've validated the content of the Pipe, push your Pipe as normal.

Automatic regression tests for your Endpoints¶

Any time you --force push a Pipe which has a public Endpoint that has received requests, some automatic regression tests are executed.

If the previous version of the Endpoint returns the same data as the version you are pushing, the CLI checks for the top ten requests. This can help you validate whether you are introducing a regression in your API.

Other times, you are consciously --force pushing a new version which returns different data. In that case you can avoid the regression tests with the --no-check flag:

tb push my_may_view_pipe.pipe --force --no-check