Monitor your query validation status

Tinybird manages ClickHouse® upgrades for you. Tinybird selects stable versions based on internal testing and upgrades only when testing indicates no impact to your queries. As part of this process, Tinybird tests all queries you've run in the past 7 days against the upcoming ClickHouse® version.

Tinybird works mainly with three versions at the same time:

  • Edge. This version has the latest features, and Tinybird uses it as a testbench.
  • Stable. Most clusters are on this version, which Tinybird keeps as the main reference. Tinybird promotes a version from Edge to Stable only after testing it for regressions on clusters with varying workloads.
  • Deprecated. When Tinybird promotes a version to Stable, the previous versions become deprecated. Tinybird keeps as few clusters as possible on deprecated versions, which typically support specific use cases.

To ensure all clusters run fully supported versions, Tinybird upgrades them regularly. Some queries might break in newer versions. The validator tests your recent queries against the next version, typically Edge, to identify breakages before an upgrade.

When Tinybird finds queries that would break, it adds them to the Service Data Source query_validator_log. This Data Source provides information about the query, such as:

  • The type and name of the originating resource (Pipe, Copy Pipe, Materialized View, or API query).
  • The full query text.
  • The error message produced in the new ClickHouse® version.
  • The new ClickHouse® version used for testing.

Use this information to fix the issues and prevent your queries from breaking when Tinybird upgrades your cluster.

Monitoring for broken queries

You can query the query_validator_log like this:

SELECT
    stable_version,
    version,
    query_last_execution,
    resource_type,
    resource_name,
    query,
    error_code,
    error,
    fix_suggestion
FROM
    (
        SELECT *
        FROM query_validator_log
        WHERE error != '' AND query_id LIKE 'MV_%'
        ORDER BY run_validation DESC
        LIMIT 1 BY query_id
        UNION ALL
        SELECT *
        FROM query_validator_log
        WHERE error != '' AND query_id NOT LIKE 'MV_%'
        ORDER BY run_validation DESC
        LIMIT 1 BY resource_type, resource_name
    )

This query returns the queries that need fixing, along with a possible fix suggestion in fix_suggestion. For Materialized Views, resource_name identifies the destination table when Tinybird can resolve it from the query; otherwise, identify the Materialized View from the query field.

The queries returned in this Data Source correspond to the full queries Tinybird sends to ClickHouse®. These queries comprise all the nodes you create in your Pipes.

Fixing queries

Some queries might be easy to fix with the provided fix_suggestion or the error field. To check that your fix worked, look up your Pipe in the Data Source the next day and confirm that the error_code and error fields are empty. Tinybird plans to provide better ways to test queries. Contact support if you need help fixing your queries.

Updated