Send LiteLLM events to Tinybird¶
LiteLLM is an LLM gateway that provides AI models access, fallbacks, and spend tracking across 100+ LLMs. It's a popular choice for many developers and organizations.
LiteLLM is open source and can be self-hosted.
To start sending LiteLLM events to Tinybird, first create a Data Source with this schema:
SCHEMA >
`model` LowCardinality(String) `json:$.model` DEFAULT 'unknown',
`messages` Array(Map(String, String)) `json:$.messages[:]` DEFAULT [],
`user` String `json:$.user` DEFAULT 'unknown',
`start_time` DateTime `json:$.start_time` DEFAULT now(),
`end_time` DateTime `json:$.end_time` DEFAULT now(),
`id` String `json:$.id` DEFAULT '',
`stream` Bool `json:$.stream` DEFAULT false,
`call_type` LowCardinality(String) `json:$.call_type` DEFAULT 'unknown',
`provider` LowCardinality(String) `json:$.provider` DEFAULT 'unknown',
`api_key` String `json:$.api_key` DEFAULT '',
`log_event_type` LowCardinality(String) `json:$.log_event_type` DEFAULT 'unknown',
`llm_api_duration_ms` Float32 `json:$.llm_api_duration_ms` DEFAULT 0,
`cache_hit` Bool `json:$.cache_hit` DEFAULT false,
`response_status` LowCardinality(String) `json:$.standard_logging_object_status` DEFAULT 'unknown',
`response_time` Float32 `json:$.standard_logging_object_response_time` DEFAULT 0,
`proxy_metadata` String `json:$.proxy_metadata` DEFAULT '',
`organization` String `json:$.proxy_metadata.organization` DEFAULT '',
`environment` String `json:$.proxy_metadata.environment` DEFAULT '',
`project` String `json:$.proxy_metadata.project` DEFAULT '',
`chat_id` String `json:$.proxy_metadata.chat_id` DEFAULT '',
`response` String `json:$.response` DEFAULT '',
`response_id` String `json:$.response.id`,
`response_object` String `json:$.response.object` DEFAULT 'unknown',
`response_choices` Array(String) `json:$.response.choices[:]` DEFAULT [],
`completion_tokens` UInt16 `json:$.response.usage.completion_tokens` DEFAULT 0,
`prompt_tokens` UInt16 `json:$.response.usage.prompt_tokens` DEFAULT 0,
`total_tokens` UInt16 `json:$.response.usage.total_tokens` DEFAULT 0,
`cost` Float32 `json:$.cost` DEFAULT 0,
`exception` String `json:$.exception` DEFAULT '',
`traceback` String `json:$.traceback` DEFAULT '',
`duration` Float32 `json:$.duration` DEFAULT 0
ENGINE MergeTree
ENGINE_SORTING_KEY start_time, organization, project, model
ENGINE_PARTITION_KEY toYYYYMM(start_time)
Install the Tinybird Python SDK with the ai extra, which adds LiteLLM. Quote the package name so shells like zsh don't expand the brackets:
pip install 'tinybird-python-sdk[ai]'
Finally, register the handler as a LiteLLM callback. Set datasource_name to the name of the Data Source you created, and api_url to your Workspace's API host:
import asyncio
import os
import litellm
from litellm import acompletion
from tb.litellm.handler import TinybirdLitellmAsyncHandler
litellm.callbacks = [
TinybirdLitellmAsyncHandler(
api_url="https://api.us-east.aws.tinybird.co",
tinybird_token=os.getenv("TINYBIRD_TOKEN"),
datasource_name="litellm",
)
]
async def main():
response = await acompletion(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hi"}],
metadata={"organization": "acme", "project": "my-app", "environment": "production"},
)
print(response.choices[0].message.content)
# The callback sends the event in the background. In a short script,
# give it a moment before exiting. Long-running apps don't need this.
await asyncio.sleep(1)
asyncio.run(main())
Each call, streamed or not, sends one event to the Events API. The metadata you pass to LiteLLM fills the proxy_metadata column and the organization, project, environment and chat_id columns, so you can break down usage and cost per customer.
AI analytics template¶
Use the LLM tracker template to bootstrap a multi-tenant, user-facing AI analytics dashboard and LLM cost calculator for your AI models. You can fork it and make it your own. The template ships its own llm_events Data Source. To send events to it, set datasource_name="llm_events".