🔥 Fire Up Your Logging Needs with Pydantic Logfire

8 min readMay 16, 2024

As developers, we’re all too familiar with the importance of logging in our applications. Good logging practices help us debug issues, monitor performance, and maintain system security. However, finding a suitable logging library that meets our needs can be a challenge.

In today’s article, we’ll be exploring logfire — a innovative logging tool brought to us by the developers at Pydantic. With its unique features and capabilities, logfire provides a flexible and powerful way to manage and analyze log data in your Python applications. Let’s discover what makes logfire so useful!

Getting Started with Pydantic Logfire

The first step to using Pydantic logfire is to install the package using a Python package manager such as pip or poetry.

poetry add logfire

pip install logfire

Now that we’ve installed the logfire package, let’s move on to importing it into our Python code and writing a simple example to get started:


import logfire

logfire.info('Hello, {name}!', name='Destiny')
logfire.error("This is an example error!")
logfire.exception("This is an example exception!")
logfire.debug("This is an example debug context")
logfire.warn("This is an example warn!")

You can configure logging levels (DEBUG, INFO, WARNING, ERROR, EXCEPTION), formats, and handlers to suit your application’s specific needs.

When we run our project, we will get error about authentication configuration for logfire has not been set up. To resolve this, you can either enter the command below in your interactive shell or follow this link to log in. Once logged in, you will be able to access the logfire Web UI and view your logs.

After authentication operations, you can create a new project from the logfire web ui. When you create a new project, a unique Project Token will be generated, which can only be accessed once. You can configure logfire with this token:

logfire.configure(token=LOGFIRE_PROJECT_TOKEN)

It’s important to have separate LogFire configurations for each project. It’s can help you to keep development, staging, and production logs organized without mixing them together.
I don’t recommend sending all log data to a single project.

Now we can see our example logs in the logfire web ui.

Working With Span

Unlike traditional logging systems, a logfire span has its own context and helps you track related logs together, giving you a more detailed understanding of what’s happening in your system. It also includes start and end dates, making it possible to measure how long code takes to run.

With Logfire span, your log calls can include information about which code snippets were executed beforehand, providing valuable insights into your application’s behavior.

Let’s say you’re building an API with FastAPI, and you have a bug in one of your database queries. When the query fails, you’d like to know what arguments were passed in the request that triggered the error. With logfire span, you can achieve this!

with logfire.span('Division of 2 numbers'):  
    first_number = int(input('Enter your first number:'))
    second_number = int(input('Enter your second number:'))
    division_result = first_number / second_number
    logfire.info(
        '{first_number=} / {second_number=} = {result=}',
        first_number=first_number,
        second_number=second_number,
        result=division_result
    )

After clicking the “+” button, you can view the values entered by the user and see the results of their input.

The other thing about spans is they automatically record any unhandled exceptions that happen within them.

If you want to record handled errors you need to call `logfire.exception():

with logfire.span('Division of 2 numbers'):  
    first_number = int(input('Enter your first number:'))
    second_number = int(input('Enter your second number:'))
    division_result = 0

    try:
        division_result = first_number / second_number
    except ZeroDivisionError:
        logfire.exception("The second number is zero", first_number=first_number, second_number=second_number)
        
    logfire.info(
        '{first_number=} / {second_number=} = {result=}',
        first_number=first_number, second_number=second_number, result=division_result
    )

Working with Pydantic

logfire.configure(
    pydantic_plugin=logfire.PydanticPlugin(record='all'),
    token=LOGFIRE_PROJECT_TOKEN
)

Logfire allows you to integrate Pydantic models and track their validation logs in your web ui. To enable this feature, you need to configure the Pydantic plugin in your logfire settings.

The Pydantic plugin has four configuration options:

off: This is the default value. Disable instrumentation, which means any Pydantic validation values will not be logged.
all: Send traces and metrics for all events.
failure: Send metrics for all validations and traces only for validation failures.
metrics: Send only metrics.

class User(BaseModel):
    name: str
    country_code: str
    dob: date

User(name='Anne', country_code='USA', dob='2000-01-01')  
User(name='Ben', country_code='USA', dob='2000-02-02')
User(name='Charlie', country_code='GBR', dob='1990-03-03')

Working with FastAPI & Pydantic

To begin, we need to install logfire using FastAPI’s extra:

poetry add logfire[fastapi]

Let’s prepare a simple example. It will ask an llama-3 model a question based on a user-provided query and return the answer. We’ll integrate logfire with FastAPI and llama-3 in just a few lines:

class Message(BaseModel):
    query: str


class ResponseData(BaseModel):
    model: str
    created_at: str
    content: str


app = FastAPI()
client = AsyncClient()

logfire.instrument_fastapi(app)  
logfire.instrument(client)


@app.post("/search", response_model=ResponseData)
async def search(message: Message) -> ResponseData:
    response = await client.chat(
        model="llama3",
        messages=[
            {"role": "user", "content": message.query},
        ],
    )
    result = ResponseData(
                model=response["model"],
                created_at=response["created_at"],
                content=response["message"]["content"]
             )
    return result

Now we can send a simple payload to the /search endpoint, after that we can watch all information on the web ui.

curl -X 'POST' \                                                                                               ─╯
         'http://localhost:8000/search' \
         -H 'accept: application/json' \
         -H 'Content-Type: application/json' \
         -d '{
     "query": "Why is the sky blue?"
   }'

The call also provides complete visibility into the arguments passed to the endpoint. This helps users reproduce errors that occur in production environments on their local machines. To achieve this, we need to set the log level to Debug. Go to the Default Levels section and select “Debug”.

Metrics

Logfire allows collect important app metrics, such as exception counts, connection numbers, and memory usage. Metrics allow you to track and analyze numerical values, providing a summary of the data over time or in other contexts, rather than looking at each individual value.

Logfire features multiple metric types for tracking and analyzing your app’s performance:

Counter: It is perfect for measuring how often something happens in your app, such as the frequency of exceptions, requests, or processed items.
Histogram: It is ideal for understanding how values are distributed in your app, such as the time it takes to process requests, file sizes, or the length of item lists.
Up-Down Counter: It allows you to both increase and decrease the count based on specific events or states, unlike regular counters that only go up. You can use this metric to track changes in quantities like active connections or items in a queue.
Gauge: It is perfect for tracking the current value of a state or event in your app such as memory usage, unlike counters which accumulate values over time.
Callback Metrics (Counter Callback, Gauge Callback, Up-Down Counter Callback): Callback metrics, or observable metrics, are a way to create metrics that are automatically updated based on a time interval.

System Metrics: By default, logfire does not collect system metrics. To start collecting them, simply install the logfire[system-metrics] extra.
poetry add logfire[system-metrics]

Here is an example code snippet for an Up-Down Counter metric:

import logfire

active_users = logfire.metric_up_down_counter(
    'active_users',
    unit='1',  
    description='Number of active users'
)

def user_logged_in():
    active_users.add(1)

def user_logged_out():
    active_users.add(-1)

user_logged_in()
user_logged_in()
user_logged_in()
user_logged_in()
user_logged_out()

After running a SQL query, you can view your metrics in the Explore tab or create a custom chart to visualize them in the Dashboard tab.

Web UI: Live & Dashboards & Explore

Logfire dashboard provides many helpful features to help you manage your logs. You can:

Filter your logs by various criteria
Run SQL queries directly on your logs
Create custom charts to visualize your log data and monitor system performance
Receive custom alert notifications via Webhook, allowing you to be notified of specific log events or conditions

Filter your logs by span name, allowing you to inspect specific spans and their related logs:

Filter your logs by specific endpoint, allowing you to inspect logs related to a particular API call or user interaction:

You can run your SQL queries directly in the Explore tab within your data. In this tab you can see previous SQL query result.

In the Dashboard tab, you can create different charts such as donut chart to monitor and visualize your data. Logfire comes with a pre-built template that you can customize as needed.

This dashboard provides an overview of your web services’ performance. It displays key metrics, including:

Requests: The total number of requests received.
Exceptions: The number of exceptions encountered during processing.
Trend Routes: A visual representation of frequently accessed routes or APIs over time.
2XX Response Rate: The percentage of successful responses (200 status codes).
Log Type Ratio: A breakdown of different log types generated, such as info, warning, and error.

To create a custom dashboard, start with a blank slate by clicking ‘Start From Scratch’ and add your desired charts and blocks.

“I would like to thank the Pydantic team for introducing logfire, which has been helpful. I’d also like to appreciate Onuralp Sezer for helping me with this piece. Additionally, I’d like to note that logfire is currently in beta and free to use.

Resources

[1] https://docs.pydantic.dev/logfire/
[2] https://docs.pydantic.dev/logfire/guides/web_ui/
[3] https://github.com/jxnl/instructor/tree/main/examples/logfire-fastapi
[4] https://github.com/pydantic/logfire-demo/tree/main