Skip to main content

freshness

dbt_project.yml
sources:
  <resource-path>:
    +freshness:
      warn_after:  
        count: <positive_integer>
        period: minute | hour | day

Definition​

A freshness block is used to define the acceptable amount of time between the most recent record, and now, for a table to be considered "fresh".

In the freshness block, one or both of warn_after and error_after can be provided. If neither is provided, then dbt will not calculate freshness for the tables in this source.

  • warn_after: Duration (for example, 12 hours) after which dbt raises a warning if the most recent available data is older than this threshold.
  • error_after: Duration (for example, 24 hours) after which dbt fails the freshness check if the most recent available data is older than this threshold.

In most cases, the loaded_at_field is required. Some adapters support calculating source freshness from the warehouse metadata tables and can exclude the loaded_at_field. (Applies to dbt v1.10 and later) Alternatively, you can define loaded_at_query to use custom SQL expression to calculate the timestamp.

If a source has a freshness: block, dbt will attempt to calculate freshness for that source:

  • If loaded_at_field is provided, dbt will calculate freshness via a select query.
  • If loaded_at_field is not provided, dbt will calculate freshness via warehouse metadata tables when possible.
(Applies to dbt v1.10 and later)
  • If loaded_at_query is provided, dbt will calculate freshness via the provided custom SQL query.
  • If loaded_at_query is provided, loaded_at_field should not be configured.

Currently, calculating freshness from warehouse metadata tables is supported on the following adapters:

(Applies to dbt v1.12 and later)
Wildcard table identifiers

On BigQuery, metadata-based freshness checks are not reliable for sources defined with wildcard table identifiers (for example, events_*).

To prevent incorrect freshness results, enable the bigquery_reject_wildcard_metadata_source_freshness flag in your dbt_project.yml. When enabled, dbt raises an error if metadata-based freshness is used with a wildcard table identifier.

To calculate freshness for wildcard tables, configure loaded_at_field to use query-based freshness checks instead.

Freshness blocks are applied hierarchically:

  • A freshness and loaded_at_field property added to a source will be applied to all tables defined in that source.
  • A freshness and loaded_at_field property added to a source table will override any properties applied to the source.

This is useful when all of the tables in a source have the same loaded_at_field, as is often the case.

To exclude a source from freshness calculations, explicitly set freshness: null.

State-aware orchestration is now dbt State

dbt State works with all engines and environments: dbt Core, dbt platform, and Fusion

If you're using state-aware orchestration prior to June 1, 2026, you can continue using it. Your dbt State trial will be extended until the billing period begins on September 1, 2026. If your trial wasn't extended, contact your account team. To get started, refer to Migrate from state-aware orchestration.

In state-aware orchestration, dbt uses the warehouse metadata by default to check if sources (or upstream models in the case of Mesh) are fresh. For more information on how freshness is used by state-aware orchestration, see Advanced configurations.

If you're using dbt State, loaded_at_field and loaded_at_query are also used for source freshness detection (for example, to ensure late-arriving records are detected). Refer to Migrate from state-aware orchestration for more details.

loaded_at_field​

Optional on adapters that support pulling freshness from warehouse metadata tables, required otherwise.

A column name (or expression) that returns a timestamp indicating freshness.

Examples:

sources:
  - name: inventory_updates
    config:
      freshness:
        error_after:
          count: 24
          period: hour
      loaded_at_field: updated_at

If using a date field, you may have to cast it to a timestamp:

sources:
  - name: work_orders
    description: |
      Work orders from ERP. The completed_date column is stored as DATE but we need to compare it as a timestamp for freshness checks.
    config:
      freshness:
        error_after:
          count: 24
          period: hour
      loaded_at_field: "completed_date::timestamp"

Or, depending on your SQL variant:

sources:
  - name: purchase_orders
    description: |
      Purchase orders. The completed_date is stored as VARCHAR in 'YYYY-MM-DD' format. Use CAST for explicit conversion.
    config:
      freshness:
        error_after:
          count: 24
          period: hour
      loaded_at_field: "CAST(completed_date AS TIMESTAMP)"

If using a non-UTC timestamp, cast it to UTC first:

sources:
  - name: customer_transactions
    description: |
      Customer transactions recorded in Sydney local time. Converting to UTC for consistent freshness comparison across sources in different timezones.
    config:
      freshness:
        error_after:
          count: 24
          period: hour
      loaded_at_field: "convert_timezone('Australia/Sydney', 'UTC', created_at_local)"
(Applies to dbt v1.10 and later)

loaded_at_query​

Specify custom SQL to generate the maxLoadedAt timestamp on the source (rather than via warehouse metadata or the loaded_at_field config). Note that loaded_at_query should not be used if loaded_at_field is defined.

Examples:


sources:
  - name: your_source
    config:
      freshness: # changed to config in v1.9
        error_after:
          count: 2
          period: hour
      loaded_at_query: |
        select max(_sdc_batched_at) from (
        select * from {{ this }}
        where _sdc_batched_at > dateadd(day, -7, current_date)
        qualify count(*) over (partition by _sdc_batched_at::date) > 2000
        )


sources: 
  - name: ecom
    schema: raw
    description: E-commerce data for the Jaffle Shop
    config: 
      freshness: 
        warn_after:
          count: 24
          period: hour
    tables:
      - name: raw_orders
        description: One record per order
        config:
          loaded_at_query: "select {{ current_timestamp() }}"
...

Should not be configured if loaded_at_field is also configured, but if it is, dbt will use which ever value is closest to the table.

Filter won't work for loaded_at_query.

count​

(Required)

A positive integer for the number of periods where a data source is still considered "fresh".

period​

(Required)

The time period used in the freshness calculation. One of minute, hour or day

filter​

(optional)

Add a where clause to the query run by dbt source freshness in order to limit data scanned.

This filter only applies to dbt's source freshness queries - it will not impact other uses of the source table.

This is particularly useful if:

  • You are using BigQuery and your source tables are partitioned tables
  • You are using Snowflake, Databricks, or Spark with large tables, and this results in a performance benefit

Examples​

Complete example​

models/<filename>.yml


sources:
  - name: jaffle_shop
    database: raw
    config: 
      # changed to config in v1.9
      freshness: # default freshness
        warn_after: {count: 12, period: hour}
        error_after: {count: 24, period: hour}

      loaded_at_field: _etl_loaded_at

    tables:
      - name: customers # this will use the freshness defined above

      - name: orders
        config:
          freshness: # make this a little more strict
            warn_after: {count: 6, period: hour}
            error_after: {count: 12, period: hour}
            # Apply a where clause in the freshness query
            filter: datediff('day', _etl_loaded_at, current_timestamp) < 2


      - name: product_skus
        config:
          freshness: # do not check freshness for this table

When running dbt source freshness, the following query will be run:

select
  max(_etl_loaded_at) as max_loaded_at,
  convert_timezone('UTC', current_timestamp()) as snapshotted_at
from raw.jaffle_shop.orders

where datediff('day', _etl_loaded_at, current_timestamp) < 2

Was this page helpful?

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

0
Loading