Configure data sync behavior

Use these settings to control how PyRel handles Snowflake-backed source data at query time. This guide explains when to change the defaults to balance fresher data, faster query startup, automatic change tracking, schema validation, and query timeouts.

Before you start, make sure that:

• You have access to a Snowflake account with the RelationalAI Native App installed. If you are unsure, contact your Snowflake administrator.
• You have a working PyRel installation. See Set Up Your Environment for instructions.

What data sync configuration covers

These settings apply during the Submit a job step in the PyRel workflow:

When your model reads Snowflake tables or views, the RAI Native App accesses that data through data streams maintained by the CDC service. These settings control query-time sync behavior, but they do not enable or repair CDC for you.

Note

Data sync depends on a healthy CDC service. Before you tune these settings, make sure CDC is enabled and running as expected. See Enable CDC and View CDC Service Status if you need to set that up or confirm that it is healthy.

The settings on this page control whether PyRel waits for synchronized data, how fresh that data must be, whether PyRel validates the synchronized schema, whether it tries to enable change tracking on sources for you, and how long Snowflake queries can run.

Use this table to find the setting to change when you need to adjust one of those behaviors:

Setting	Default	What it controls	Change it when
Stream synchronization	true	Whether PyRel waits for the data stream to catch up to the latest source data before it runs your query. This assumes the CDC service is already healthy and processing changes.	You prefer lower latency over waiting for the latest synchronized data.
Freshness tolerance	None	Lets PyRel evaluate a query when the source data is recent enough, even if synchronization is not fully caught up. This applies only when data.wait_for_stream_sync is true.	You want recent enough data without requiring full catch-up every time.
Column type checking	true	Whether PyRel validates that synchronized source columns still match the types your model expects.	You need to iterate faster on your model without waiting for full type validation.
Change tracking	false	Whether PyRel tries to enable change tracking on source tables or views that data stream synchronization depends on. This helps with source setup, but it does not enable or repair the CDC service.	You want PyRel to enable change tracking for eligible source tables or views automatically.
Query timeout	None	Lets PyRel set an explicit timeout for Snowflake queries.	You want a shorter configured timeout than Snowflake’s default timeout of 24 hours.

Each section shows how to configure the setting in raiconfig.yaml or programmatically in Python.

Related resources:

• Choose a configuration method
• Configure Snowflake authentication

Enable or disable stream synchronization before query execution

Set data.wait_for_stream_sync to control whether PyRel waits for streams to synchronize before running queries. It is enabled by default. Disable it when you prefer lower latency and can tolerate stale results.

Tip

If you keep stream synchronization enabled, you can also set a data freshness threshold to allow queries to proceed when the data is fresh enough.

raiconfig.yaml

1. Set data.wait_for_stream_sync in raiconfig.yaml:

   connections:
     # ...
   
   data:
     wait_for_stream_sync: false

2. Inspect the parsed config:

   from relationalai.semantics import Model
   
   m = Model("MyModel")
   
   print(m.config.data.wait_for_stream_sync)

   If this prints False, PyRel loaded the setting you configured. If it still prints True, check that you updated the config source this model is actually using.

Programmatic

Set data.wait_for_stream_sync using DataConfig or a plain Python dict. These examples assume you already have a valid connection configured through file discovery.

1. Set the value with a typed config class:

   from relationalai.config import DataConfig, create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data=DataConfig(wait_for_stream_sync=False))
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.wait_for_stream_sync)

   If this prints False, PyRel loaded the typed config object as expected.

2. Alternatively, set the value with a dict:

   from relationalai.config import create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data={"wait_for_stream_sync": False})
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.wait_for_stream_sync)

   If this prints False, PyRel loaded the dict-backed config object as expected.

Related resources:

• Set the data freshness threshold for stream synchronization
• DataConfig API reference

Set the data freshness threshold for stream synchronization

Set an optional freshness threshold when you want PyRel to accept data that is fresh enough without requiring full stream synchronization. It is unset by default. This threshold applies only when PyRel still waits for stream synchronization. If you set data.data_freshness_mins above 30240, PyRel clamps it to 30240 minutes (3 weeks) when it loads the config and emits a warning.

raiconfig.yaml

1. Set data.data_freshness_mins in raiconfig.yaml:

   connections:
     # ...
   
   data:
     data_freshness_mins: 5

2. Inspect the parsed config:

   from relationalai.semantics import Model
   
   m = Model("MyModel")
   
   print(m.config.data.data_freshness_mins)

   If this prints 5, PyRel loaded the freshness threshold you configured.

Programmatic

Set data.data_freshness_mins using DataConfig or a plain Python dict. These examples assume you already have a valid connection configured through file discovery.

1. Set the freshness threshold with a typed config class:

   from relationalai.config import DataConfig, create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data=DataConfig(data_freshness_mins=5))
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.data_freshness_mins)

   If this prints 5, PyRel loaded the typed config object as expected.

2. Alternatively, set the freshness threshold with a dict:

   from relationalai.config import create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data={"data_freshness_mins": 5})
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.data_freshness_mins)

   If this prints 5, PyRel loaded the dict-backed config object as expected.

Note that:

• data.data_freshness_mins works with data.wait_for_stream_sync, not instead of it.
• If you disable wait_for_stream_sync, PyRel does not wait long enough to enforce the freshness threshold before running queries.

Related resources:

• Enable or disable stream synchronization before query execution
• DataConfig API reference

Enable or disable column type checking during stream synchronization

Set data.check_column_types to control whether PyRel validates Snowflake source column types for synchronized data before it runs your query. It is enabled by default. Disable it only when you deliberately accept the risk of type mismatches reaching later stages.

Caution

Disabling column type checks can let mismatches pass into later stages of the workflow, where they are harder to debug.

raiconfig.yaml

1. Set data.check_column_types in raiconfig.yaml:

   connections:
     # ...
   
   data:
     check_column_types: false

2. Inspect the parsed config:

   from relationalai.semantics import Model
   
   m = Model("MyModel")
   
   print(m.config.data.check_column_types)

   If this prints False, PyRel loaded the relaxed schema-validation setting you configured.

Programmatic

Set data.check_column_types using DataConfig or a plain Python dict. These examples assume you already have a valid connection configured through file discovery.

1. Set the value with a typed config class:

   from relationalai.config import DataConfig, create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data=DataConfig(check_column_types=False))
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.check_column_types)

   If this prints False, PyRel loaded the typed config object as expected.

2. Alternatively, set the value with a dict:

   from relationalai.config import create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data={"check_column_types": False})
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.check_column_types)

   If this prints False, PyRel loaded the dict-backed config object as expected.

Related resources:

• Create configuration in code
• DataConfig API reference
• Configure results access

Enable or disable automatic change tracking

Change tracking must be enabled on Snowflake tables and views before PyRel can create and synchronize data streams. Set data.ensure_change_tracking to control whether PyRel tries to enable Snowflake change tracking automatically. This setting is disabled by default.

Enable this when you want PyRel to try to turn on Snowflake change tracking for eligible source tables or views automatically. Leave it off when your team manages change tracking outside PyRel, or when your Snowflake role does not have permission to alter those tables or views. For most users, the safe default is to leave this off unless missing change tracking is a recurring setup problem in your Snowflake environment.

Note

Only tables where you have OWNERSHIP privileges can be altered by this setting.

raiconfig.yaml

1. Set data.ensure_change_tracking in raiconfig.yaml:

   connections:
     # ...
   
   data:
     ensure_change_tracking: true

2. Inspect the parsed config:

   from relationalai.semantics import Model
   
   m = Model("MyModel")
   
   print(m.config.data.ensure_change_tracking)

   If this prints True, PyRel loaded ensure_change_tracking: true from the raiconfig.yaml file you expected.

Programmatic

Set data.ensure_change_tracking using DataConfig or a plain Python dict. These examples assume you already have a valid connection configured through file discovery.

1. Set the value with a typed config class:

   from relationalai.config import DataConfig, create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data=DataConfig(ensure_change_tracking=True))
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.ensure_change_tracking)

   If this prints True, the model is using the config object you passed into Model.

2. Alternatively, set the value with a dict:

   from relationalai.config import create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data={"ensure_change_tracking": True})
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.ensure_change_tracking)

   If this prints True, PyRel loaded the dict-backed config object as expected.

Related resources:

• Configure Snowflake authentication
• DataConfig API reference

Set the query timeout

Set data.query_timeout_mins when you want to set an explicit timeout for Snowflake queries from your model. If you do not set it, Model.config.data.query_timeout_mins stays None after PyRel loads your config.

Note

Leaving this field unset still uses the default 24-hour runtime timeout imposed by Snowflake. Set data.query_timeout_mins only when you want a shorter timeout than that default.

raiconfig.yaml

1. Set data.query_timeout_mins in raiconfig.yaml:

   connections:
     # ...
   
   data:
     query_timeout_mins: 10

2. Inspect the parsed config:

   from relationalai.semantics import Model
   
   m = Model("MyModel")
   
   print(m.config.data.query_timeout_mins)

   If this prints 10, PyRel loaded the timeout you configured. If it prints None, PyRel did not load that field from the config source you expected.

Programmatic

Set data.query_timeout_mins using DataConfig or a plain Python dict. These examples assume you already have a valid connection configured through file discovery.

1. Set the query timeout with a typed config class:

   from relationalai.config import DataConfig, create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data=DataConfig(query_timeout_mins=10))
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.query_timeout_mins)

   If this prints 10, PyRel loaded the typed config object as expected.

2. Alternatively, set the query timeout with a dict:

   from relationalai.config import create_config
   from relationalai.semantics import Model
   
   cfg = create_config(data={"query_timeout_mins": 10})
   m = Model("MyModel", config=cfg)
   
   print(m.config.data.query_timeout_mins)

   If this prints 10, PyRel loaded the dict-backed config object as expected.

Related resources:

• Create configuration in code
• DataConfig API reference