In last week’s release, we introduced a new internal data format that significantly improves query performance, especially on cold engines.
This week’s release makes this new storage format the default for all new RAI Native App installations.
This release includes important performance improvements and enhancements to the Snowflake native app installation process and internal storage cleanup reliability.
Note that RelationalAI Native App upgrades are applied automatically and require no action on your part, unless you have disabled automatic upgrades.
One exception this week is a manual migration step required to take advantage of new performance improvements.
We’ve migrated the RAI Native App to a new internal data format that significantly improves query performance!
The new data format will allow us to dramatically reduce the compilation time, also called warmup time, that our logic engines require to process queries.
Benefits include:
Up to 50–80% reduction in query warmup time.
Faster and more responsive developer experience.
From our internal testing we have seen that on TPC-H scale factor 1, these changes reduced cold runtime from ~200s to ~80s.
Beginning with the next release, all new installations will use this new format.
We’re asking all existing users to manually migrate their app to this new format to take advantage of these performance improvements.
See the Migration Guide for step-by-step instructions.
The RAI Native App now uses Snowflake Manifest Version 2, which enables automatic privilege management.
All Snowflake privileges required by the app are granted automatically during installation, so manual grants are no longer needed.
Existing installations already have the necessary privileges, so no action is needed for current users.
This change mainly affects new installations.
Note that after this change, the privileges granted to the app cannot be revoked, even if you installed the app before this release.
See the Installation guide for more details.
A one-time full storage vacuum (BlobGC) pass will run automatically in all accounts during this upgrade to repair any previously corrupted storage state caused by a past bug that could prevent storage cleanup from running correctly.
This repair is applied automatically with no user action needed.
We recommend performing the migration during a maintenance window or when your app is not in active use.
During the migration:
The RAI Native App will be taken offline for a few minutes while the migration steps are performed.
All existing streams will be deleted and re-created.
You will incur standard Snowflake costs for setting up the streams.
All data loaded via streams will be wiped and re-ingested.
To migrate your app to the new data format:
First, delete any old or unused CDC streams that do not need to be migrated.
This will reduce the amount of data that needs to be migrated and speed up the process.
See Delete a Data Stream for instructions.
Next, execute the following SQL as an app admin (or ACCOUNTADMIN) to begin the migration:
This will block for a few minutes while the app is taken offline and the migration steps are prepared.
When it returns, the migration steps will be completed, and the CDC rebuilding process will have begun.
Use the following SQL to poll for CDC rebuilding status:
You can resume your ordinary workloads at any time during CDC rebuilding, and they will simply block until the data is loaded, per normal RAI data stream behavior.
Note that running the migration on an already migrated account will produce an error message and is otherwise a noop.
Enhanced error reporting for data streams.
Users will now see more detailed error messages for schema and data validation errors when creating a Type from a Snowflake table or view.
Additionally, all source tables are validated before reporting errors rather than stopping at the first error encountered, as was the case previously.
Messaging related to quarantined data streams has also been improved.
Users will now receive more detailed error messages and processing information when a data stream is quarantined due to repeated batch processing failures.
New engine sizes for greater scalability.
Depending on your Snowflake account’s cloud provider, you can now use:
HIGHMEM_X64_L (AWS only)
HIGHMEM_X64_SL (Azure only)
Use these sizes when you need more memory for complex workloads or larger datasets.
See the updated Engine Sizes and Compute Pools sections for details.
SQL API error messages no longer include data like stream sync details.
They now show only the command to retrieve it, helping prevent exposure of sensitive information.
When rounding a float value to zero from below, the result is now -0.0 instead of 0.0.
RAI Python models that call, for example, relationalai.std.math.ceil(-0.5) will see -0.0 now in the result.
This is consistent with the behavior of Snowflake SQL.
Added a new rai_userapplication role with more restricted permissions than the all_admin role that is currently required for Python users to run RAI models.
The rai_user role is the minimum role required for RAI Python API users and includes all of the permissions of the following application roles:
app_user: Can use the RAI Native App.
eng_user: Can use RAI engines.
eng_admin: Can create and delete RAI engines.
cdc_admin: Can create, delete, and manage data streams and enable and disable the CDC service.
Improved data streams with faster data synchronization performance are now created by default.
Data streams that take advantage of the improvements have version 1.0 in the api.data_streams view’sVERSION column.
Made other minor improvements to performance and stability.
The app.finalize_upgrade() procedure is deprecated.
Prior to this release, manual upgrades were required to call this procedure in order to complete the upgrade process.
Going forward, its functionality is now handled automatically by app.upgrade().
To preserve compatibility, finalize_upgrade() is now a no-op and returns a deprecation warning.
Fixed a bug that caused some upgrades to fail for users with a large number of user engines due to timeouts.
In some cases, the CDC engine would not be resumed and the CDC background task would remain suspended, which prevented updates to source tables and views in Snowflake from propagating to the RAI Native App.
This fix increases the timeout for engine upgrades to ensure that the upgrade process has enough time to complete.
Added a new status called CANNOT_READ_SOURCE to data streams that indicates the reference to a Snowflake source table has been removed, for example, by using the SYSTEM$REMOVE_REFERENCE Snowflake SQL function. You may see this status in the api.data_streams view or when calling the api.get_data_stream() procedure. If you encounter this status, the data stream must be deleted and then recreated.
Fixed a bug that exposed the internal compcache engine in the api.engines view. The compcache engine is reserved for internal use by the RAI Native App and should not be used by users.
Added a VERSION column to the api.data_streams and api.data_stream_batches views containing the current version number of the data stream. Versioning streams allows us to improve data stream functionality without breaking existing streams. For now, all new and existing streams have version 0.0 and no action is required on your part.
Improved data stream error reporting.
Change data capture (CDC) processing errors now appear in the api.data_stream_batches view, within the processing_details column’s processingErrors array (first 10 errors).
Each error includes a command to retrieve the full set using api.get_transaction_problems().
Improved error reporting during data loading. Several errors that were previously reported as an internal system error are now accurately reported with a less opaque error message.
Deprecated the api.update_libraries() procedure that is used to manually upgrade internal libraries after an automatic native app upgrade.
The procedure may still be called, but no longer performs any operation.
Python API users may encounter an error telling them to use update_libraries() the first time they query a model after an upgrade. Users may silence this error and update the internal libraries by setting the use_package_manager parameter to True when calling the Model constructor:
import relationalai as rai
# Set the use_package_manager parameter to True in order to silence the error about
# upgrading schema libraries using the update_libraries SQL procedure.
model = rai.Model("MyModel", use_package_manager=True)
In an upcoming Python API release, the error will be removed and the default value of use_package_manager will be set to True so that users do not need to set it manually.
Fixed a bug that caused the error 400 Bad Request: transaction already is in terminal state. This would sometimes occur when an engine transaction was cancelled by the user or the engine connection was closed by the debugger.
Fixed a bug that caused an Unknown error due to a client’s Snowflake session no longer existing. Now, a warning with the failure’s reason is logged and the connection is retried.
Improved error messages for queries and other transactions that are aborted, either by a user cancelling the transaction or from some other issue.
The error message now includes a SQL command that can be used to retrieve more details about why the transaction was aborted.
Made other minor improvements to stability and performance.
Removed support for RAI Schemas.
We don’t expect this to impact any users in practice as this feature was deprecated before RAI was generally available on Snowflake.
If you experience any issues please contact customer support at support@relational.ai.
Fixed a bug that caused data stream creation to fail when creating a stream on a table or view that is already used by another data stream.
Fixed a bug that caused a “Data stream does not exist” error in some cases when new Type objects were declared using the Python API.
Data streams are now created as expected for those types.
Cached query results are no longer stored indefinitely and will be removed from the RAI Native App after two weeks.
This ensures better data hygiene and improves storage efficiency.
Improved status and error details in the api.get_data_stream() SQL procedure.
Streams for which the underlying table or view is missing or does not have change tracking enabled will now have a NOT_INITIALIZED status and a more helpful message in the LAST_PROCESSED_ERROR_MESSAGE column.
Made other minor improvements to performance and stability.
Fixed a bug in an internal SQL procedure that sometimes caused automatic data stream creation to fail when queries from the RAI Python API were sent to the RAI Native App.
A new api.resume_engine_async() procedure has been added to resume an engine without waiting for it to be ready.
This procedure is useful when you want to resume an engine in the background and continue executing other tasks.
Minor improvements to internal error logs for improved support.
The api.resume_engine() procedure now blocks until the engine is ready to accept transactions.
Previously, the procedure returned immediately after the engine was resumed, which could result in errors if you attempted to use the engine before it was fully initialized.
Use the new api.resume_engine_async() procedure if you want to resume an engine without waiting for it to be ready.
Fixed a bug that caused shared object references to be deleted when a data stream was deleted.
This bug could cause unexpected behavior when multiple data streams were created from the same Snowflake table or view.
Enhanced performance when resuming the CDC Service to process source data changes queued during suspension.
As of version 2024.11.30-7b90c864, new CDC engines are automatically suspended after thirty minutes of inactivity.
In this release, CDC engines created prior to that version are automatically migrated to the new auto-suspension behavior.
The deprecated app.create_service() procedure has been removed. The work that this procedure was responsible for is performed by app.activate().
An older version of the api.create_engine() procedure that accepted a compute_pool parameter has been removed. Engine compute pools are fully managed by the RAI Native App.
To migrate to the new procedure, replace the compute_pool argument with the appropriate engine size.
For example:
Added a workaround to a Snowflake behavior change that prevented RAI Native Apps from automatically upgrading to the latest version if the app was deactivated at the time of the upgrade.
Error messages for api.get_data_stream() have been improved.
When background tasks necessary for processing stream updates are not running, you will now see an error message that includes instructions for resolving the issue.
A new value for the DATA_STREAM_SYNC_STATUS column returned by api.get_data_stream() has been added.
The value NOT_INITIALIZED indicates that Snowflake objects, such as tasks, that are required by the stream are missing or configured incorrectly.
To resolve this issue, delete the data stream and recreate it.
A new compute pool for the RAI Native App has been created named RELATIONAL_AI_COMPILE_CACHE_SPCS with instance family HIGHMEM_X64_M.
This compute pool is reserved for internal app purposes and has been added to avoid situations where all available nodes in engine compute pools are occupied by user engines.
There is no impact to cost associated with this change.
You may now optionally specify which underlying RAI Native App resources are dropped when you call the app.deactivate() procedure.
For example, the following deactivates the app and drops the app’s warehouse:
CALL relationalai.app.deactivate('warehouse')
Supported options are:
'all'
'warehouse'
'compute_pool'
'service'
The default value is NULL, which means no resources are dropped.
This is consistent with the behavior of app.deactivate() prior to this release.
The STATUS column in the api.engines view and the table returned by the api.get_engine() procedure now shows the status UPGRADING if an engine is currently being upgraded.
Query performance on engines that are resumed after having been suspended has been slightly improved.
The api.engines view no longer returns a row for an engine named compcache.
The compcache resource may not be used as an engine and is reserved for internal use by the RAI Native App.
When an engine compute pool is at capacity and you attempt to create an engine with an existing name, you will now see an engine already exists error instead of a compute pool at capacity error.
Fixed a bug that occasionally caused the app.activate() procedure to fail due to attempting to communicate with the RAI SPCS Service before it was available.
Fixed a bug that caused some users to see the following error:
Object 'RELATIONALAI.APP_STATE.RAI_DATABASE_SOURCES' does not exist or not authorized.
New engines now auto-suspend after one hour of inactivity.
Previously, auto-suspension was disabled by default for manually created engines and enabled by default for those created via the RAI Python API.
Existing engines keep their current settings.
Use alter_engine_auto_suspend_mins() to update an engine’s configuration.
See Engine Suspension for details.
The RAI Native App is now limited to 10,000 concurrently executed RAI models.
Previously, there was no limit.
This limit is not expected to occur in practice.
Improved query performance for RAI models with unchanged source data.
This reduces time spent on the Preparing your data... spinner in notebooks and REPLs.
The CDC Engine now auto-suspends after thirty minutes of activity and is automatically resumed when changes to a data stream’s source data are detected or a new data stream is created. As part of this change, the CDC engine is now suspended when the CDC service is disabled and resumed (or, if needed, created) when the CDC service is enabled. Previously, the CDC engine was deleted if you disabled the CDC service.
The timeout for provisioning an engine has been reduced from thirty minutes to twenty minutes.
Engine names now require a minimum of three characters. If you pass a name with fewer than three characters to the api.create_engine() procedure, you will get the following error: invalid engine name - valid characters are a-z, A-Z, 0-9, and _, name must be between 3 and 50 characters long.
Improved error messages in various resource management edge cases, including when the warehouse runs out of capacity and when warehouses are deleted mid-run.
Improved error messages when a transaction is aborted due to extremely large rule-sets.
In this release, older apps with CPU_X64_S compute pools will be automatically migrated to CPU_X64_XS. This reduces the number of Snowflake credits consumed by the application without affecting performance.
The deprecated relationalai.app.finalize_install() procedure has been removed. The work that this procedure was responsible for is performed by app.activate().
You can now configure engines to automatically suspend after a period of inactivity:
The api.create_engine() procedure has a new engine_config parameter that accepts a JSON object, like {'auto_suspend_mins': 60, 'await_storage_vacuum': false}, for configuring how many minutes an engine must be inactive before being suspended and whether or not garbage collection tasks must complete before suspending the engine. If engine_config is NULL, the engine is configured with auto_suspend_mins set to 0, and the engine will not be automatically suspended.
The default value for auto_suspend_mins will change from 0 to 60 minutes in a future release. To avoid unexpected behavior, we recommend creating engines with an explicit auto_suspend_mins value. Beginning with version 0.6.0 of the RAI Python API, engines auto-created by the API will be configured with auto_suspend_min set to value of the auto_suspend_minconfiguration key, or 60 minutes if the key is missing.
An api.alter_engine_auto_suspend_mins() procedure has been added to alter an existing engine’s auto_suspend_mins configuration parameter. Note that this procedure only works for engines created using the new api.create_engine() procedure mentioned above.
Use the new api.resume_engine() procedure to resume a suspended engine. Beginning with version 0.6.0 of the RAI Python API, suspended engines will be resumed automatically when Python queries are executed.
The api.engines view, as well as the table returned by api.get_engine(), now include an AUTO_SUSPEND_MINS and AWAIT_STORAGE_VACUUM columns that contain the configured values for each engine. The new SUSPENDS_AT column shows the timestamp of the next scheduled suspension for inactive engines.
