Multiple CDPs in Live Query
A Live Query workflow has a primary CDP, which provides the main execution environment for the workflow.
Primary CDP
The primary CDP is determined by the connection technology selected when you create the Live Query workflow.
The primary CDP hosts temporary objects and the primary workflow processing environment.
The workspace's configured base storage can constrain which CDP can be used as the primary CDP. Base-storage configuration is controlled by the workspace configuration rather than selected by the workflow user.
For supported base-storage and primary-CDP combinations, go to Get Started with Live Query.
When Data Movement Occurs
Cross-platform data movement occurs when a workflow references data that isn't located in the primary CDP.
Examples include:
Data from a secondary CDP.
An uploaded file.
Data from a non-CDP connection.
The data follows this general path:
Source → Extract → Base storage → Temporary table in primary CDP → Processing in primary CDP
What Data Is Copied
The data copied depends on the input:
For a file, the complete file is copied.
For a database table, the dataset table is copied.
For a custom SQL dataset, the result produced by the custom SQL is copied.
Temporary Objects
Temporary objects are created in the primary CDP's configured Live Query schema.
Staged files can additionally exist in configured base storage.
These temporary objects use the same expiration and scheduled cleanup mechanisms used for other Live Query temporary objects.
Performance Considerations
Moving data into the primary CDP introduces a materialization step.
The process requires extracting the data, writing it to staging storage, and loading it into the primary CDP before workflow processing can continue.
As a result, this operation can introduce:
Additional latency.
Additional storage usage.
Additional warehouse load cost.
Supported CDP Combinations
Not every combination of primary CDP, secondary CDP, base storage, and workspace configuration is supported.
Support depends on the workspace base-storage configuration and the CDPs used by the workflow.
Primary CDP | BigQuery Secondary | Snowflake Secondary | Databricks Secondary |
|---|---|---|---|
BigQuery | Yes | No | No |
Snowflake | No | Yes | Yes |
Databricks | No | Yes | Yes |