Live Query Configuration and Compatibility
This page determines which cloud data platform (CDP) can run a Live Query workflow in your Alteryx One workspace and what happens when a workflow uses data from another CDP.
A connection can exist in Alteryx One even when the workspace configuration doesn't allow it to act as the Live Query execution CDP. Live Query compatibility determines whether you can select the connection as the execution connection when you create a workflow.
Supported Execution CDPs
The workspace's base storage determines which CDPs can act as the execution CDP. Databricks also requires a Databricks workspace setup in Alteryx One.
Base Storage | BigQuery | Snowflake | Databricks |
|---|---|---|---|
GCS | Supported | Unsupported | Unsupported |
ADS | Unsupported | Supported | Supported |
Amazon S3 | Unsupported | Supported | Supported |
ADLS | Unsupported | Supported | Supported |
Note
Databricks Live Query requires Databricks workspace setup in Alteryx One. Without this configuration, you can't select Databricks as the execution CDP, even when the workspace uses ADS, Amazon S3, or ADLS base storage.
If a connection isn't compatible with the workspace configuration, you can't select it as the execution connection when you create a Live Query workflow.
Connection Creation and Live Query Compatibility
BigQuery, Snowflake, and Databricks connections are platform-level connections. Live Query compatibility doesn't determine whether these connections can exist for other uses in Alteryx One.
For Live Query, compatibility rules determine whether you can select a connection as the workflow's execution CDP.
Work with Data from Multiple CDPs
A Live Query workflow has one execution CDP, determined by the connection you select when you create the workflow.
You can add a dataset from another CDP to the workflow. Live Query reads the dataset from its source CDP, uploads it to the execution connection, and materializes it as a temporary object in the catalog or schema selected for the workflow.
All workflow computation then runs in the execution CDP. Live Query doesn't push workflow computation back to the imported dataset's source CDP.
The following CDP pairs can participate in the same workflow when one acts as the execution CDP and the other provides an imported dataset:
BigQuery and Snowflake
Snowflake and Databricks
BigQuery and Databricks
The workspace's base storage still determines which CDP can act as the execution CDP.
Within a workspace, separate workflows can use any execution CDP allowed by the workspace configuration:
With ADS, Amazon S3, or ADLS base storage and a Databricks workspace setup, separate workflows can use Snowflake or Databricks as the execution CDP.
A workspace configured for BigQuery execution uses GCS base storage. Snowflake and Databricks can't act as execution CDPs in that workspace.
Imported datasets follow the same expiration and cleanup behavior as other temporary objects.
Data Residency, Performance, and Cost
Before you use data from multiple CDPs, consider these items:
Data residency: Imported data physically leaves the source CDP and is stored temporarily in the execution CDP. The execution CDP can be in a different cloud or region.
Performance: Live Query must extract and load the imported dataset before it can produce downstream design-time results. Larger datasets take longer to extract and load.
Cost: Reading the dataset can consume compute resources in the source CDP and incur data-transfer or egress costs. Loading, querying, and storing the materialized copy can consume compute and storage resources in the execution CDP. These costs can recur when imported data is refreshed.
Output Destinations
Live Query writes output to the execution CDP. Where supported, you can also write output to a file in the workspace's base storage. A workflow can't write output to multiple CDPs.