Configure for Azure Databricks

Beginning in Release 7.1, the integration with Azure Databricks switched from using a Hive-based driver to a Simba driver for the integration with Spark.

No installation is required to use this new driver.

For more information on the Simba driver, see https://databricks.com/spark/odbc-driver-download.

Supported versions:

For more information on supported versions, seehttps://docs.databricks.com/release-notes/runtime/releases.html#supported-databricks-runtime-releases-and-support-schedule.

By default, the number of jobs permitted on an Azure Databricks cluster is set to 1000.

Managing limits:

To enable retrieval and auditing of job information after a job has been completed, theDesigner Cloud Powered by Trifacta platformdoes not delete jobs from the cluster. As a result, jobs can accumulate over time to exceeded the number of jobs permitted on the cluster. If you reach these limits, you may receive aQuota for number of jobs has been reachedlimit. For more information, see https://docs.databricks.com/user-guide/jobs.html.

Optionally, you can allow the Designer Cloud Powered by Trifacta platform to manage your jobs to avoid these limitations. For more information, see "Configure Databricks job management" below.

When a user submits a job, the Designer Cloud Powered by Trifacta Enterprise Edition provides all the cluster specifications in the Databricks API and it creates cluster only for per-user or per-job. that means once the job is complete, the cluster is terminated. Cluster creation may take less than 30 seconds if instance pools are used. If the instance pools are not used, it may take 10-15 minutes.

For more information on job clusters, see https://docs.databricks.com/clusters/configure.html.

The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to Azure Databricks access.

Optionally, you can configure the Designer Cloud Powered by Trifacta platform to use the Databricks cluster policies that have been specified by your Databricks administrator for creating and using clusters. These policies are effectively templates for creation and use of Databricks clusters and govern aspects of clusters such as the type and count of nodes the resources that can be accessed via the cluster, and other settings. For more information on Databricks cluster policies, see https://docs.databricks.com/administration-guide/clusters/policies.html.

Prerequisites

Notes:

Steps:

Job overrides:

A user's cluster policy can overridden when a job is executed via API. Set the request attribute for the clusterPolicyId.

For more information, see API Task - Run Job.

Policy template for Azure - without instance pools:

The following example cluster policy can provide a basis for creating your own Azure cluster policies when instance pools are not in use:

{
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
  "driver_node_type_id": {
    "type": "fixed",
    "value": "Standard_D3_v2",
    "hidden": true
  },
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  },
  "node_type_id": {
    "type": "fixed",
    "value": "Standard_D3_v2",
    "hidden": true
  }
}

Policy template for Azure - with instance pools:

The following example cluster policy can provide a basis for creating your own Azure cluster policies when instance pools are in use:

{
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  },
  "instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "driver_instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
}

Please review and modify the following configuration settings.

Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The Designer Cloud Powered by Trifacta platform can be configured to leverage instance pooling on the Azure Databricks cluster for both worker and driver nodes.

For more information, see https://docs.azuredatabricks.net/clusters/instance-pools/index.html.

Azure Databricks enforces a hard limit of 1000 created jobs per workspace, and by default cluster jobs are not deleted. To support jobs more than 1000 jobs per cluster, you can enable job management for Azure Databricks.

Steps:

By default, Databricks workspaces apply limits on the number of jobs that can be submitted before the cluster begins to fail jobs. These limits are the following:

Depending on how your clusters are configured, these limits can vary. For example, if the maximum number of concurrent jobs per cluster is set to 20, then the 21st concurrent job submitted to the cluster fails.

To prevent unnecessary job failure, the Designer Cloud Powered by Trifacta platform submits the throttling of jobs to Databricks. When job throttling is enabled and the 21 concurrent job is submitted, the Designer Cloud Powered by Trifacta platform holds that job internally the first of any of the following events happens:

Steps:

Please complete the following steps to enable job throttling.

Optionally, you can leverage Databricks Secrets Management to store sensitive Databricks configuration properties. When this feature is enabled and a set of properties are specified, those properties and their values are stored in masked form. For more information on Databricks Secrets Management, see https://docs.databricks.com/security/secrets/index.html.

Steps:

Each user must insert a Databricks Personal Access Token to access Databricks resources. For more information, see Databricks Settings Page.

When profiling is enabled for a Spark job, the transform and profiling tasks are combined by default. As needed, you can separate these two tasks. Publishing behaviors vary depending on the approach. For more information, see Configure for Spark.

To enable SSO authentication with Azure Databricks, you enable SSO integration with Azure AD. For more information, see Configure SSO for Azure AD.

For enhanced security, you can configure the Designer Cloud Powered by Trifacta platform to use an Azure Managed Identity. When this feature is enabled, the platform queries the Key Vault for the secret holding the applicationId and secret to the service principal that provides access to the Azure services.

To enable, the following parameters for the Designer Cloud Powered by Trifacta platform must be specified.

You can apply this change through the Admin Settings Page (recommended) or trifacta-conf.json. For more information, see Platform Configuration Methods.

Save your changes.

Optionally, you can provide to the Designer Cloud Powered by Trifacta platform the name of a shared Databricks cluster to be used to access Databricks Tables.

Prerequisites:

"dfs.adls.oauth2.client.id"
"dfs.adls.oauth2.credential"
"dfs.adls.oauth2.refresh.url"
"dfs.adls.oauth2.access.token.provider.type"
"dfs.adls.oauth2.access.token.provider"
"fs.azure.account.oauth2.client.endpoint"
"spark.hadoop.dfs.adls.oauth2.client.id"
"spark.hadoop.dfs.adls.oauth2.credential"
"spark.hadoop.dfs.adls.oauth2.refresh.url"
"spark.hadoop.fs.azure.account.oauth2.client.endpoint"
"spark.hadoop.dfs.adls.oauth2.access.token.provider.type"
"spark.hadoop.dfs.adls.oauth2.access.token.provider"

"fs.azure.account.auth.type"
"fs.azure.account.oauth.provider.type"
"fs.azure.account.oauth2.client.id"
"fs.azure.account.oauth2.client.secret"
"fs.azure.account.oauth2.client.endpoint"
"spark.hadoop.fs.azure.account.auth.type"
"spark.hadoop.fs.azure.account.oauth.provider.type"
"spark.hadoop.fs.azure.account.oauth2.client.id"
"spark.hadoop.fs.azure.account.oauth2.client.secret"
"spark.hadoop.fs.azure.account.oauth2.client.endpoint"

fs.azure.sas.<container-name>.<storage-account-name>.blob.core.windows.net
spark.hadoop.fs.azure.sas.<container-name>.<storage-account-name>.blob.core.windows.net
azure.wasb.stores

Individual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page.

As needed, you can pass additional properties to the Spark running environment through the spark.props configuration area.

Steps:

For more information on modifying these settings, see Configure for Spark.

There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server. By default, the Designer Cloud Powered by Trifacta platform re-attempts to submit a request 5 times and then fails the job if the request is not accepted.

If you want to change the number of retries, change the value for the databricks.maxAPICallRetries flag.

When the above configuration has been completed, you can select the running environment through the application.

See Run Job Page.

You can use API calls to execute jobs.

Please make sure that the request body contains the following:

    "execution": "databricksSpark",

For more information, see https://api.trifacta.com/ee/9.7/index.html#operation/runJobGroup

When running a job using Spark on Azure Databricks, the job may fail with the above invalid version error. In this case, the Databricks version of Spark has been deprecated.

Solution:

Since an Azure Databricks cluster is created for each user, the solution is to identify the cluster version to use, configure the platform to use it, and then restart the platform.

When you run a job on Databricks, the job may fail with the following error:

java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

The job.log file may contain something similar to the following:

2022-07-19T15:41:24.832Z - [sid=0cf0cff5-2729-4742-a7b9-4607ca287a98] - [rid=83eb9826-fc3b-4359-8e8f-7fbf77300878] - [Async-Task-9] INFO com.trifacta.databricks.spark.JobHelper - Got error org.apache.spark.SparkException: Stage 0 failed. Error: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-243-149-238.eu-west-1.compute.internal, executor driver): java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
...

This error is due to a class mismatch between the Designer Cloud Powered by Trifacta platform and Databricks.

Solution:

The solution is to disable the precedence of using the Spark JARs provided from the Designer Cloud Powered by Trifacta platform over the Databricks Spark JARs. Please perform the following steps: