Configure for AWS Databricks

By default, the number of jobs permitted on an AWS workspace is set to 1000.

These limits apply to any jobs that use workspace data on the cluster.

To enable retrieval and auditing of job information after a job has been completed, the Designer Cloud Powered by Trifacta platform does not delete jobs from the cluster. As a result, jobs can accumulate over time to exceed the number of jobs permitted on the cluster. If you reach these limits, you may receive a Quota for number of jobs has been reached limit. For more information, see https://docs.databricks.com/user-guide/jobs.html.

Optionally, you can allow the Designer Cloud Powered by Trifacta platform to manage your jobs to avoid these limitations. For more information, see "Configure Databricks job management" below.

When a user submits a job, the Designer Cloud Powered by Trifacta Enterprise Edition provides all the cluster specifications in the Databricks API and it creates cluster only for per-user or per-job, that means once the job is complete, the cluster is terminated. Cluster creation may take less than 30 seconds if instance pools are used. If the instance pools are not used, it may take 10-15 minutes.

For more information on job clusters, see https://docs.databricks.com/clusters/configure.html.

The job clusters automatically terminate after the job is completed. A new cluster is automatically created when the user next requests access to AWS Databricks access.

Optionally, you can configure the Designer Cloud Powered by Trifacta platform to use the Databricks cluster policies that have been specified by your Databricks administrator for creating and using clusters. These policies are effectively templates for creation and use of Databricks clusters and govern aspects of clusters such as the type and count of nodes the resources that can be accessed via the cluster, and other settings. For more information on Databricks cluster policies, see https://docs.databricks.com/administration-guide/clusters/policies.html.

Prerequisites

Notes:

Steps:

Job overrides:

A user's cluster policy can overridden when a job is executed via API. Set the request attribute for the clusterPolicyId.

For more information, see API Task - Run Job.

Policy template for AWS - without instance pools:

The following example cluster policy can provide a basis for creating your own AWS cluster policies when instance pools are not in use:

{
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
  "aws_attributes.availability": {
    "type": "fixed",
    "value": "SPOT_WITH_FALLBACK",
    "hidden": false
  },
  "aws_attributes.ebs_volume_count": {
    "type": "fixed",
    "value": 0,
    "hidden": false
  },
  "aws_attributes.ebs_volume_size": {
    "type": "fixed",
    "value": 0,
    "hidden": false
  },
  "aws_attributes.first_on_demand": {
    "type": "fixed",
    "value": 1,
    "hidden": false
  },
  "aws_attributes.spot_bid_price_percent": {
    "type": "fixed",
    "value": 100,
    "hidden": false
  },
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "arn:aws:iam::9999999999999:instance-profile/SOME_Role_ARN",
    "hidden": false
  },
  "driver_node_type_id": {
    "type": "fixed",
    "value": "i3.xlarge",
    "hidden": true
  },
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  },
  "node_type_id": {
    "type": "fixed",
    "value": "i3.xlarge",
    "hidden": true
  }
}

Policy template for AWS - without instance pools:

The following example cluster policy can provide a basis for creating your own AWS cluster policies when instance pools are in use:

{
  "autoscale.max_workers": {
    "type": "fixed",
    "value": 3,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "fixed",
    "value": 1,
    "hidden": true
  },
  "aws_attributes.instance_profile_arn": {
    "type": "fixed",
    "value": "arn:aws:iam::9999999999:instance-profile/SOME_POLICY",
    "hidden": false
  },
  "enable_local_disk_encryption": {
    "type": "fixed",
    "value": false
  },
  "instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "driver_instance_pool_id": {
    "type": "fixed",
    "value": "SOME_POOL",
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10,
    "hidden": true
  },
}

Designer Cloud Powered by Trifacta platform EC2 instances can be configured with permissions to access AWS resources like S3 by attaching an IAM instance profile. Similarly, instance profiles can be attached to EC2 instances for use with AWS Databricks clusters.

To configure the instance profile for AWS Databricks, you must provide an IAM instance profile ARN in databricks.awsAttributes.instanceProfileArn parameter.

For more information, see Configure for AWS Authentication.

Instance pooling reduces cluster node spin-up time by maintaining a set of idle and ready instances. The Designer Cloud Powered by Trifacta platform can be configured to leverage instance pooling on the AWS Databricks cluster for both worker and driver nodes.

For more information, see https://docs.databricks.com/clusters/instance-pools/configure.html.

Review and modify the following configuration settings, as required:

Following is the list of parameters that have to be set to integrate the AWS Databricks with Designer Cloud Powered by Trifacta platform:

Required Parameters

metadata.cloud

Following is the list of parameters that can be reviewed or modified based on your requirements:

Optional Parameters

AWS Databricks enforces a hard limit of 1000 created jobs per workspace, and by default cluster jobs are not deleted. To support jobs more than 1000 jobs per cluster, you can enable job management for AWS Databricks.

Steps:

Optionally, you can leverage Databricks Secrets Management to store sensitive Databricks configuration properties. When this feature is enabled and a set of properties are specified, those properties and their values are stored in masked form. For more information on Databricks Secrets Management, see https://docs.databricks.com/security/secrets/index.html.

Steps:

Optionally, you can provide to the Designer Cloud Powered by Trifacta platform the name of a shared Databricks cluster to be used to access Databricks Tables.

Prerequisites:

"fs.s3a.access.key"
"fs.s3a.secret.key"
"spark.hadoop.fs.s3a.access.key"
"spark.hadoop.fs.s3a.secret.key"

"fs.s3a.credentialsType"
"fs.s3a.stsAssumeRole.arn"
"fs.s3a.canned.acl"
"fs.s3a.acl.default"
"spark.hadoop.fs.s3a.credentialsType"
"spark.hadoop.fs.s3a.stsAssumeRole.arn"
"spark.hadoop.fs.s3a.canned.acl"
"spark.hadoop.fs.s3a.acl.default"

"fs.s3a.canned.acl"
"fs.s3a.acl.default"
"spark.hadoop.fs.s3a.canned.acl"
"spark.hadoop.fs.s3a.acl.default"

The AWS Secrets Manager is a secure vault for storing access credentials to AWS resources.

For more information, see Configure for AWS Secrets Manager.

A single AWS Databricks account can have access to multiple Databricks workspaces. You can create more than one workspace by using Account API if you are account is on the E2 version of the platform or on a selected custom plan that allows multiple workspaces per account.

For more information, see https://docs.databricks.com/administration-guide/account-api/new-workspace.html

Each workspace has a unique deployment name associated with it that defines the workspace URL. For example: https://<deployment-name>.cloud.databricks.com.

By default, Databricks workspaces apply limits on the number of jobs that can be submitted before the cluster begins to fail jobs. These limits are the following:

Depending on how your clusters are configured, these limits can vary. For example, if the maximum number of concurrent jobs per cluster is set to 20, then the 21st concurrent job submitted to the cluster fails.

To prevent unnecessary job failure, the Designer Cloud Powered by Trifacta platform submits the throttling of jobs to Databricks. When job throttling is enabled and the 21 concurrent job is submitted, the Designer Cloud Powered by Trifacta platform holds that job internally the first of any of the following events happens:

Steps:

Please complete the following steps to enable job throttling.

Each user must insert a Databricks Personal Access Token to access Databricks resources. For more information, see Databricks Settings Page.

Individual users can specify the name of the cluster to which they are permissioned to access Databricks Tables. This cluster can also be shared among users. For more information, see Databricks Settings Page.

There is a limit of 30 requests per second per workspace on the Databricks REST APIs. If this limit is reached, then a HTTP status code 429 error is returned, indicating that rate limiting is being applied by the server. By default, theDesigner Cloud Powered by Trifacta platform re-attempts to submit a request 5times and then fails the job if the request is not accepted.

If you want to change the number of retries, change the value for the databricks.maxAPICallRetries flag.

When the above configuration has been completed, you can select the running environment through the application.

See Run Job Page.

You can use API calls to execute jobs.

Make sure that the request body contains the following:

    "execution": "databricksSpark",

For more information, see https://api.trifacta.com/ee/9.7/index.html#operation/runJobGroup

When running a job using Spark on AWS Databricks, the job may fail with the above invalid version error. In this case, the Databricks version of Spark has been deprecated.

Solution:

Since an AWS Databricks cluster is created for each user, the solution is to identify the cluster version to use, configure the platform to use it, and then restart the platform.

When you run a job on Databricks, the job may fail with the following error:

java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

The job.log file may contain something similar to the following:

2022-07-19T15:41:24.832Z - [sid=0cf0cff5-2729-4742-a7b9-4607ca287a98] - [rid=83eb9826-fc3b-4359-8e8f-7fbf77300878] - [Async-Task-9] INFO com.trifacta.databricks.spark.JobHelper - Got error org.apache.spark.SparkException: Stage 0 failed. Error: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-243-149-238.eu-west-1.compute.internal, executor driver): java.lang.ClassCastException: org.apache.spark.scheduler.ResultTask cannot be cast to org.apache.spark.scheduler.Task
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:616)
...

This error is due to a class mismatch between the Designer Cloud Powered by Trifacta platform and Databricks.

Solution:

The solution is to disable the precedence of using the Spark JARs provided from the Designer Cloud Powered by Trifacta platform over the Databricks Spark JARs. Please perform the following steps: