Skip to main content

API Task - Run Deployment

This section describes how to run a deployment using the APIs available in Designer Cloud Powered by Trifacta Enterprise Edition.

  • A deployment is a packaging mechanism for versioning your production-level flows.

  • Deployments are created and managed through a separate interface.

  • For more information, see Overview of Deployment Manager.

A note about API URLs:

In the listed examples, URLs are referenced in the following manner:

<protocol>://<platform_base_url>/

In your product, these map references map to the following:

<http or https>://<hostname>:<port_number>/

For more information, see API Reference.

Run Job Endpoints

Depending on the type of job that you are running, you must use one of the following endpoints:

Run job:

Run a job to generate the outputs from a single recipe in a flow.

Endpoint

/v4/jobGroups/:id

Method

POST

Reference documentation

https://api.trifacta.com/ee/9.7/index.html#operation/runJobGroup

Run flow:

Run all outputs specified in a flow. Optionally, you can run all scheduled outputs.

Endpoint

/v4/flows/:id/run

Method

POST

Reference documentation

https://api.trifacta.com/ee/9.7/index.html#operation/runFlow

Run deployment:

Run the primary flow in the active release of the specified deployment.

Tip

This method is covered on this page.

Deployments are available only through the Deployment Manager. For more information, see Overview of Deployment Manager.

Endpoint

/v4/deployments/:id/run

Method

POST

Reference documentation

https://api.trifacta.com/ee/9.7/index.html#operation/runDeployment

Prerequisites

Before you begin, you should verify the following:

  1. Get authentication credentials. As part of each request, you must pass in authentication credentials to the Designer Cloud Powered by Trifacta platform.

    Tip

    The recommended method is to use an access token, which can be generated from the Trifacta Application. For more information, see Access Tokens Page.

    See https://api.trifacta.com/ee/9.7/index.html#operation/Authentication

  2. Develop your flow. Before you deploy a flow through the Deployment Manager, you should build and test your flow in a development environment. See API Task - Develop a Flow.

  3. Deploy your flow. After you have developed a flow, you can deploy it. See API Task - Deploy a Flow.

  4. Acquire deployment identifier.In Deployment Manager, acquire the numeric value for the deployment. See Deployment Manager Page.

Step - Run Deployment

Through the APIs, you can run a deployment. When a deployment is executed:

  • All of the recipes that are included in the active release of the deployment are executed.

  • All default values for outputs are applied.

In the following example, the deployment identifier is 2.

Endpoint

<protocol>://<platform_base_url>/v4/deployments/2/run

Authentication

Required

Method

POST

Request Body

None.

Response Code

201 - Created

Response Body

{
    "data": [
        {
            "reason": "JobStarted",
            "sessionId": "dd6a90e0-c353-11e7-ad4e-7f2dd2ae4621",
            "id": 33
        }
    ]
}

If the 201 response code is returned, then the deployment job has been queued for execution.

Tip

Retain the id value in the response. In the above, 961247 is the internal identifier for the job group for the job. You will need this value to check on your job status.

For more information, see API Task - Deploy a Flow.

Tip

You have queued your deployment job for execution.

Step - Monitoring Your Deployment Job

You can monitor the status of your deploy job through the following endpoint using the id value that was returned in the previous step:

<protocol>://<platform_base_url>/v4/jobGroups/<id>/status

For more information, see API Task - Run Job.

Step - Run Deployment with Overrides

When you run a deployment, you cannot apply overrides to the request. However, you can complete the following steps to apply overrides when you execute the jobs within the active release. In this task, you run jobs on the specific recipes of the active deployment, applying overrides as needed.

Suppose you are running the jobs for deployment id 2, and you want to apply some overrides.

Note

A deployment can trigger several different jobs within a single flow. In the following example, it is assumed that there is only one output.

Acquire the active outputs for the deployment

The first step is to acquire all of the active outputs for the deployment.

Endpoint

<protocol>://<platform_base_url>/v4/deployments/2/activeoutputs

Authentication

Required

Method

GET

Request Body

None.

Response Code

200 - OK

Response Body

{
    "data": [
        {
            "outputObjectId": 6,
            "flowNodeId": 27,
            "recipeName": "USDA_Farmers_Market_2014"
        }
    ]
}

in the above response:

  • The flowNodeId value corresponds to the recipe (wrangledDataset) identifier that you wish to modify.

  • The outputObjectId value corresponds to the output object that is produced by default from the recipe.

Acquire output object information

The next step is to review the output object to determine what needs to be overridden. Since you are overriding a file-based publication, you can query directly for the writeSettings objects associated with the output object:

Endpoint

<protocol>://<platform_base_url>/v4/outputObjects/6/writeSettings

Authentication

Required

Method

GET

Request Body

None.

Response Code

200 - OK

Response Body

{
    "data": [
        {
            "delim": ",",
            "id": 17,
            "path": "hdfs://hadoop:50070/example/joe@example.com/USDA_Farmers_Market_2014.csv",
            "action": "create",
            "format": "csv",
            "compression": "none",
            "header": true,
            "asSingleFile": true,
            "prefix": null,
            "suffix": "_increment",
            "hasQuotes": true,
            "createdAt": "2019-11-05T18:26:31.972Z",
            "updatedAt": "2019-11-05T18:30:56.756Z",
            "creator": {
                "id": 2
            },
            "updater": {
                "id": 2
            },
            "outputObject": {
                "id": 6
            }
        },
        {
            "delim": ",",
            "id": 16,
            "path": "hdfs://hadoop:50070/example/joe@example.com/USDA_Farmers_Market_2014.json",
            "action": "create",
            "format": "json",
            "compression": "none",
            "header": false,
            "asSingleFile": false,
            "prefix": null,
            "suffix": "_increment",
            "hasQuotes": false,
            "createdAt": "2019-11-05T18:26:44.983Z",
            "updatedAt": "2019-11-05T18:30:56.743Z",
            "creator": {
                "id": 2
            },
            "updater": {
                "id": 2
            },
            "outputObject": {
                "id": 6
            }
        }
    ]
}

Run deployment with overrides

Now that you have access to the outputs generated by the deployment, you decide to override the following for each file:

  • Filename:

    • Remove the year information at the end of the filename

    • Store in a separate folder called final

  • Wipe the table and reload it each time (action=overwrite)

  • Disable profiling

From the activeOutputs endpoint, you retrieved the flowNodeId (27).

Endpoint

<protocol>://<platform_base_url>/v4/jobGroups/

Authentication

Required

Method

POST

Request Body

{
  "wrangledDataset": {
    "id": 27
  },
  "overrides": {
    "profiler": false,
    "writesettings": [
      {
        "path": "hdfs://hadoop:50070/example/joe@example.com/final/USDA_Farmers_Market.csv",
        "format": "csv",
        "action": "overwrite"
      },
      {
        "path": "hdfs://hadoop:50070/example/joe@example.com/final/USDA_Farmers_Market.json",
        "format": "json",
        "action": "overwrite"
      }
    ]
  },
  "ranfrom": null
}

Response Code

201 - Created

Response Body

{
    "sessionId": "b29467c3-fc59-499e-aed6-d797746a86eb",
    "reason": "JobStarted",
    "jobGraph": {
        "vertices": [
            10,
            11,
            12
        ],
        "edges": [
            {
                "source": 10,
                "target": 11
            },
            {
                "source": 10,
                "target": 12
            }
        ]
    },
    "id": 4,
    "jobs": {
        "data": [
            {
                "id": 10
            },
            {
                "id": 11
            },
            {
                "id": 12
            }
        ]
    }
}

Tip

Your job with overrides has been queued for execution.

You can use the job identifier (4) to monitor the job status.

Step - Run Deployment with Spark Overrides

When you run a deployment, you can specify override values to the Spark properties that have been made available for overrides.

Note

These overrides only apply if you are running the job on Spark and if the feature has been enabled in your deployment.

  • This feature and the properties to override must be enabled. See Enable Spark Job Overrides.

  • All of the recipes that are included in the active release of the deployment are executed.

All other default values are applied. In the following example, the deployment identifier is 2.

Endpoint

<protocol>://<platform_base_url>/v4/deployments/2/run

Authentication

Required

Method

POST

Request Body

{
  "sparkOptions": [
    {
      "key": "spark.executor.cores",
      "value": "2"
    },
    {
      "key": "spark.executor.memory",
      "value": "4GB"
    }
  ]
}

Response Code

201 - Created

Response Body

{
    "data": [
        {
            "reason": "JobStarted",
            "sessionId": "dd6a90e0-c353-11e7-ad4e-7f2dd2ae4621",
            "id": 33
        }
    ]
}

If the 201 response code is returned, then the deployment job has been queued for execution.

Tip

You have queued your deployment job for execution.

In this section: