API Task - Develop a Flow

Locate or create flow. The datasets that you wrangle must be contained within a flow. You can add them to an existing flow or create a new one through the APIs.

Create dataset. Through the APIs, you create an imported dataset from an asset that is accessible through one of the established connections. Then, you create the recipe object through the API.

1. For the recipe, you must retrieve the internal identifier.

2. Through the application, you modify the recipe for the dataset.

Automate job execution. Using the APIs, you can automate execution of the wrangling of the dataset.

1. As needed, this job can be re-executed on a periodic basis or whenever the source files are updated.

Login through the application.

In the Flows page, select or create the flow to contain the above datasets.

In the Flow Details page for that flow, locate the flow identifier in the URL:

http://www.example.com:3005/flows/10

10

Retain this identifier for later use.

Through the APIs, you can create a flow using the following call:

http://www.example.com:3005/v4/flows

POST

{
  "name": "Point of Sale - 2013",
  "description": "Point of Sale data for state"
}

The response should be status code 201 - Created with a response body like the following:

{
  "id": 10,
  "updatedAt": "2017-02-17T17:08:57.848Z",
  "createdAt": "2017-02-17T17:08:57.848Z",
  "name": "Point of Sale - 2013",
  "description": "Point of Sale data for state",
  "creator": {
    "id": 1
  },
  "updater": {
    "id": 1
  },
  "workspace": {
    "id": 1
  }
}

Retain the flow identifier (10) for later use.

Create an imported dataset for each file.

For each imported dataset, create a recipe, which can be used to transform the imported dataset.

To create an imported dataset, you must acquire the following information about the source. In the above example, the source is the POS-r01.txt file.

1. uri

2. name

3. description

4. bucket (if a file stored on S3)

Construct the following request:

http://www.example.com:3005/v4/importedDatasets

POST

{
  "uri": "hdfs:///user/pos/POS-r01.txt",
  "name": "POS-r01.txt",
  "description": "POS-r01.txt"
}

You should receive a 201 - Created response with a response body similar to the following:

{
  "id": 8,
  "size": "281032",
  "uri": "hdfs:///user/pos/POS-r01.txt",
  "dynamicPath": null,
  "bucket": null,
  "isSchematized": true,
  "isDynamic": false,
  "disableTypeInference": false,
  "updatedAt": "2017-02-08T18:38:56.640Z",
  "createdAt": "2017-02-08T18:38:56.560Z",
  "parsingScriptId": {
    "id": 14
  },
   "runParameters": {
        "data": []
    },
    "name": "POS-r01.txt",
    "description": "POS-r01.txt",
    "creator": {
        "id": 1
    },
    "updater": {
        "id": 1
    },
    "connection": null
}

You must retain the id value so you can reference it when you create the recipe.

See Dataprep by Trifacta: API Reference docs

Next, you create the recipe. Construct the following request:

http://www.example.com:3005/v4/wrangledDatasets

POST

{ "name":"POS-r01",
  "importedDataset": {
    "id":8
  },
  "flow": {
    "id":10
  }
}

You should receive a 201 - Created response with a response body similar to the following:

{
    "id": 23,
    "wrangled": true,
    "updatedAt": "2018-02-06T19:59:22.735Z",
    "createdAt": "2018-02-06T19:59:22.698Z",
    "name": "POS-r01",
    "active": true,
    "referenceInfo": null,
    "activeSample": {
        "id": 23
    },
    "creator": {
        "id": 1
    },
    "updater": {
        "id": 1
    },
    "recipe": {
        "id": 23
    },
    "flow": {
        "id": 10
    }
}

From the recipe, you must retain the value for the id. For more information, see Dataprep by Trifacta: API Reference docs

Repeat the above steps for each of the source files that you are adding to your flow.

Open the POS-r01 dataset. It's loaded in the Transformer page.

To chain together the other transactional data into this dataset, you use a union transform. In the Search panel, enter union in the textbox and press ENTER.

In the Union page:

Open the POS-r01 dataset.

In Search panel, enter join datasets for the transform. The Join window opens.

Select the RED_PROD dataset. Click Accept. Click Next.

Review the two keys to verify that they are the proper columns on which to structure the join. Click Next.

Click the All tab. Select all fields to add. Click Review.

After reviewing your join, click Add to Recipe.

For each Item_Nbr value that has a matching ITEM_NBR value in the reference dataset, all of the other reference fields are pulled into the POS-r01 dataset.

Acquire the internal identifier for the recipe for which you wish to execute a job. In the previous example, this identifier was 23.

Construct a request using the following:

http://www.example.com:3005/v4/jobGroups

POST

Request Body:

{
  "wrangledDataset": {
    "id": 23
  },
  "overrides": {
    "execution": "photon",
    "profiler": true,
    "writesettings": [
      {
        "path": "hdfs://hadoop:50070/trifacta/queryResults/admin@example.com/POS-r01.csv",
        "action": "create",
        "format": "csv",
        "compression": "none",
        "header": false,
        "asSingleFile": false
      }
    ]
  },
  "ranfrom": null
}

In the above example, the specified job has been launched for recipe 23 to execute on the Trifacta Photon running environment with profiling enabled.

1. Output format is CSV to the designated path. For more information on these properties, see Dataprep by Trifacta: API Reference docs

2. Output is written as a new file with no overwriting of previous files.

A response code of 201 - Created is returned. The response body should look like the following:

{

    "sessionId": "79276c31-c58c-4e79-ae5e-fed1a25ebca1",
    "reason": "JobStarted",
    "jobGraph": {
        "vertices": [
            21,
            22
        ],
        "edges": [
            {
                "source": 21,
                "target": 22
            }
        ]
    },
    "id": 3,
    "jobs": {
        "data": [
            {
                "id": 21
            },
            {
                "id": 22
            }
        ]
    }
}

Retain the id value, which is the job identifier, for monitoring.