Parameterize Files for Import

Each file that is included as part of the dataset with parameters should have identical structures:

When working with datasets with parameters, it may be useful to do the following if you expect the underlying datasets to be less than 100% consistent with each other.

Datetime parameters require the following elements:

Format: You must specify the format of the matching date and/or time values using alphanumeric patterns. To review a list of example formats, click Browse Date/Timestamp Patterns.

You can also create custom formats using patterns. For example, the following regex pattern matches patterns like MM.DD.YYYY:

/[0-9][0-9]\.[0-9][0-9]\.[0-9][0-9][0-9][0-9]/

Date range: Use these controls to specify the range that matching dates must fall within.

Time zone: The default time zone is the location of the host of the application. To change the current time zone, click Change.

For a list of supported time zone values, see Supported Time Zone Values.

A parameterized dataset can support only one Datetime parameter. If you have multiple parts of the path that contain date information, you can create a Datetime element for each part.

Steps:

A variable parameter is a key-value pair that can be inserted into the path.

Name: The name of the variable is used to identify its purpose.

Default Value: If the variable value is not overridden at execution time, this value is inserted in the variable location in the path.

You can create environment parameters to specify your bucket names. An environment parameter is a variable name and String value that can be referenced by all users of the environment.

Uses:

For example, suppose you have two environments: Dev and Prod. You can create an environment parameter called env.sourceBucketName to store the name of the bucket from which all data in the workspace or project is imported.

MyCo_Dev

$env.sourceBucketName = 'MyCo_Dev'

MyCo_Prod

$env.sourceBucketName = 'MyCo_Prod'

For more information, seeEnvironment Parameters Page.

In the screen above, you can see an example of pattern-based parameterization. In this case, you are trying to parameterize the two digits after the value: POS-r.

When you create a wildcard or pattern-based parameter, you have the option to scan any nested folders for matching sources.

Suppose your file and folder structure look like the following:

//source/user/me/datasets/thisfile.txt
//source/user/me/datasets/thatfile.txt
//source/user/me/datasets/anotherfile.csv
//source/user/me/datasets/detail/anestedfile.txt
//source/user/me/datasets/detail/anestedfile2.txt
//source/user/me/datasets/detail/anestedfile4.txt
//source/user/me/ahigherfile.txt

Since the filenames vary significantly, it may be easiest to create your pattern based on a wildcard. You create a wildcard parameter on the first file in the //source/user/me/datasets directory:

//source/user/me/datasets/*.txt

For the specified directory, the above pattern matches on any text file (.txt). In the example, it matches on the first two files but does not match on the CSV file.

When the Include nested folders checkbox is selected:

Example 2: pattern-based files

Suppose your file and folder structure look like the following:

//source/user/me/datasets/file01.csv
//source/user/me/datasets/file02.csv
//source/user/me/datasets/file03.csv
//source/user/me/datasets/detail/file04.csv
//source/user/me/datasets/detail/file05.csv
//source/user/me/datasets/detail/file06.csv
//source/user/me/file07.csv

You create a pattern parameter on the first file in the //source/user/me/datasets directory with the following pattern-based parameter:

`file{digit}+`

The above pattern matches on the word file and a sequence of one or more digits. For example, suppose file100.csv lands in the directory at some point in the future. This pattern would capture it.

In the above example, this pattern matches on the first three files, which are all in the same directory.

When the Include nested folders checkbox is selected:

The easiest way to is to add a wildcard: *

A wildcard can be any value of any length, including an empty string.

Instead of a wildcard match, you could specify a regular expression match. Regular expressions are a standardized means of expressing patterns.

Regular expressions are specified between forward slashes, as in the following:

/my_regular_expression/

The following regular expression matches the same two sources in the previous screen:

/\_[0-9]*\_[0-9]*/

The above expression matches an underscore (_) followed by any number of digits, another underscore, and any number of digits.

While the above matches the two sources, it also matches any of the following:

_2_1
__1
_1231231231231231235245234343_

These may not be proper matches. Instead, you can add some specificity to the expression to generate a better match:

/\_[0-9]{13}\_[0-9]{4}/

The above pattern matches an underscore, followed by exactly 13 digits, another underscore, and then another 4 digits. This pattern matches the above two sources exactly, without introducing the possibility of matching other numeric patterns.

A Alteryx pattern is a platform-specific mechanism for specifying patterns, which is much simpler to use than regular expressions. These simple patterns can cover much of the same range of pattern expression as regular expressions without the same risks of expression and sometimes ugly syntax.

Alteryx patterns are specified between back-ticks, as in the following:

`my_pattern`

In the previous example, the following regular expression was used to match the proper set of files:

/\_[0-9]{13}\_[0-9]{4}/

In a Alteryx pattern, the above can be expressed in a simpler format:

`\_{digit}{13}\_{digit}{4}`

This simpler syntax is easier to parse and performs the same match as the regular expression version.