Python Test Harness Guide
The SDK includes a Python test harness that simulates running a plugin outside Alteryx Designer. It uses local test data. You use it to validate a plugin during development, check outputs, and reproduce behavior without opening Designer.
The test harness is part of the AYX Python SDK package (ayx_python_sdk.test_harness).
What the Test Harness Is
The harness starts a plugin, loads the plugin configuration, and feeds input data from CSV files. It then runs the plugin with the transport type and the run mode that you select. For help with the format of the input CSV data, go to the tutorial later on this page.
You usually provide these items:
The plugin entry point.
The package that contains the plugin.
The plugin name.
The plugin configuration XML file.
Optional CSV input files.
The harness is a local test runner, not a replacement for Designer. It lets you verify plugin behavior in a controlled environment before you install the plugin in Designer.
Use the test harness when you need to do these things:
Verify a plugin without opening Designer.
Repeat the same input scenario while you debug.
Compare the expected output with the actual output.
Check update-only behavior separately from a full run.
Validate local changes before you move into Designer.
Command Overview
The current test harness entry point exposes a run_plugin command.
The command has four required positional arguments:
plugin_entrypoint: The path to the Python file that contains the plugin entry point.plugins_package: The name of the package that contains the plugin code.tool_name: The name of the plugin to run.config_xml: The plugin configuration XML file that holds the input and output anchor configuration.
The command also accepts these optional inputs:
--input-csv: One or more CSV files to use as input data. You can provide more than one input file when the plugin expects multiple inputs.--transport-type: The data transport. Valid values arearrowandengine. The default isarrow.--run-mode: The execution mode. Valid values arefull_runandupdate_only. The default isfull_run.
Here is an example:
python -m ayx_python_sdk.test_harness run_plugin `
<plugin_entrypoint> `
<plugins_package> `
<tool_name> `
<config_xml> `
--input-csv <input_csv> `
--transport-type <arrow|engine> `
--run-mode <full_run|update_only>
The way that you launch the harness can vary with your environment, but the inputs are always the same. You point the harness at the plugin, provide the configuration, and supply any data files that the plugin needs.
Transport Types
The transport type controls how data moves between the harness and the plugin under test. It affects the data format that you work with. It also affects how closely local testing matches the platform.
The harness supports two transport types: arrow and engine.
Arrow
arrow is the default transport, and it’s usually the best place to start. It passes data in the Arrow columnar format. This format is efficient for tabular data and maps naturally to Python data processing.
Use arrow when you want these things:
Fast, table-oriented data exchange.
A Python-first development and testing experience.
Simple local iteration with less setup.
Predictable handling of structured rows and columns.
Arrow helps most when your plugin reads rows, transforms fields, filters records, or produces tabular output.
Example Use Cases
Validating a plugin that adds or renames columns.
Transforming input rows into a new output schema.
Testing aggregation logic over a table of records.
Building and debugging a plugin locally before you compare it with engine behavior.
Example
Suppose your input looks like this:
Name | Age |
|---|---|
Ana | 31 |
Ben | 28 |
An arrow test typically lets you work with that table directly. You can focus on the transformation logic, such as adding an IsAdult column or normalizing text values.
Why Choose Arrow
Choose arrow for simple, efficient day-to-day development. It’s often the right default because it gives you quick feedback and is easy to reason about.
Engine
engine is useful when you need behavior that more closely resembles the engine-side data flow of the platform. You usually choose it to investigate compatibility, execution differences, or runtime behavior that depends on engine semantics.
Use engine when you need these things:
A closer match to how the platform processes data internally.
A comparison of local behavior with engine-oriented execution.
Validation of edge cases that depend on the runtime or the execution order.
An investigation of differences between your plugin and the platform environment.
Use engine transport when a test passes with arrow but the plugin still behaves differently in the platform. Use it also to reproduce a bug that appears only during engine-style execution.
Example Use Cases
Reproducing a failure that happens only in the platform runtime.
Comparing output from local development with engine execution.
Testing behavior that depends on how the engine materializes or routes data.
Checking compatibility with scenarios that are sensitive to platform-specific data flow.
Example
Suppose a plugin works with a simple table locally but fails in the platform. The cause might be how the platform handles nulls, ordering, or other runtime details. Switching to engine can reproduce the issue more faithfully. It also helps you tell whether the problem is in the plugin logic or in an assumption of the local test.
Why Choose Engine
Choose engine when your main goal is fidelity, not convenience. It’s more useful for debugging platform-specific behavior than for early-stage development.
How to Choose
A good rule of thumb is to start with arrow for normal development and fast iteration. Use engine when you need to compare against platform behavior or reproduce execution-specific issues.
If you aren’t sure, start with arrow. If the result looks correct there but still differs in the platform, switch to engine and compare the outcomes.
Run Modes
The run mode controls how much of the plugin lifecycle the harness runs.
full_run
full_run performs the normal end-to-end execution path for the plugin. It’s the most common mode for local validation.
Use full_run to test these things:
Incoming record handling.
Output writing.
Cleanup behavior.
The overall flow of the plugin.
update_only
update_only runs only the update portion of the lifecycle. Use it when you need to review metadata or configuration behavior before record processing begins.
This mode is useful for plugins that do these things:
Adjust metadata during the update.
Need to react before data processing starts.
Should be verified without running a full data batch.
Quick Tutorial
Step 1: Prepare Input Data
Create a small CSV file to use as test input. The harness expects the CSV column names to include field metadata in this form:
FieldName-FieldType-FieldSizeFor example:
Name-String-50,Value-Int32-8
Alpha,10
Beta,20
If your plugin has multiple input anchors, provide one CSV file for each input anchor.
Step 2: Run the Plugin Through the Harness
Launch the harness and point it at these items:
The plugin entry point file.
The Python package that contains the plugin.
The plugin name.
The plugin configuration XML file. For example,
UppercaseText_1_0Config.xml, which the CLI generates in the folder of the plugin underconfiguration/.One or more input CSV files.
Here is an example:
python -m ayx_python_sdk.test_harness run_plugin `
path/to/plugin.py `
my_package `
MyTool `
path/to/MyTool_1_0Config.xml `
--input-csv path/to/input.csv `
--transport-type arrow `
--run-mode full_run
Step 3: Check the Output
The harness runs the plugin out of process, sends in the input records, and collects the resulting output.
Use this first run to confirm these things:
The plugin starts successfully.
The configuration file matches the plugin.
The harness accepts the input data.
The output looks like what you expect.
For a passthrough plugin, the output matches the input. For a transform plugin, the output reflects the changes that you expect.
The test harness doesn’t have integrated unit test support. You validate the results manually, or you direct them to a test wrapper. For more information about automated testing, go to the Test Scaffolding page, which describes SdkToolTestService.
Step 4: Iterate
If the output isn’t correct, update the plugin and run the same command again. This quick edit-run-check loop is the main purpose of the harness during development.
Practical Tips
Start with a very small input file so that it’s easy to inspect the results.
Make sure that the configuration XML file belongs to the plugin that you run.
Make sure that the number of input CSV files matches the number of input anchors.
Run the same command repeatedly while you refine the plugin logic.
How Input Data Is Handled
The harness loads input data from CSV files before the plugin runs. This makes it easy to test with small, focused data sets, instead of building the same scenario by hand in Designer.
For most authors, a local test fixture includes these items:
A plugin source file.
A workspace package.
A configuration XML file.
One or more CSV input files.
The harness derives metadata from the structure of the CSV input that it receives. Use predictable column names and data types in your test inputs.
Work with Multiple Inputs
If a plugin accepts more than one input connection, you can supply more than one input CSV file.
For a plugin with multiple input anchors, pass one --input-csv for each anchor. Use the same order as the anchors have in the plugin configuration XML file.
Here is an example with two inputs:
python -m ayx_python_sdk.test_harness `
run_plugin path/to/plugin.py `
my_package MyTool `
path/to/MyTool_1_0Config.xml `
--input-csv path/to/input_left.csv `
--input-csv path/to/input_right.csv `
--transport-type arrow `
--run-mode full_run
At first, keep the test data small and readable. It’s easier to confirm anchor behavior when each input file clearly represents one stream of data.
Suggested Development Flow
A simple testing loop looks like this:
Build or update the plugin.
Prepare a small CSV file that represents the scenario that you want to test.
Run the harness with
full_run.Review the results and adjust the plugin.
Use
update_onlywhen metadata or update-time behavior needs attention.
This workflow helps while you learn the plugin lifecycle. It gives you a fast feedback loop.
Relationship to Designer
The test harness is for local validation and development. It doesn’t replace Designer. It’s a useful step before you test the plugin in Designer.
If a plugin works in the harness but not in Designer, check the workspace configuration and the user interface behavior. Also check what the plugin assumes about the Designer runtime.