Metadata

This method is called at plugin creation when Designer requests the output data types from the plugin before the workflow runs. You can send metadata to Designer at this time.

def __init__(self, provider: AMPProviderV2):
    provider.push_outgoing_metadata("Output", create_schema({
        "volts": {
            "type": FieldType.int16
        },
        "device" {
            "type": FieldType.string
        }
    }))

This method is called each time Designer sends data through the plugin. You can extract metadata from the input data for information and modify output data metadata. However, we don't recommend modifying the type, size, and scale as unexpected results can occur in Designer. You can modify the source and description.

On input plugins, a complete schema created with metadata can be exported to Designer for a precise data schema.

In this example, we specify type int16 for column volts and type string for column device.

def __init__(self, provider: AMPProviderV2):
    provider.push_outgoing_metadata("Output", create_schema({
        "volts": {
            "type": FieldType.int16
        },
        "device" {
            "type": FieldType.string
        }
    }))

In this case, description metadata is added to the batch.

def on_record_batch(self, batch: "pa.Table", anchor: Anchor) - > None:
    batch = set_metadata(batch, {
        "volts": {
            "description": "Define the tension of the current"
        }
    })

    self.provider.write_to_anchor("Output", batch)

This example shows how to use the source metadata. In this case, the source metadata is a file path, but it can be anything that describes the address of the source of the data.

def on_record_batch(self, batch: "pa.Table", anchor: Anchor) - > None:
    meta = get_metadata(batch, "volts")
    filepath = meta["source"]
    with f = open(Path(filepath)):
        content = f.read()
        #...additional operations...
    
    self.provider.write_to_anchor("Output", batch)

set_metadata(table, col_meta={}, schema=None)

This function updates column metadata from the given PyArrow table with a dictionary of column metadata or directly with a PyArrow Schema. It returns the new table with the new metadata. The input table remains unchanged.

def on_record_batch(self, batch: "pa.Table", anchor: Anchor) - > None:
    batch = set_metadata(batch, {
        "volts": {
            "description": "tension"
        },
        "ampere": {
            "source": "https://something.com"
        }
    })

self.provider.write_to_anchor("Output", batch)

get_metadata(table, col_name)

Get all columns' metadata or 1 column’s metadata if col_name is given, from the input PyArrow table.

Examples

def on_record_batch(self, batch: "pa.Table", anchor: Anchor) -> None:
        meta_for_column_volts = get_metadata(batch, "volts")

The above example get_metadata returns the metadata for 1 column, of the following format (in this example):

meta_for_column_volts = {
    "type": 3,
    "size": 2,
    "scale": 0,
    "source": "",
    "description": ""
}

Nota

You can see in the above example that a number represents the type. Please refer to the Types section below for the mapping between type IDs and type names.

This example returns all metadata:

def on_record_batch(self, batch: "pa.Table", anchor: Anchor) -> None:
        all_meta = get_metadata(batch)

The result dictionary looks like this:

all_meta = {
    "volts": {
        "type": 3,
        "size": 2,
        "scale": 0,
        "source": "",
        "description": ""
    }
    "device": {
        "type": 9,
        "size": 64,
        "scale": 0,
        "source": "",
        "description": ""
    }
}

Refer to this sequence diagram for a visual representation of the metadata lifecycle:

You can find a more complete example at Metadata Plugin Example.