Unique Tool

The Unique tool distinguishes whether a data record is unique or a duplicate by grouping on one or more specified fields, then sorting on those fields. The first record in each group is sent to the Unique output stream while the remaining records are sent to the Duplicate output stream.

Example: Input looks like

Field 1 Field Unique

A

1

B

1

C

1

D

2

E

2

F

2

G

3

H

4

I

5

Joining to a Unique Tool and Specifying "Field Unique" produces the following through the unique stream:

Field 1 Field Unique

A

1

D

2

G

3

H

4

I

5

And the following is produced through the duplicate stream:

Field 1 Field Unique

B

1

C

1

E

2

F

2

 

Configuration Properties

Specify the field or fields to group by to determine the record's unique or duplicate status by checking the appropriate fields.

The data is sorted based on the Unique fields. Therefore if there is a specific sort order desired, use the Sort Tool to assign the specific sort order of the file prior to using the Unique tool.