How can we add only unique rows to the data source?

To ensure that only unique rows are added to your data source's tables, you need to specify a primary key (PK) for them. A primary key is the unique identifier of a row in a table. This guarantees that no two rows in a table can have the same value in the column or columns that you've specified as the PK.

Even if your source data has duplicate rows, Panoply ingests only the last sequential row among them into the destination table. It uses the upsert mechanism to ensure that no duplicates are created.

To specify a PK for your data source's tables, follow these steps:

1. Select and open a data source in the Data Sources page.
2. On the data source card, look to the Advanced section and click Show to expand it.
3. Provide your desired PK value in the Primary Key field. Make sure that it's one of your table's columns or fields. Your changes are then saved automatically.

Lastly, keep these in mind when specifying PK's:

• Always enclose the attributes/fields you want as PK using curly brackets. For example, use {account_no} if you want the account_no field as the PK.

  If you fail to enclose the fields with curly brackets, Panoply won't recognize them as the PK. It will default to using the id column as the PK, assuming your source data has one.

• You can also use multiple fields to create a PK. For example, if you want to use both first_name and last_name fields and with a dash between them, enter this as such: {first_name}-{last_name}.
 
• If you simply want to use the id field of your source data, then just leave the Primary Key field blank. By default, Panoply recognizes that field and uses it as the primary key.