The Incremental Key option for database data sources

All of Panoply's database data sources provide an option for collecting data incrementally. This technique of collecting data is done through the use of an incremental key (see accompanying link for an in-depth explanation). The key can be any field in the source data.

Set the key in the Incremental Key field, under the Advanced options of a data source's configuration page:

Our Incremental loads articles for both PostgreSQL and MySQL data sources provides a detailed example of how to set an incremental key.

Keep these things in mind when using an incremental key:

1. Enter exactly the name of the key as it appears in your source data. And do not wrap it in curly brackets as you would do in primary keys and destination tables.
   
   For example, if you have a lastUpdate field in your source data, enter it as such:
   
 
2. Prefer using a timestamp or integer field as an incremental key. Unlike other data types, they're easily updated or incremented.
 
3. Select only a single table to import from. Avoid collect collecting data from multiple tables because it will cause missing records in Panoply's side.