Incremental loads

The Postgres data source collects all the records from the selected table by default. However, you can set the data source to collect data incrementally. That depends on the actual source's table, but when applicable, it speeds up the ingestion process as it only ingests fewer records every time the data source runs. For an in-depth explanation of this feature, please read the Incremental Key article of our documentation.

How to set up incremental loads:

Set the incremental key and value under Advanced in the data source's configuration page.

The incremental key should be set correctly, as it's named in the source table. For example, if the field name is lastUpdate, it should be entered exactly like that with the upper case U.

Each data source can have a single incremental key and a single value. If you don't provide a starting value, the data source will ingest all the existing data in the source table. Then after a data collection, Panoply can determine and track the maximum value for the incremental key. This value will then be used as the starting point for the succeeding run or ingestion.

Every Postgres data source that you set to work with an incremental load should only have a single table selected in it. Otherwise, collecting data from multiple tables might cause missing records in Panoply's side.