The order of ingestion from files

When loading data from a file, Panoply extracts it based on its sequential order in the file. Some files might have "duplicated records" based on the selected primary key. In these cases, Panoply only ingests the last record that it encounters based on the primary key defined on the source. To ensure that Panoply ingests the correct data, you need to make sure that its order in the file is the correct one.

For example, if you have a list of users and their age like so:

If you set the user_id field as the primary key, then the records stored in the Panoply table looks like this:

Note that the age saved for the user with an ID of 1 is 9 instead of 10. This is because that record was the latest or the last sequential one for this specific user.