Collecting specific folders and files

In Panoply's S3 data source, you can set the address to point to an S3 bucket or to a specific file or folder. You can also use a prefix or a wildcard character to filter out the particular files or folders you need. The screenshots below show examples of these options:

• Bucket name - Returns all files and folders within a bucket.

  Example: s3://samplebucket-8

  
 
• Folder - Returns all files and folders under a folder. Take note of the trailing slash at the end because it makes the filtering specific to a folder. If you don't add it, this option acts similarly to using a prefix (see the Prefix item in this list).
  

  Example: s3://samplebucket-8/dc/

  
 
• Prefix - Returns the files and folders whose name starts with the prefix.

  Example: s3://samplebucket-8/employee

  
 
• Wildcard * asterisk character - Use it to filter out the files that contain the text you're looking for. This option is valid only for filtering files.
  
  
  • Example #1 - s3://samplebucket-8/*employee*
   
  • Example #2 - s3://samplebucket-8/*.csv
 
• Specific file - Returns a particular file.

  Example: s3://samplebucket-8/dc/villains.jsonl

  

The options presented above are valid, and each one of them returns a different set of files back. What you decide to choose depends on your use case and the distribution of the files in your S3 bucket.