S3 Partitioning

S3 bucket partitioning

S3 buckets will contain partitions for the data files to store. The partitions are created at the S3 input bucket & the S3 Output bucket.

S3 emission bucket partitions

• All the files in the S3 emission bucket will be in the CSV format.

• S3 emission bucket follows the folder hierarchy based on the data sources.

S3 -> Bucket name -> Data source -> emitted zip files with timestamp

• The emitted zip file contains the CSV data files with timestamp and a manifest file with a timestamp.

• The folders and the files will be removed from the S3 emission bucket once NIFI copies the data.

• Unprocessed data files will remain in the S3 emission bucket for one week and then they will be deleted automatically at the end of the week.

S3 input bucket partitions

• All the files in the S3 input bucket will be in the CSV format.

• S3 input bucket follows a hierarchical partitioning based on the Data source, Year, Month, date and timestamp

S3 -> Bucket name -> Data source -> Year -> Year - Month -> date_Source name

Example for the S3 input bucket:

S3/cqube-gj-input/student_attendance/2020/2020-05/2020-05-29_student_attendance

S3 output bucket partitions

• All the files in the S3 Output bucket will be in the JSON format.

• S3 output bucket follows the hierarchical partitioning based on the data source, Year, Month, date and timestamp, similar to the partitioning that the S3 input bucket follows.

• Metadata files will have information of the latest updated output files which helps cQube to consider the latest output file during the visualization stage.