Nifi section

Nifi in cQube is used to create a data flow and automate the processing of data with the help of processor groups. These processor groups are responsible for accessing the data files from the cloud storage and moving it into the processing-ms codebase. Once the data files are moved the commands will be executed which will ingest the data into the database.

Here we have different processor groups to process the data(Nifi canvas). 1. To run adapters.

2. To run all programs.

3. To run program wise.

The Processor groups in nifi can be scheduled using API called scheduled API

1. To run adapters: This processor group is used to run the adapters shell script code.

2. To run all programs: This processor group is used to move files from process_input folder to processing_ms folder according to folder structures. .i.e., (dimensions/,programs/). Then it will run yarn cli ingest(where it will ingest all schemas and grammars to the database) and yarn cli ingest-data (where it will ingest all data files to the database) commands.

3.To run program wise: This processor group is used to move particular program files from process_input folder to processing_ms folder. Then it will run yarn cli ingest and yarn cli ingest-data --filter=’program_name’.(where it will ingest particular program data files).

In order to automate these processor groups we have written REST-APIs. Please look into the below link to understand th Nifi codebase.

Nifi-Rest-api’s code