Step-wise Ingestion Process
Lists down the steps to ingest data (events, dimensions & datasets) into cQube V 5.0 for state as well as national programs
This focusses on ingestion of programs like student attendance, teacher attendance & review meetings.
Data ingestion can be done using APIs. Please use postman as a tool to run these APIs. If you have already set up postman on your system, you can directly read the document from here.
Note: There could be some common errors in the files which could lead to unsuccessful data ingestion into cQube. Look at these errors here and resolve them before ingesting data into cQube.
- Select the import option in postman to import the collection. Please refer to the screenshot.
The data can be ingested into cQube in two ways:
Events, Dimensions & Datasets can be ingested into cQube via the ingestion APIs.
Step 1: Select the 'csv_import' folder in the postman collection.
Step 2: Open the specified request & add details.
API Endpoint: <domain_name>/ingestion/new_programs
HTTP Method: POST
This API will import the event / dimension / dataset CSV and upload it into the combined_input folder in cloud / local SDC if there are no errors. Then, the adapter in cQube will use the same files to breakdown the combined input into multiple input files. Later, those files will be used by the NiFi processor to process and create datasets into the database.
Step 3: Build the request body with reference to YAML file. The request body for the above API is attached here for yaml: https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml
Provide valid input details for the parameters shown below.
- file : Attach the CSV file for the importing
- ingestion_type : Specify the type of ingestion
- ingestion_name : Name of the event / dimension / dataset (event / dimension / dataset name should be present in the database)
Ingestion of Events
Ingestion of Dimensions
Step 4: Click on the send button for request and if the request is successful the user should see a response message. Please refer to the below screenshot.
Ingestion of Events
Ingestion of Dimensions
After successful execution of the CSV import API, we get the response and we can see the file status indicating if the file is uploaded or not through GET file status API. If the file is successfully uploaded we will get the response as uploaded and if there is any error it will send us the response indicating there was an error in the file.
Events, Dimensions & Datasets can be ingested into cQube via the ingestion APIs.
Step 1: Open the specified request & add the details
API Endpoint:
- 1.<domain_name>/ingestion/event
- 2.<domain_name>/ingestion/dimension
- 3.<domain_name>/ingestion/dataset
HTTP Method: POST
Ingestion of Events
Ingestion of Dimensions
Ingestion of Datasets
This API will be used to write events into the CSV file and upload it in to the combined_input folder in cloud / SDC if there are no errors. Then, the adapter will use the same files to break down the combined input into multiple input files. Later, those files will then be used by the NiFi processor to ingest the data into the database. The API can be used to add individual events into CSV.
Step 2: Build the request body with reference to YAML file. The request body for the above API is linked here: https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml
Provide the valid input details for the parameters as shown below. The request body should conform to the schema stored in the database for the particular event name.
Ingestion of Events
Ingestion of Dimensions
Ingestion of Datasets
Step 3: Click on the send button for the request and if the request is successful the user should see a response message. Please refer to the screenshot.
Ingestion of Events
Ingestion of Dimensions
Ingestion of Datasets
After successful execution of the event / dimension / dataset API, we get the response and the data sent in the request body will be written to the CSV file. Any errors will be written to a CSV file and valid data will be written to a separate CSV file.
This focusses on ingestion of programs like UDISE, PGI, NISHTHA, DIKSHA, NAS, PM-POSHAN.
Note: There could be some common errors in the files which could lead to unsuccessful data ingestion into cQube. Look at these errors here and resolve them before ingesting data into cQube.
The already existing files will need to be ingested into cQube via the following process:
This API accepts the data in a zip file format and adds it to the emission folder in the respective cloud storage / SDC.
Step 1: Open the specified request and add the details.
API Endpoint: <domain_name>/ingestion/national_programs
HTTP Method: POST
Step 2: Build the request body with reference to YAML file. The request body for the above API is attached here for yaml: https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml. Provide the valid input details for the parameters as shown below.
Step 3: Click on the send button for the request and if the request is successful, the user should see a response message as “File uploaded successfully”. The files will be uploaded to the emission folder created in the respective cloud storage / SDC.
The zip file will be extracted and will be read by the adapters and then moved into the input folders.
