Emission Process

Emission flow diagram

Uploading data to S3 Emission bucket:

  • Data from the state education system will be emitted into the S3 emission data folder using the cQube data emission API.

  • The data emission will happen from the data source location.

  • Data emission will be performed periodically as per the scheduled time interval from different data sources with the help of an automated extraction process.

  • Once the emission process extracts the data fields which are used by cQube, it is converted into csv formatted, pipe delimited files.

  • The CSV data files will then be compressed to zip format

  • Emission process will invoke cQube data-ingestion APIs to emit the data.

  • The API will have different end points

  • Data emission users will request the cQube admin for the emission API token.

  • Data emission users incorporate the API token into the emission process code.

  • The Emission process makes an API call to generate AWS S3, one time preassigned URL.

  • API calls to emit the data files using the https protocol into cQube.

  • API takes the data file as a parameter.

  • The API sends an acknowledgement on successful emission.

  • Create cqube_emission directory and place the data files as shown in file structure below inside the cqube_emission folder.

Static Files

cqube_emission
|
├── district_master
│   └── district_mst.zip
│       └── district_mst.csv
├── block_master
│   └── block_mst.zip
│       └── block_mst.csv
├── cluster_master
│   └── cluster_mst.zip
│       └── cluster_mst.csv
├── school_master
│   └── school_mst.zip
│       └── school_mst.csv
├── school_category
│   └── school_category_master.zip
│       └── school_category_master.csv
├── school_management
│   └── school_management_master.zip
│       └── school_management_master.csv
├── grade_details
│   └── grade_details.zip
│       └── grade_details.csv
├── subject_details
│   └── subject_details.zip
│       └── subject_details.csv
├── pat
│   └── periodic_exam_mst.zip
│       └── periodic_exam_mst.csv
├── pat
│   └── periodic_exam_qst_mst.zip
│       └── periodic_exam_qst_mst.csv
├── sat
│   └── semester_exam_mst.zip
│       └── semester_exam_mst.csv
├── sat
│   └── semester_exam_qst_mst.zip
│       └── semester_exam_qst_mst.csv
├── sat
│   └── semester_exam_subject_details.zip
│       └── semester_exam_subject_details.csv
├── diksha
│   └── diksha_tpd_mapping.zip
│       └── diksha_tpd_mapping.csv
├── diksha
│   └── diksha_api_progress_exhaust_batch_ids.zip
│       └── diksha_api_progress_exhaust_batch_ids.csv
├── diksha_enrolment
│   └── diksha_enrolment.zip
│       └── diksha_enrolment.csv
├── diksha_enrolment
│   └── diksha_etb_enrolment.zip
│       └── diksha_etb_enrolment.csv

Transaction Files

cqube_emission
|
├── student_attendance
│   └── student_attendance.zip
│       └── student_attendance.csv
├── teacher_attendance
│   └── teacher_attendance.zip
│       └── teacher_attendance.csv
├── user_location_master
│   └── user_location_master.zip
│       └── user_location_master.csv
├── inspection_master
│   └── inspection_master.zip
│       └── inspection_master.csv
├── infra_trans
│   └── infra_trans.zip
│       └── infra_trans.csv
├── diksha
│   └── diksha_data_summary.zip
│       └── diksha_data_summary.csv
├── diksha
│   └── diksha_tpd.zip
│       └── diksha_tpd.csv
├── pat
│   └── periodic_exam_result_trans.zip
│       └── periodic_exam_result_trans.csv
├── sat
│   └── semester_exam_result_trans.zip
│       └── semester_exam_result_trans.csv

UDISE Files

├── udise
│   └── sch_facility.zip
│       └── sch_facility.csv
├── udise
│   └── nsqf_plcmnt_c12.zip
│       └── nsqf_plcmnt_c12.csv
├── udise
│   └── nsqf_plcmnt_c10.zip
│       └── nsqf_plcmnt_c10.csv
├── udise
│   └── nsqf_class_cond.zip
│       └── nsqf_class_cond.csv
├── udise
│   └── sch_exmres_c12.zip
│       └── sch_exmres_c12.csv
├── udise
│   └── sch_exmres_c10.zip
│       └── sch_exmres_c10.csv
├── udise
│   └── sch_exmres_c5.zip
│       └── sch_exmres_c5.csv
├── udise
│   └── sch_incen_cwsn.zip
│       └── sch_incen_cwsn.csv
├── udise
│   └── sch_enr_by_stream.zip
│       └── sch_enr_by_stream.csv
├── udise
│   └── sch_enr_cwsn.zip
│       └── sch_enr_cwsn.csv
├── udise
│   └── sch_enr_medinstr.zip
│       └── sch_enr_medinstr.csv
├── udise
│   └── sch_enr_age.zip
│       └── sch_enr_age.csv
├── udise
│   └── sch_enr_newadm.zip
│       └── sch_enr_newadm.csv
├── udise
│   └── sch_enr_reptr.zip
│       └── sch_enr_reptr.csv
├── udise
│   └── sch_enr_fresh.zip
│       └── sch_enr_fresh.csv
├── udise
│   └── sch_staff_posn.zip
│       └── sch_staff_posn.csv
├── udise
│   └── sch_exmmarks_c10.zip
│       └── sch_exmmarks_c10.csv
├── udise
│   └── sch_exmmarks_c12.zip
│       └── sch_exmmarks_c12.csv
├── udise
│   └── nsqf_exmres_c10.zip
│       └── nsqf_exmres_c10.csv
├── udise
│   └── nsqf_exmres_c12.zip
│       └── nsqf_exmres_c12.csv
├── udise
│   └── nsqf_trng_prov.zip
│       └── nsqf_trng_prov.csv
├── udise
│   └── nsqf_faculty.zip
│       └── nsqf_faculty.csv
├── udise
│   └── sch_exmres_c8.zip
│       └── sch_exmres_c8.csv
├── udise
│   └── sch_profile.zip
│       └── sch_profile.csv
├── udise
│   └── tch_profile.zip
│       └── tch_profile.csv
├── udise
│   └── sch_recp_exp.zip
│       └── sch_recp_exp.csv
├── udise
│   └── nsqf_basic_info.zip
│       └── nsqf_basic_info.csv
├── udise
│   └── nsqf_enr_caste.zip
│       └── nsqf_enr_caste.csv
├── udise
│   └── nsqf_enr_sub_sec.zip
│       └── nsqf_enr_sub_sec.csv
├── udise
│   └── sch_pgi_details.zip
│       └── sch_pgi_details.csv
├── udise
│   └── sch_safety.zip
│       └── sch_safety.csv
├── udise
│   └── sch_incentives.zip
│       └── sch_incentives.csv
  • The example spec for the emission API will be as like below

Emission file naming conventions & structure:

  • All the files should be in a CSV format with PIPE("|") separated.

  • The data files should be emitted individually in zip format.

  • The list of columns which have to be emitted for a data source are provided in the file.

  • The header should be in the same order as described in the file, for that respective data set.

  • Data emission overview of data flow can be viewed here

  • Below is the list of data sources used in various reports.

API headers

{
   "Authorization":"Bearer access_token",
   "Content-Type":"application/json"
}

List Buckets Endpoint

GET https://cqube.tibilprojects.com/data/list_s3_buckets

Sample Output:

{
   "input":"cqube-qa10-input",
   "output":"cqube-qa10-output",
   "emission":"cqube-qa10-emission"
}

List S3 Files Endpoint

POST https://cqube.tibilprojects.com/data/list_s3_files

Request Body:

{
   "bucket":"cqube-qa10-emission"
}

S3 Preassigned Download Endpoint

POST https://cqube.tibilprojects.com/data/download_uri

Request Body:

{
   "filename":"school_master/2020/2020-06/2020-06-04_school_master/04-06-2020_13:12:59.574_5e160862-c5b3-4121-9a96-ecefa34fc264_school_mst.zip",
   "bucket":"cqube-gj-input"
}
  • After getting the emission access token from admin console, update the details mentioned below in cQube_Workflow/development/python/client/config.py.

    • Emission access token

    • File location of the cqube_emission directory where the files are placed as below. Example:/home/ubuntu/cqube_emission/

    • Emission End point i.e., emission_url Ex:https://<cqube-domain>/data Note: URL depends upon the server configured in firewall which includes SSL and reverse proxy location)

  • After completing the configuration. Save and close the file.

Emission order & instructions:

  • Static files as defined in above in this page need to be emitted first.

  • For the CRC report, the inspection_master needs to be emitted prior to user_location_master to visualize the CRC reports.

  • For PAT (Periodic assessment report), periodic_exam_mst and periodic_exam_qst_mst needs to be emitted prior to periodic_exam_result_trans

  • school_grade_enrolment file can be emitted either as periodic_exam_grade_details or semester_exam_grade_details. The data will be updated to school_grade_enrolment and will be used in school_hierarchy_details. The count from school_hierarchy details will be used in pat & sat reports. Before the sat & pat workflow the school_grade_enrolment needs to be emitted.

  • Execute the client.py file located in cQube_Workflow/development/python/client/ directory, as mentioned below to emit the data files to s3_emission bucket.

python3 client.py
  • Finally see the output in https://<cqube_domain>/

Last updated