Emission Process
%20(1).png?alt=media)
- Data from the state education system will be emitted into the S3 emission data folder using the cQube data emission API.
- The data emission will happen from the data source location.
- Data emission will be performed periodically as per the scheduled time interval from different data sources with the help of an automated extraction process.
- Once the emission process extracts the data fields which are used by cQube, it is converted into csv formatted, pipe delimited files.
- The CSV data files will then be compressed to zip format
- Emission process will invoke cQube data-ingestion APIs to emit the data.
- The API will have different end points
- Data emission users will request the cQube admin for the emission API token.
- Data emission users incorporate the API token into the emission process code.
- The Emission process makes an API call to generate AWS S3, one time preassigned URL.
- API calls to emit the data files using the https protocol into cQube.
- API takes the data file as a parameter.
- The API sends an acknowledgement on successful emission.
- Create cqube_emission directory and place the data files as shown in file structure below inside the cqube_emission folder.
cqube_emission
|
├── district_master
│ └── district_mst.zip
│ └── district_mst.csv
├── block_master
│ └── block_mst.zip
│ └── block_mst.csv
├── cluster_master
│ └── cluster_mst.zip
│ └── cluster_mst.csv
├── school_master
│ └── school_mst.zip
│ └── school_mst.csv
├── school_category
│ └── school_category_master.zip
│ └── school_category_master.csv
├── school_management
│ └── school_management_master.zip
│ └── school_management_master.csv
├── grade_details
│ └── grade_details.zip
│ └── grade_details.csv
├── subject_details
│ └── subject_details.zip
│ └── subject_details.csv
├── pat
│ └── periodic_exam_mst.zip
│ └── periodic_exam_mst.csv
├── pat
│ └── periodic_exam_qst_mst.zip
│ └── periodic_exam_qst_mst.csv
├── sat
│ └── semester_exam_mst.zip
│ └── semester_exam_mst.csv
├── sat
│ └── semester_exam_qst_mst.zip
│ └── semester_exam_qst_mst.csv
├── sat
│ └── semester_exam_subject_details.zip
│ └── semester_exam_subject_details.csv
├── diksha
│ └── diksha_tpd_mapping.zip
│ └── diksha_tpd_mapping.csv
├── diksha
│ └── diksha_api_progress_exhaust_batch_ids.zip
│ └── diksha_api_progress_exhaust_batch_ids.csv
├── diksha_enrolment
│ └── diksha_enrolment.zip
│ └── diksha_enrolment.csv
├── diksha_enrolment
│ └── diksha_etb_enrolment.zip
│ └── diksha_etb_enrolment.csv
cqube_emission
|
├── student_attendance
│ └── student_attendance.zip
│ └── student_attendance.csv
├── teacher_attendance
│ └── teacher_attendance.zip
│ └── teacher_attendance.csv
├── user_location_master
│ └── user_location_master.zip
│ └── user_location_master.csv
├── inspection_master
│ └── inspection_master.zip
│ └── inspection_master.csv
├── infra_trans
│ └── infra_trans.zip
│ └── infra_trans.csv
├── diksha
│ └── diksha_data_summary.zip
│ └── diksha_data_summary.csv
├── diksha
│ └── diksha_tpd.zip
│ └── diksha_tpd.csv
├── pat
│ └── periodic_exam_result_trans.zip
│ └─ ─ periodic_exam_result_trans.csv
├── sat
│ └── semester_exam_result_trans.zip
│ └── semester_exam_result_trans.csv
├── udise
│ └── sch_facility.zip
│ └── sch_facility.csv
├── udise
│ └── nsqf_plcmnt_c12.zip
│ └── nsqf_plcmnt_c12.csv
├── udise
│ └── nsqf_plcmnt_c10.zip
│ └── nsqf_plcmnt_c10.csv
├── udise
│ └── nsqf_class_cond.zip
│ └── nsqf_class_cond.csv
├── udise
│ └── sch_exmres_c12.zip
│ └── sch_exmres_c12.csv
├── udise
│ └── sch_exmres_c10.zip
│ └── sch_exmres_c10.csv
├── udise
│ └── sch_exmres_c5.zip
│ └── sch_exmres_c5.csv
├── udise
│ └── sch_incen_cwsn.zip
│ └── sch_incen_cwsn.csv
├── udise
│ └── sch_enr_by_stream.zip
│ └── sch_enr_by_stream.csv
├── udise
│ └── sch_enr_cwsn.zip
│ └── sch_enr_cwsn.csv
├── udise
│ └── sch_enr_medinstr.zip
│ └── sch_enr_medinstr.csv
├── udise
│ └── sch_enr_age.zip
│ └── sch_enr_age.csv
├── udise
│ └── sch_enr_newadm.zip
│ └── sch_enr_newadm.csv
├── udise
│ └── sch_enr_reptr.zip
│ └── sch_enr_reptr.csv
├── udise
│ └── sch_enr_fresh.zip
│ └── sch_enr_fresh.csv
├── udise
│ └── sch_staff_posn.zip
│ └── sch_staff_posn.csv
├── udise
│ └── sch_exmmarks_c10.zip
│ └── sch_exmmarks_c10.csv
├── udise
│ └── sch_exmmarks_c12.zip
│ └── sch_exmmarks_c12.csv
├── udise
│ └── nsqf_exmres_c10.zip
│ └── nsqf_exmres_c10.csv
├── udise
│ └── nsqf_exmres_c12.zip
│ └── nsqf_exmres_c12.csv
├── udise
│ └── nsqf_trng_prov.zip
│ └── nsqf_trng_prov.csv
├── udise
│ └── nsqf_faculty.zip
│ └── nsqf_faculty.csv
├── udise
│ └── sch_exmres_c8.zip
│ └── sch_exmres_c8.csv
├── udise
│ └── sch_profile.zip
│ └── sch_profile.csv
├── udise
│ └── tch_profile.zip
│ └── tch_profile.csv
├── udise
│ └── sch_recp_exp.zip
│ └── sch_recp_exp.csv
├── udise
│ └── nsqf_basic_info.zip
│ └── nsqf_basic_info.csv
├── udise
│ └── nsqf_enr_caste.zip
│ └── nsqf_enr_caste.csv
├── udise
│ └── nsqf_enr_sub_sec.zip
│ └── nsqf_enr_sub_sec.csv
├── udise
│ └── sch_pgi_details.zip
│ └── sch_pgi_details.csv
├── udise
│ └── sch_safety.zip
│ └── sch_safety.csv
├── udise
│ └── sch_incentives.zip
│ └── sch_incentives.csv
- The example spec for the emission API will be as like below
- All the files should be in a CSV format with PIPE("|") separated.
- The data files should be emitted individually in zip format.
- Below is the list of data sources used in various reports.
Source Name | Used in report | cQube Data Source Name |
|---|---|---|
District Master | All reports | district_master |
Block Master | All reports | block_master |
Cluster Master | All reports | cluster_master |
School Master | All reports | school_master |
CRC Inspection Master | CRC Report | inspection_master |
CRC User Location Master | CRC Report | user_location_master |
Student Attendance | SAR Report | student_attendance |
Infrastructure Master | Infrastructure Report | infra_trans |
Semester | SEMESTER Report | semester |
Diksha | DIKSHA | diksha |
School Facility | UDISE | sch_facility |
NSQF Class 12 Placement | UDISE | nsqf_plcmnt_c12 |
NSQF Class 10 Placement | UDISE | nsqf_plcmnt_c10 |
NSQF Class Condition | UDISE | nsqf_class_cond |
School Exam Result Class 12 | UDISE | sch_exmres_c12 |
School Exam Result Class 10 | UDISE | sch_exmres_c10 |
School Exam Result Class 5 | UDISE | sch_exmres_c5 |
School Incentives CWSN | UDISE | sch_incen_cwsn |
School Enrollment by stream | UDISE | sch_enr_by_stream |
School Enrollment by CWSN | UDISE | sch_enr_cwsn |
School Enrollment by Medium of Instruction | UDISE | sch_enr_medinstr |
School Enrollment by Age | UDISE | sch_enr_age |
School Enrollment by Report | UDISE | sch_enr_reptr |
School Enrollment by Fresh | UDISE | sch_enr_fresh |
School Enrollment by New Admission | UDISE | sch_enr_newadm |
School staff position | UDISE | sch_staff_posn |
School Exam Class 10 marks | UDISE | sch_exmmarks_c10 |
School Exam Class 12 marks | UDISE | sch_exmmarks_c12 |
NSQF Exam Class 10 Results | UDISE | nsqf_exmres_c10 |
NSQF Exam Class 10 Results | UDISE | nsqf_exmres_c12 |
NSQF Training Provision | UDISE | nsqf_trng_prov |
NSQF Faculty | UDISE | nsqf_faculty |
School Exam class 8 results | UDISE | sch_exmres_c8 |
School Profile | UDISE | sch_profile |
Teacher Profile | UDISE | tch_profile |
School Receipt Expenditure | UDISE | sch_recp_exp |
NSQF Basic information | UDISE | nsqf_basic_info |
NSQF Enrollment by Caste | UDISE | nsqf_enr_caste |
NSQF Enrollment by Sub section | UDISE | nsqf_enr_sub_sec |
School PGI Indicators | UDISE | sch_pgi_details |
School Safety | UDISE | sch_safety |
School Incentives | UDISE | sch_incentives |
Periodic Exam Master | Periodic Exams | periodic_exam_mst |
Periodic Exam Question Master | Periodic Exams | periodic_exam_qst_mst |
Periodic Exam Results | Periodic Exams | periodic_exam_result_trans |
Semester Exam Master | Semester Exams | semester_exam_mst |
Semester Exam Question Master | Semester Exams | semester_exam_qst_mst |
Semester Exam Results | Semester Exams | semester_exam_result_trans |
API headers
{
"Authorization":"Bearer access_token",
"Content-Type":"application/json"
}
List Buckets Endpoint
Sample Output:
{
"input":"cqube-qa10-input",
"output":"cqube-qa10-output",
"emission":"cqube-qa10-emission"
}
List S3 Files Endpoint
Request Body:
{
"bucket":"cqube-qa10-emission"
}
S3 Preassigned Download Endpoint
Request Body:
{
"filename":"school_master/2020/2020-06/2020-06-04_school_master/04-06-2020_13:12:59.574_5e160862-c5b3-4121-9a96-ecefa34fc264_school_mst.zip",
"bucket":"cqube-gj-input"
}
- After getting the emission access token from admin console, update the details mentioned below in cQube_Workflow/development/python/client/config.py.
- Emission access token
- File location of the cqube_emission directory where the files are placed as below. Example:
/home/ubuntu/cqube_emission/ - Emission End point i.e., emission_url Ex:
https://<cqube-domain>/dataNote: URL depends upon the server configured in firewall which includes SSL and reverse proxy location)
- After completing the configuration. Save and close the file.
- Static files as defined in above in this page need to be emitted first.
- For the CRC report, the inspection_master needs to be emitted prior to user_location_master to visualize the CRC reports.
- For PAT (Periodic assessment report), periodic_exam_mst and periodic_exam_qst_mst needs to be emitted prior to periodic_exam_result_trans
- school_grade_enrolment file can be emitted either as periodic_exam_grade_details or semester_exam_grade_details. The data will be updated to school_grade_enrolment and will be used in school_hierarchy_details. The count from school_hierarchy details will be used in pat & sat reports. Before the sat & pat workflow the school_grade_enrolment needs to be emitted.
- Execute the client.py file located in
cQube_Workflow/development/python/client/directory, as mentioned below to emit the data files to s3_emission bucket.
python3 client.py
- Finally see the output in
https://<cqube_domain>/