Emission Process

Emission flow diagram

Uploading data to S3 Emission bucket:

• Data from the state education system will be emitted into the S3 emission data folder using the cQube data emission API.

• The data emission will happen from the data source location.

• Data emission will be performed periodically as per the scheduled time interval from different data sources with the help of an automated extraction process.

• Once the emission process extracts the data fields which are used by cQube, it is converted into csv formatted, pipe delimited files.

• The CSV data files will then be compressed to zip format

• Emission process will invoke cQube data-ingestion APIs to emit the data.

• The API will have different end points

• Data emission users will request the cQube admin for the emission API token.

• Data emission users incorporate the API token into the emission process code.

• The Emission process makes an API call to generate AWS S3, one time preassigned URL.

• API calls to emit the data files using the https protocol into cQube.

• API takes the data file as a parameter.

• The API sends an acknowledgement on successful emission.

• Create cqube_emission directory and place the data files as shown in file structure below inside the cqube_emission folder.

Static Files

cqube_emission
|
├── district_master
│   └── district_mst.zip
│       └── district_mst.csv
├── block_master
│   └── block_mst.zip
│       └── block_mst.csv
├── cluster_master
│   └── cluster_mst.zip
│       └── cluster_mst.csv
├── school_master
│   └── school_mst.zip
│       └── school_mst.csv
├── school_category
│   └── school_category_master.zip
│       └── school_category_master.csv
├── school_management
│   └── school_management_master.zip
│       └── school_management_master.csv
├── grade_details
│   └── grade_details.zip
│       └── grade_details.csv
├── subject_details
│   └── subject_details.zip
│       └── subject_details.csv
├── pat
│   └── periodic_exam_mst.zip
│       └── periodic_exam_mst.csv
├── pat
│   └── periodic_exam_qst_mst.zip
│       └── periodic_exam_qst_mst.csv
├── sat
│   └── semester_exam_mst.zip
│       └── semester_exam_mst.csv
├── sat
│   └── semester_exam_qst_mst.zip
│       └── semester_exam_qst_mst.csv
├── sat
│   └── semester_exam_subject_details.zip
│       └── semester_exam_subject_details.csv
├── diksha
│   └── diksha_tpd_mapping.zip
│       └── diksha_tpd_mapping.csv
├── diksha
│   └── diksha_api_progress_exhaust_batch_ids.zip
│       └── diksha_api_progress_exhaust_batch_ids.csv
├── diksha_enrolment
│   └── diksha_enrolment.zip
│       └── diksha_enrolment.csv
├── diksha_enrolment
│   └── diksha_etb_enrolment.zip
│       └── diksha_etb_enrolment.csv

Transaction Files

cqube_emission
|
├── student_attendance
│   └── student_attendance.zip
│       └── student_attendance.csv
├── teacher_attendance
│   └── teacher_attendance.zip
│       └── teacher_attendance.csv
├── user_location_master
│   └── user_location_master.zip
│       └── user_location_master.csv
├── inspection_master
│   └── inspection_master.zip
│       └── inspection_master.csv
├── infra_trans
│   └── infra_trans.zip
│       └── infra_trans.csv
├── diksha
│   └── diksha_data_summary.zip
│       └── diksha_data_summary.csv
├── diksha
│   └── diksha_tpd.zip
│       └── diksha_tpd.csv
├── pat
│   └── periodic_exam_result_trans.zip
│       └── periodic_exam_result_trans.csv
├── sat
│   └── semester_exam_result_trans.zip
│       └── semester_exam_result_trans.csv

UDISE Files

├── udise
│   └── sch_facility.zip
│       └── sch_facility.csv
├── udise
│   └── nsqf_plcmnt_c12.zip
│       └── nsqf_plcmnt_c12.csv
├── udise
│   └── nsqf_plcmnt_c10.zip
│       └── nsqf_plcmnt_c10.csv
├── udise
│   └── nsqf_class_cond.zip
│       └── nsqf_class_cond.csv
├── udise
│   └── sch_exmres_c12.zip
│       └── sch_exmres_c12.csv
├── udise
│   └── sch_exmres_c10.zip
│       └── sch_exmres_c10.csv
├── udise
│   └── sch_exmres_c5.zip
│       └── sch_exmres_c5.csv
├── udise
│   └── sch_incen_cwsn.zip
│       └── sch_incen_cwsn.csv
├── udise
│   └── sch_enr_by_stream.zip
│       └── sch_enr_by_stream.csv
├── udise
│   └── sch_enr_cwsn.zip
│       └── sch_enr_cwsn.csv
├── udise
│   └── sch_enr_medinstr.zip
│       └── sch_enr_medinstr.csv
├── udise
│   └── sch_enr_age.zip
│       └── sch_enr_age.csv
├── udise
│   └── sch_enr_newadm.zip
│       └── sch_enr_newadm.csv
├── udise
│   └── sch_enr_reptr.zip
│       └── sch_enr_reptr.csv
├── udise
│   └── sch_enr_fresh.zip
│       └── sch_enr_fresh.csv
├── udise
│   └── sch_staff_posn.zip
│       └── sch_staff_posn.csv
├── udise
│   └── sch_exmmarks_c10.zip
│       └── sch_exmmarks_c10.csv
├── udise
│   └── sch_exmmarks_c12.zip
│       └── sch_exmmarks_c12.csv
├── udise
│   └── nsqf_exmres_c10.zip
│       └── nsqf_exmres_c10.csv
├── udise
│   └── nsqf_exmres_c12.zip
│       └── nsqf_exmres_c12.csv
├── udise
│   └── nsqf_trng_prov.zip
│       └── nsqf_trng_prov.csv
├── udise
│   └── nsqf_faculty.zip
│       └── nsqf_faculty.csv
├── udise
│   └── sch_exmres_c8.zip
│       └── sch_exmres_c8.csv
├── udise
│   └── sch_profile.zip
│       └── sch_profile.csv
├── udise
│   └── tch_profile.zip
│       └── tch_profile.csv
├── udise
│   └── sch_recp_exp.zip
│       └── sch_recp_exp.csv
├── udise
│   └── nsqf_basic_info.zip
│       └── nsqf_basic_info.csv
├── udise
│   └── nsqf_enr_caste.zip
│       └── nsqf_enr_caste.csv
├── udise
│   └── nsqf_enr_sub_sec.zip
│       └── nsqf_enr_sub_sec.csv
├── udise
│   └── sch_pgi_details.zip
│       └── sch_pgi_details.csv
├── udise
│   └── sch_safety.zip
│       └── sch_safety.csv
├── udise
│   └── sch_incentives.zip
│       └── sch_incentives.csv

• The example spec for the emission API will be as like below

Emission file naming conventions & structure:

• All the files should be in a CSV format with PIPE("|") separated.

• The data files should be emitted individually in zip format.

• The list of columns which have to be emitted for a data source are provided in the file.

• The header should be in the same order as described in the file, for that respective data set.

• Data emission overview of data flow can be viewed here

• Below is the list of data sources used in various reports.

Source Name	Used in report	cQube Data Source Name
District Master	All reports	district_master
Block Master	All reports	block_master
Cluster Master	All reports	cluster_master
School Master	All reports	school_master
CRC Inspection Master	CRC Report	inspection_master
CRC User Location Master	CRC Report	user_location_master
Student Attendance	SAR Report	student_attendance
Infrastructure Master	Infrastructure Report	infra_trans
Semester	SEMESTER Report	semester
Diksha	DIKSHA	diksha
School Facility	UDISE	sch_facility
NSQF Class 12 Placement	UDISE	nsqf_plcmnt_c12
NSQF Class 10 Placement	UDISE	nsqf_plcmnt_c10
NSQF Class Condition	UDISE	nsqf_class_cond
School Exam Result Class 12	UDISE	sch_exmres_c12
School Exam Result Class 10	UDISE	sch_exmres_c10
School Exam Result Class 5	UDISE	sch_exmres_c5
School Incentives CWSN	UDISE	sch_incen_cwsn
School Enrollment by stream	UDISE	sch_enr_by_stream
School Enrollment by CWSN	UDISE	sch_enr_cwsn
School Enrollment by Medium of Instruction	UDISE	sch_enr_medinstr
School Enrollment by Age	UDISE	sch_enr_age
School Enrollment by Report	UDISE	sch_enr_reptr
School Enrollment by Fresh	UDISE	sch_enr_fresh
School Enrollment by New Admission	UDISE	sch_enr_newadm
School staff position	UDISE	sch_staff_posn
School Exam Class 10 marks	UDISE	sch_exmmarks_c10
School Exam Class 12 marks	UDISE	sch_exmmarks_c12
NSQF Exam Class 10 Results	UDISE	nsqf_exmres_c10
NSQF Exam Class 10 Results	UDISE	nsqf_exmres_c12
NSQF Training Provision	UDISE	nsqf_trng_prov
NSQF Faculty	UDISE	nsqf_faculty
School Exam class 8 results	UDISE	sch_exmres_c8
School Profile	UDISE	sch_profile
Teacher Profile	UDISE	tch_profile
School Receipt Expenditure	UDISE	sch_recp_exp
NSQF Basic information	UDISE	nsqf_basic_info
NSQF Enrollment by Caste	UDISE	nsqf_enr_caste
NSQF Enrollment by Sub section	UDISE	nsqf_enr_sub_sec
School PGI Indicators	UDISE	sch_pgi_details
School Safety	UDISE	sch_safety
School Incentives	UDISE	sch_incentives
Periodic Exam Master	Periodic Exams	periodic_exam_mst
Periodic Exam Question Master	Periodic Exams	periodic_exam_qst_mst
Periodic Exam Results	Periodic Exams	periodic_exam_result_trans
Semester Exam Master	Semester Exams	semester_exam_mst
Semester Exam Question Master	Semester Exams	semester_exam_qst_mst
Semester Exam Results	Semester Exams	semester_exam_result_trans

API headers

{
   "Authorization":"Bearer access_token",
   "Content-Type":"application/json"
}

List Buckets Endpoint

GET https://cqube.tibilprojects.com/data/list_s3_buckets

Sample Output:

{
   "input":"cqube-qa10-input",
   "output":"cqube-qa10-output",
   "emission":"cqube-qa10-emission"
}

List S3 Files Endpoint

POST https://cqube.tibilprojects.com/data/list_s3_files

Request Body:

{
   "bucket":"cqube-qa10-emission"
}

S3 Preassigned Download Endpoint

POST https://cqube.tibilprojects.com/data/download_uri

Request Body:

{
   "filename":"school_master/2020/2020-06/2020-06-04_school_master/04-06-2020_13:12:59.574_5e160862-c5b3-4121-9a96-ecefa34fc264_school_mst.zip",
   "bucket":"cqube-gj-input"
}

• After getting the emission access token from admin console, update the details mentioned below in cQube_Workflow/development/python/client/config.py.

  • Emission access token

  • File location of the cqube_emission directory where the files are placed as below. Example:/home/ubuntu/cqube_emission/

  • Emission End point i.e., emission_url Ex:https://<cqube-domain>/data Note: URL depends upon the server configured in firewall which includes SSL and reverse proxy location)

• After completing the configuration. Save and close the file.

Emission order & instructions:

• Static files as defined in above in this page need to be emitted first.

• For the CRC report, the inspection_master needs to be emitted prior to user_location_master to visualize the CRC reports.

• For PAT (Periodic assessment report), periodic_exam_mst and periodic_exam_qst_mst needs to be emitted prior to periodic_exam_result_trans

• school_grade_enrolment file can be emitted either as periodic_exam_grade_details or semester_exam_grade_details. The data will be updated to school_grade_enrolment and will be used in school_hierarchy_details. The count from school_hierarchy details will be used in pat & sat reports. Before the sat & pat workflow the school_grade_enrolment needs to be emitted.

• Execute the client.py file located in cQube_Workflow/development/python/client/ directory, as mentioned below to emit the data files to s3_emission bucket.

python3 client.py

• Finally see the output in https://<cqube_domain>/