Sunbird cQube
Search…
Emission Process

Emission flow diagram

Uploading data to S3 Emission bucket:

  • Data from the state education system will be emitted into the S3 emission data folder using the cQube data emission API.
  • The data emission will happen from the data source location.
  • Data emission will be performed periodically as per the scheduled time interval from different data sources with the help of an automated extraction process.
  • Once the emission process extracts the data fields which are used by cQube, it is converted into csv formatted, pipe delimited files.
  • The CSV data files will then be compressed to zip format
  • Emission process will invoke cQube data-ingestion APIs to emit the data.
  • The API will have different end points
  • Data emission users will request the cQube admin for the emission API token.
  • Data emission users incorporate the API token into the emission process code.
  • The Emission process makes an API call to generate AWS S3, one time preassigned URL.
  • API calls to emit the data files using the https protocol into cQube.
  • API takes the data file as a parameter.
  • The API sends an acknowledgement on successful emission.
  • Create cqube_emission directory and place the data files as shown in file structure below inside the cqube_emission folder.

Static Files

1
cqube_emission
2
|
3
├── block_master
4
│ └── block_mst.zip
5
│ └── block_mst.csv
6
├── cluster_master
7
│ └── cluster_mst.zip
8
│ └── cluster_mst.csv
9
├── district_master
10
│ └── district_mst.zip
11
│ └── district_mst.csv
12
├── school_master
13
│ └── school_mst.zip
14
│ └── school_mst.csv
15
├── pat
16
│ └── periodic_exam_mst.zip
17
│ └── periodic_exam_mst.csv
18
├── pat
19
│ └── periodic_exam_qst_mst.zip
20
│ └── periodic_exam_qst_mst.csv
21
├── diksha
22
│ └── diksha_tpd_mapping.zip
23
│ └── diksha_tpd_mapping.csv
24
├── diksha
25
│ └── diksha_api_progress_exhaust_batch_ids.zip
26
│ └── diksha_api_progress_exhaust_batch_ids.csv
27
├── sat
28
│ └── semester_exam_mst.zip
29
│ └── semester_exam_mst.csv
30
├── sat
31
│ └── semester_exam_qst_mst.zip
32
│ └── semester_exam_qst_mst.csv
33
├── sat
34
│ └── semester_exam_subject_details.zip
35
│ └── semester_exam_subject_details.csv
36
├── school_category
37
│ └── school_category_master.zip
38
│ └── school_category_master.csv
39
├── school_management
40
│ └── school_management_master.zip
41
│ └── school_management_master.csv
42
├── sat
43
│ └── semester_exam_subject_details.zip
44
│ └── semester_exam_subject_details.csv
45
├── sat
46
│ └── semester_exam_grade_details.zip
47
│ └── semester_exam_grade_details.csv
48
├── pat
49
│ └── periodic_exam_subject_details.zip
50
│ └── periodic_exam_subject_details.csv
51
├── pat
52
│ └── periodic_exam_grade_details.zip
53
│ └── periodic_exam_grade_details.csv
Copied!

Transaction Files

1
cqube_emission
2
|
3
├── student_attendance
4
│ └── student_attendance.zip
5
│ └── student_attendance.csv
6
├── teacher_attendance
7
│ └── teacher_attendance.zip
8
│ └── teacher_attendance.csv
9
├── user_location_master
10
│ └── user_location_master.zip
11
│ └── user_location_master.csv
12
├── inspection_master
13
│ └── inspection_master.zip
14
│ └── inspection_master.csv
15
├── infra_trans
16
│ └── infra_trans.zip
17
│ └── infra_trans.csv
18
├── diksha
19
│ └── diksha.zip
20
│ └── diksha.csv
21
├── pat
22
│ └── periodic_exam_result_trans.zip
23
│ └── periodic_exam_result_trans.csv
24
├── sat
25
│ └── semester_exam_result_trans.zip
26
│ └── semester_exam_result_trans.csv
Copied!
UDISE Files
1
├── udise
2
│ └── sch_facility.zip
3
│ └── sch_facility.csv
4
├── udise
5
│ └── nsqf_plcmnt_c12.zip
6
│ └── nsqf_plcmnt_c12.csv
7
├── udise
8
│ └── nsqf_plcmnt_c10.zip
9
│ └── nsqf_plcmnt_c10.csv
10
├── udise
11
│ └── nsqf_class_cond.zip
12
│ └── nsqf_class_cond.csv
13
├── udise
14
│ └── sch_exmres_c12.zip
15
│ └── sch_exmres_c12.csv
16
├── udise
17
│ └── sch_exmres_c10.zip
18
│ └── sch_exmres_c10.csv
19
├── udise
20
│ └── sch_exmres_c5.zip
21
│ └── sch_exmres_c5.csv
22
├── udise
23
│ └── sch_incen_cwsn.zip
24
│ └── sch_incen_cwsn.csv
25
├── udise
26
│ └── sch_enr_by_stream.zip
27
│ └── sch_enr_by_stream.csv
28
├── udise
29
│ └── sch_enr_cwsn.zip
30
│ └── sch_enr_cwsn.csv
31
├── udise
32
│ └── sch_enr_medinstr.zip
33
│ └── sch_enr_medinstr.csv
34
├── udise
35
│ └── sch_enr_age.zip
36
│ └── sch_enr_age.csv
37
├── udise
38
│ └── sch_enr_newadm.zip
39
│ └── sch_enr_newadm.csv
40
├── udise
41
│ └── sch_enr_reptr.zip
42
│ └── sch_enr_reptr.csv
43
├── udise
44
│ └── sch_enr_fresh.zip
45
│ └── sch_enr_fresh.csv
46
├── udise
47
│ └── sch_staff_posn.zip
48
│ └── sch_staff_posn.csv
49
├── udise
50
│ └── sch_exmmarks_c10.zip
51
│ └── sch_exmmarks_c10.csv
52
├── udise
53
│ └── sch_exmmarks_c12.zip
54
│ └── sch_exmmarks_c12.csv
55
├── udise
56
│ └── nsqf_exmres_c10.zip
57
│ └── nsqf_exmres_c10.csv
58
├── udise
59
│ └── nsqf_exmres_c12.zip
60
│ └── nsqf_exmres_c12.csv
61
├── udise
62
│ └── nsqf_trng_prov.zip
63
│ └── nsqf_trng_prov.csv
64
├── udise
65
│ └── nsqf_faculty.zip
66
│ └── nsqf_faculty.csv
67
├── udise
68
│ └── sch_exmres_c8.zip
69
│ └── sch_exmres_c8.csv
70
├── udise
71
│ └── sch_profile.zip
72
│ └── sch_profile.csv
73
├── udise
74
│ └── tch_profile.zip
75
│ └── tch_profile.csv
76
├── udise
77
│ └── sch_recp_exp.zip
78
│ └── sch_recp_exp.csv
79
├── udise
80
│ └── nsqf_basic_info.zip
81
│ └── nsqf_basic_info.csv
82
├── udise
83
│ └── nsqf_enr_caste.zip
84
│ └── nsqf_enr_caste.csv
85
├── udise
86
│ └── nsqf_enr_sub_sec.zip
87
│ └── nsqf_enr_sub_sec.csv
88
├── udise
89
│ └── sch_pgi_details.zip
90
│ └── sch_pgi_details.csv
91
├── udise
92
│ └── sch_safety.zip
93
│ └── sch_safety.csv
94
├── udise
95
│ └── sch_incentives.zip
96
│ └── sch_incentives.csv
Copied!
  • The example spec for the emission API will be as like below

Emission file naming conventions & structure:

  • All the files should be in a CSV format with PIPE("|") separated.
  • The data files should be emitted individually in zip format.
  • The list of columns which have to be emitted for a data source are provided in the file.
  • The header should be in the same order as described in the file, for that respective data set.
  • Data emission overview of data flow can be viewed here
  • Below is the list of data sources used in various reports.
Source Name
Used in report
cQube Data Source Name
District Master
All reports
district_master
Block Master
All reports
block_master
Cluster Master
All reports
cluster_master
School Master
All reports
school_master
CRC Inspection Master
CRC Report
inspection_master
CRC User Location Master
CRC Report
user_location_master
Student Attendance
SAR Report
student_attendance
Infrastructure Master
Infrastructure Report
infra_trans
Semester
SEMESTER Report
semester
Diksha
DIKSHA
diksha
School Facility
UDISE
sch_facility
NSQF Class 12 Placement
UDISE
nsqf_plcmnt_c12
NSQF Class 10 Placement
UDISE
nsqf_plcmnt_c10
NSQF Class Condition
UDISE
nsqf_class_cond
School Exam Result Class 12
UDISE
sch_exmres_c12
School Exam Result Class 10
UDISE
sch_exmres_c10
School Exam Result Class 5
UDISE
sch_exmres_c5
School Incentives CWSN
UDISE
sch_incen_cwsn
School Enrollment by stream
UDISE
sch_enr_by_stream
School Enrollment by CWSN
UDISE
sch_enr_cwsn
School Enrollment by Medium of Instruction
UDISE
sch_enr_medinstr
School Enrollment by Age
UDISE
sch_enr_age
School Enrollment by Report
UDISE
sch_enr_reptr
School Enrollment by Fresh
UDISE
sch_enr_fresh
School Enrollment by New Admission
UDISE
sch_enr_newadm
School staff position
UDISE
sch_staff_posn
School Exam Class 10 marks
UDISE
sch_exmmarks_c10
School Exam Class 12 marks
UDISE
sch_exmmarks_c12
NSQF Exam Class 10 Results
UDISE
nsqf_exmres_c10
NSQF Exam Class 10 Results
UDISE
nsqf_exmres_c12
NSQF Training Provision
UDISE
nsqf_trng_prov
NSQF Faculty
UDISE
nsqf_faculty
School Exam class 8 results
UDISE
sch_exmres_c8
School Profile
UDISE
sch_profile
Teacher Profile
UDISE
tch_profile
School Receipt Expenditure
UDISE
sch_recp_exp
NSQF Basic information
UDISE
nsqf_basic_info
NSQF Enrollment by Caste
UDISE
nsqf_enr_caste
NSQF Enrollment by Sub section
UDISE
nsqf_enr_sub_sec
School PGI Indicators
UDISE
sch_pgi_details
School Safety
UDISE
sch_safety
School Incentives
UDISE
sch_incentives
Periodic Exam Master
Periodic Exams
periodic_exam_mst
Periodic Exam Question Master
Periodic Exams
periodic_exam_qst_mst
Periodic Exam Results
Periodic Exams
periodic_exam_result_trans
Semester Exam Master
Semester Exams
semester_exam_mst
Semester Exam Question Master
Semester Exams
semester_exam_qst_mst
Semester Exam Results
Semester Exams
semester_exam_result_trans
API headers
1
{
2
"Authorization":"Bearer access_token",
3
"Content-Type":"application/json"
4
}
Copied!
List Buckets Endpoint
Sample Output:
1
{
2
"input":"cqube-qa10-input",
3
"output":"cqube-qa10-output",
4
"emission":"cqube-qa10-emission"
5
}
Copied!
List S3 Files Endpoint
Request Body:
1
{
2
"bucket":"cqube-qa10-emission"
3
}
Copied!
S3 Preassigned Download Endpoint
Request Body:
1
{
2
"filename":"school_master/2020/2020-06/2020-06-04_school_master/04-06-2020_13:12:59.574_5e160862-c5b3-4121-9a96-ecefa34fc264_school_mst.zip",
3
"bucket":"cqube-gj-input"
4
}
Copied!
  • After getting the emission access token from admin console, update the details mentioned below in cQube_Workflow/development/python/client/config.py.
    • Emission access token
    • File location of the cqube_emission directory where the files are placed as below. Example:/home/ubuntu/cqube_emission/
    • Emission End point i.e., emission_url Ex:https://<cqube-domain>/data Note: URL depends upon the server configured in firewall which includes SSL and reverse proxy location)
  • After completing the configuration. Save and close the file.

Emission order & instructions:

  • Static files as defined in above in this page need to be emitted first.
  • For the CRC report, the inspection_master needs to be emitted prior to user_location_master to visualize the CRC reports.
  • For PAT (Periodic assessment report), periodic_exam_mst and periodic_exam_qst_mst needs to be emitted prior to periodic_exam_result_trans
  • school_grade_enrolment file can be emitted either as periodic_exam_grade_details or semester_exam_grade_details. The data will be updated to school_grade_enrolment and will be used in school_hierarchy_details. The count from school_hierarchy details will be used in pat & sat reports. Before the sat & pat workflow the school_grade_enrolment needs to be emitted.
  • Execute the client.py file located in cQube_Workflow/development/python/client/ directory, as mentioned below to emit the data files to s3_emission bucket.
1
python3 client.py
Copied!
  • Finally see the output in https://<cqube_domain>/
Last modified 1mo ago