Sunbird cQube
  • 📖KNOW ABOUT CQUBE
    • What is cQube & what does it solve
      • Business case
      • cQube ed
      • Design Principles
    • cQube adoptions
    • Discuss more about cQube
  • 👍TRY CQUBE
    • cQube on Gitpod
  • 🌅WHAT IS NEXT IN CQUBE
    • cQube Roadmap
  • 💻TECHANICAL OVERVIEW
    • Architecture
    • Design Principles
    • Key Components
    • Details of microservices
  • 👩‍💻Get started on cQube
    • Suggested Team Structure
    • Hardware Requirements
    • Prerequisites Checklist
    • Checking pre existing ports
    • Instance Creation
    • Copying SSL Certificate
  • 🛃USE CQUBE
    • How can I install cQube?
      • High level understanding of deployment
      • Oracle Installation
      • AWS Installation
      • SDC Installation
      • Azure Installation
    • How to prepare the data
      • Schema
      • How to prepare schemas for dimension files
      • How to prepare schemas for event files
      • Examples of dimension & event files
      • All cQube schemas used for VSK
    • Ingesting the data
      • High-level understanding of ingestion
      • Steps to ingest schema
      • Steps to ingest data files
        • Upload the .csv data file for state specific programs using ingestion API
        • API to upload starter pack data files for NVSK programs
        • Get file status API
        • Scheduled API
      • Error during ingestion
        • Error Monitoring
        • Common errors in data files during ingestion
      • Processor group name
    • Adapter details
    • Postman details
    • Processing of data
      • Data Processing using CLI command
      • API Details for Nifi-Rest
      • Nifi section
    • Visualizing the data
      • High level understanding of how visualizations work in cQube
      • Programs and reports out-of-the-box
      • Enhance /Customize cQube
        • Available customizations
          • Changing Dashboard Logos and Headers
          • Changing Program Name, Icon and Side Menu Sequence
          • Adding a new KPI
          • Adding a Map KPI into dashboard ms
          • Table Drill Down Customization
          • Adding a Scatter Plot KPI into dashboard ms
          • Configure default date range across app/specific report
        • How to add a New Indicator
        • How to add a new report in an existing program
        • How to add a new program (end to end)
    • Additional Features
      • Public/Private dashboards
      • Role based access control
      • Saving geographical preferences
      • Admin Panel
        • Data Debugger
        • Schema Generator
        • System Monitoring
    • Adding Users
      • Adding an individual user
      • Adding bulk users
  • 🖥️MONITOR cQUBE
    • Infra health monitoring
    • Usage monitoring
  • 🔎QA testing
    • Testing approaches & activities
    • Manual & Automated testing
    • Functional Testing
      • Smoke Testing
      • Functional tests
      • Regression Testing
      • System Testing
    • Non Functional Testing
      • Performance Testing
        • Load Testing
        • Volume Testing
        • Performance testing results
    • Test for One-Step Installation
    • Test for Ingestion
    • Test for nifi processing
    • Test for UI Application
    • Test for KPIs
  • ☀️DEPLOYMENT PROCESS
    • State List
    • AWS Deployment
    • SDC Deployment
    • Adapter Details During the Processing
  • 🈴UPGRADING TO LATEST VERSION
    • How can I upgrade cQube to the latest release
  • 🆘Common issues and their solutions
    • Deployment & ingestion related issues & their solutions
  • ⏱️Standard Operating Procedure
    • Reporting a Bug
    • Protocol for issue reporting & resolution
    • Suggesting Enhancements
    • Raising a PR
  • ❓Frequently Asked Questions
    • Running List
  • 🧑‍🏫🧑🏫 Recording of trainings
    • Link to the training videos
  • 🧠Key Terms & Concepts
    • Definitions
  • 🚀cQube Release Notes
    • cQube - Release V 5.0.5
    • cQube - Release V 5.0.3
    • cQube - Release V 5.0.2
    • cQube - Release V 5.0.1
    • cQube - Release V 5.0
    • cQube - Release V 4.1-beta
    • cQube - Release V 4.0-beta
    • cQube - Release V 4.0-alpha
    • cQube - Release V 3.7
    • cQube - Release V 3.6
    • cQube - Release V 3.5
    • cQube - Release V 3.4
    • cQube - Release V 3.3
    • cQube - Release V 3.2
    • cQube - Release V 3.1
    • cQube - Release V 3
    • cQube - Release V 2
    • cQube - Release V 1.13 and V 1.13.1
    • cQube - Release V 1.12 and V 1.12.1
    • cQube - Release V 1.11
    • cQube - Release V 10 and V 10.1
    • cQube - Release V 1.9
    • cQube - Release V 1.8 and V 1.8.1
    • cQube - Release Notes V 1.7
    • cQube - Release Notes V 1.6 and V 1.6.1
    • cQube - Release Notes V 1.5
    • cQube - Release Notes V 1.4
    • cQube - Release Notes V 1.3
    • cQube - Release Notes V 1.2 and V 1.2.1
    • cQube - Release Notes V 1.1
    • cQube - Release Notes V 1.0
  • 📂cQube V 4.1 - Beta
    • Sunbird cQube Overview
    • cQube Product Description
    • Listen to Experts (Youtube)
    • Software Requirements
    • Acronyms
    • cQube Software Architecture
    • AWS - Network Architecture
      • Hardware requirements
      • Data Storage Locations
    • Security Implementations
    • Prerequisites for Installation process
    • New Use-Case Creation
    • cQube Setup & configuration
    • Base Installation steps
    • Base Upgradation steps
    • Workflow Installation steps
    • Workflow Upgradation steps
    • Laptop/Desktop Installation
      • Base Installation
      • Workflow Installation
      • Mock Data Processing
    • Ad-hoc analysis
    • Workflow process
    • Emission Process
    • cQube ER Diagrams
    • Data Validation after Ingestion
    • User Authentication Process
    • Admin Login Process
    • Admin Features
    • cQube Datasource Configuration
    • cQube data replay process
    • S3 Partitioning
    • Reports
    • Troubleshooting Issues
      • Data Processing-NIFI Issues
      • Data Processing-PostgreSQL Issues
      • Data Emission Issues
      • Angular & Node Issues
    • FAQs
    • Discuss
    • Report
    • Source Code
Powered by GitBook
On this page
  • Emission flow diagram
  • Uploading data to S3 Emission bucket:
  • Static Files
  • Transaction Files
  • UDISE Files
  • Emission file naming conventions & structure:
  • Emission order & instructions:

Was this helpful?

Edit on GitHub
  1. cQube V 4.1 - Beta

Emission Process

PreviousWorkflow processNextcQube ER Diagrams

Last updated 2 years ago

Was this helpful?

Emission flow diagram

Uploading data to S3 Emission bucket:

  • Data from the state education system will be emitted into the S3 emission data folder using the cQube data emission API.

  • The data emission will happen from the data source location.

  • Data emission will be performed periodically as per the scheduled time interval from different data sources with the help of an automated extraction process.

  • Once the emission process extracts the data fields which are used by cQube, it is converted into csv formatted, pipe delimited files.

  • The CSV data files will then be compressed to zip format

  • Emission process will invoke cQube data-ingestion APIs to emit the data.

  • The API will have different end points

  • Data emission users will request the cQube admin for the emission API token.

  • Data emission users incorporate the API token into the emission process code.

  • The Emission process makes an API call to generate AWS S3, one time preassigned URL.

  • API calls to emit the data files using the https protocol into cQube.

  • API takes the data file as a parameter.

  • The API sends an acknowledgement on successful emission.

  • Create cqube_emission directory and place the data files as shown in file structure below inside the cqube_emission folder.

Static Files

cqube_emission
|
├── district_master
│   └── district_mst.zip
│       └── district_mst.csv
├── block_master
│   └── block_mst.zip
│       └── block_mst.csv
├── cluster_master
│   └── cluster_mst.zip
│       └── cluster_mst.csv
├── school_master
│   └── school_mst.zip
│       └── school_mst.csv
├── school_category
│   └── school_category_master.zip
│       └── school_category_master.csv
├── school_management
│   └── school_management_master.zip
│       └── school_management_master.csv
├── grade_details
│   └── grade_details.zip
│       └── grade_details.csv
├── subject_details
│   └── subject_details.zip
│       └── subject_details.csv
├── pat
│   └── periodic_exam_mst.zip
│       └── periodic_exam_mst.csv
├── pat
│   └── periodic_exam_qst_mst.zip
│       └── periodic_exam_qst_mst.csv
├── sat
│   └── semester_exam_mst.zip
│       └── semester_exam_mst.csv
├── sat
│   └── semester_exam_qst_mst.zip
│       └── semester_exam_qst_mst.csv
├── sat
│   └── semester_exam_subject_details.zip
│       └── semester_exam_subject_details.csv
├── diksha
│   └── diksha_tpd_mapping.zip
│       └── diksha_tpd_mapping.csv
├── diksha
│   └── diksha_api_progress_exhaust_batch_ids.zip
│       └── diksha_api_progress_exhaust_batch_ids.csv
├── diksha_enrolment
│   └── diksha_enrolment.zip
│       └── diksha_enrolment.csv
├── diksha_enrolment
│   └── diksha_etb_enrolment.zip
│       └── diksha_etb_enrolment.csv

Transaction Files

cqube_emission
|
├── student_attendance
│   └── student_attendance.zip
│       └── student_attendance.csv
├── teacher_attendance
│   └── teacher_attendance.zip
│       └── teacher_attendance.csv
├── user_location_master
│   └── user_location_master.zip
│       └── user_location_master.csv
├── inspection_master
│   └── inspection_master.zip
│       └── inspection_master.csv
├── infra_trans
│   └── infra_trans.zip
│       └── infra_trans.csv
├── diksha
│   └── diksha_data_summary.zip
│       └── diksha_data_summary.csv
├── diksha
│   └── diksha_tpd.zip
│       └── diksha_tpd.csv
├── pat
│   └── periodic_exam_result_trans.zip
│       └── periodic_exam_result_trans.csv
├── sat
│   └── semester_exam_result_trans.zip
│       └── semester_exam_result_trans.csv

UDISE Files

├── udise
│   └── sch_facility.zip
│       └── sch_facility.csv
├── udise
│   └── nsqf_plcmnt_c12.zip
│       └── nsqf_plcmnt_c12.csv
├── udise
│   └── nsqf_plcmnt_c10.zip
│       └── nsqf_plcmnt_c10.csv
├── udise
│   └── nsqf_class_cond.zip
│       └── nsqf_class_cond.csv
├── udise
│   └── sch_exmres_c12.zip
│       └── sch_exmres_c12.csv
├── udise
│   └── sch_exmres_c10.zip
│       └── sch_exmres_c10.csv
├── udise
│   └── sch_exmres_c5.zip
│       └── sch_exmres_c5.csv
├── udise
│   └── sch_incen_cwsn.zip
│       └── sch_incen_cwsn.csv
├── udise
│   └── sch_enr_by_stream.zip
│       └── sch_enr_by_stream.csv
├── udise
│   └── sch_enr_cwsn.zip
│       └── sch_enr_cwsn.csv
├── udise
│   └── sch_enr_medinstr.zip
│       └── sch_enr_medinstr.csv
├── udise
│   └── sch_enr_age.zip
│       └── sch_enr_age.csv
├── udise
│   └── sch_enr_newadm.zip
│       └── sch_enr_newadm.csv
├── udise
│   └── sch_enr_reptr.zip
│       └── sch_enr_reptr.csv
├── udise
│   └── sch_enr_fresh.zip
│       └── sch_enr_fresh.csv
├── udise
│   └── sch_staff_posn.zip
│       └── sch_staff_posn.csv
├── udise
│   └── sch_exmmarks_c10.zip
│       └── sch_exmmarks_c10.csv
├── udise
│   └── sch_exmmarks_c12.zip
│       └── sch_exmmarks_c12.csv
├── udise
│   └── nsqf_exmres_c10.zip
│       └── nsqf_exmres_c10.csv
├── udise
│   └── nsqf_exmres_c12.zip
│       └── nsqf_exmres_c12.csv
├── udise
│   └── nsqf_trng_prov.zip
│       └── nsqf_trng_prov.csv
├── udise
│   └── nsqf_faculty.zip
│       └── nsqf_faculty.csv
├── udise
│   └── sch_exmres_c8.zip
│       └── sch_exmres_c8.csv
├── udise
│   └── sch_profile.zip
│       └── sch_profile.csv
├── udise
│   └── tch_profile.zip
│       └── tch_profile.csv
├── udise
│   └── sch_recp_exp.zip
│       └── sch_recp_exp.csv
├── udise
│   └── nsqf_basic_info.zip
│       └── nsqf_basic_info.csv
├── udise
│   └── nsqf_enr_caste.zip
│       └── nsqf_enr_caste.csv
├── udise
│   └── nsqf_enr_sub_sec.zip
│       └── nsqf_enr_sub_sec.csv
├── udise
│   └── sch_pgi_details.zip
│       └── sch_pgi_details.csv
├── udise
│   └── sch_safety.zip
│       └── sch_safety.csv
├── udise
│   └── sch_incentives.zip
│       └── sch_incentives.csv
  • The example spec for the emission API will be as like below

Emission file naming conventions & structure:

  • All the files should be in a CSV format with PIPE("|") separated.

  • The data files should be emitted individually in zip format.

  • Below is the list of data sources used in various reports.

Source Name
Used in report
cQube Data Source Name

District Master

All reports

district_master

Block Master

All reports

block_master

Cluster Master

All reports

cluster_master

School Master

All reports

school_master

CRC Inspection Master

CRC Report

inspection_master

CRC User Location Master

CRC Report

user_location_master

Student Attendance

SAR Report

student_attendance

Infrastructure Master

Infrastructure Report

infra_trans

Semester

SEMESTER Report

semester

Diksha

DIKSHA

diksha

School Facility

UDISE

sch_facility

NSQF Class 12 Placement

UDISE

nsqf_plcmnt_c12

NSQF Class 10 Placement

UDISE

nsqf_plcmnt_c10

NSQF Class Condition

UDISE

nsqf_class_cond

School Exam Result Class 12

UDISE

sch_exmres_c12

School Exam Result Class 10

UDISE

sch_exmres_c10

School Exam Result Class 5

UDISE

sch_exmres_c5

School Incentives CWSN

UDISE

sch_incen_cwsn

School Enrollment by stream

UDISE

sch_enr_by_stream

School Enrollment by CWSN

UDISE

sch_enr_cwsn

School Enrollment by Medium of Instruction

UDISE

sch_enr_medinstr

School Enrollment by Age

UDISE

sch_enr_age

School Enrollment by Report

UDISE

sch_enr_reptr

School Enrollment by Fresh

UDISE

sch_enr_fresh

School Enrollment by New Admission

UDISE

sch_enr_newadm

School staff position

UDISE

sch_staff_posn

School Exam Class 10 marks

UDISE

sch_exmmarks_c10

School Exam Class 12 marks

UDISE

sch_exmmarks_c12

NSQF Exam Class 10 Results

UDISE

nsqf_exmres_c10

NSQF Exam Class 10 Results

UDISE

nsqf_exmres_c12

NSQF Training Provision

UDISE

nsqf_trng_prov

NSQF Faculty

UDISE

nsqf_faculty

School Exam class 8 results

UDISE

sch_exmres_c8

School Profile

UDISE

sch_profile

Teacher Profile

UDISE

tch_profile

School Receipt Expenditure

UDISE

sch_recp_exp

NSQF Basic information

UDISE

nsqf_basic_info

NSQF Enrollment by Caste

UDISE

nsqf_enr_caste

NSQF Enrollment by Sub section

UDISE

nsqf_enr_sub_sec

School PGI Indicators

UDISE

sch_pgi_details

School Safety

UDISE

sch_safety

School Incentives

UDISE

sch_incentives

Periodic Exam Master

Periodic Exams

periodic_exam_mst

Periodic Exam Question Master

Periodic Exams

periodic_exam_qst_mst

Periodic Exam Results

Periodic Exams

periodic_exam_result_trans

Semester Exam Master

Semester Exams

semester_exam_mst

Semester Exam Question Master

Semester Exams

semester_exam_qst_mst

Semester Exam Results

Semester Exams

semester_exam_result_trans

API headers

{
   "Authorization":"Bearer access_token",
   "Content-Type":"application/json"
}

List Buckets Endpoint

Sample Output:

{
   "input":"cqube-qa10-input",
   "output":"cqube-qa10-output",
   "emission":"cqube-qa10-emission"
}

List S3 Files Endpoint

Request Body:

{
   "bucket":"cqube-qa10-emission"
}

S3 Preassigned Download Endpoint

Request Body:

{
   "filename":"school_master/2020/2020-06/2020-06-04_school_master/04-06-2020_13:12:59.574_5e160862-c5b3-4121-9a96-ecefa34fc264_school_mst.zip",
   "bucket":"cqube-gj-input"
}
  • After getting the emission access token from admin console, update the details mentioned below in cQube_Workflow/development/python/client/config.py.

    • Emission access token

    • File location of the cqube_emission directory where the files are placed as below. Example:/home/ubuntu/cqube_emission/

    • Emission End point i.e., emission_url Ex:https://<cqube-domain>/data Note: URL depends upon the server configured in firewall which includes SSL and reverse proxy location)

  • After completing the configuration. Save and close the file.

Emission order & instructions:

  • Static files as defined in above in this page need to be emitted first.

  • For the CRC report, the inspection_master needs to be emitted prior to user_location_master to visualize the CRC reports.

  • For PAT (Periodic assessment report), periodic_exam_mst and periodic_exam_qst_mst needs to be emitted prior to periodic_exam_result_trans

  • school_grade_enrolment file can be emitted either as periodic_exam_grade_details or semester_exam_grade_details. The data will be updated to school_grade_enrolment and will be used in school_hierarchy_details. The count from school_hierarchy details will be used in pat & sat reports. Before the sat & pat workflow the school_grade_enrolment needs to be emitted.

  • Execute the client.py file located in cQube_Workflow/development/python/client/ directory, as mentioned below to emit the data files to s3_emission bucket.

python3 client.py
  • Finally see the output in https://<cqube_domain>/

The list of columns which have to be emitted for a data source are provided in the .

The header should be in the same order as described in the , for that respective data set.

Data emission overview of data flow can be viewed

GET

POST

POST

📂
file
file
here
https://cqube.tibilprojects.com/data/list_s3_buckets
https://cqube.tibilprojects.com/data/list_s3_files
https://cqube.tibilprojects.com/data/download_uri