# What is cQube & what does it solve Details out the technical architecture of cQube cQube is envisioned as a ready-to-use / pre-packaged, configurable, and extendable DPG solution to enable observability and action towards effective policy implementation in education and other sectors, involving various stakeholders across govt, society, and private sectors. # Business case ### Problem Statement There are several challenges that government program owners face while leveraging technology for monitoring public programs: * Program administrators face challenges in defining actionable insights and relevant indicators for tracking * Data ingestion, processing & visualisation available as disaggregated products, not packaged into a solution * Governments are unable to hire expert technical staff (architects, engineers) to smartly leverage a combination of available products together for end-to-end monitoring * Governments have budget and digital infrastructure (server) limitations ### **cQube as a Solution** cQube offers two core differential value propositions for monitoring programs: * A domain-agnostic product packaging of ingestion, processing and visualisation layer in a manner that is: * Easy to deploy with minimal engineering staff * Lite deployment (minimal infrastructure requirements optimised to program needs) * Built for scale (to be scale tested with \~500k users) * Leverages reference schemas and spec-compliant API based implementation * Brownfield ready with modular components interoperable with existing systems * Domain-specific configuration layer allowing out-of-the-box contextualisation of indicators, insights, role based nudges and actions # cQube ed **Need** Vidya Samiksha Kendra was initiated by Ministry of Education as a [program](https://pib.gov.in/PressReleaseIframePage.aspx?PRID=1843168) with the following objectives: * Improve monitoring of schemes / programs in education * Effectively collect, monitor, correlate and analyze data to take timely decisions * Envisioned as state-level systems to monitor schemes / programs and build accountability in field-staff * To drive a big leap in learning outcomes through meaningful data based action There are several challenges that government program owners face while leveraging technology for monitoring public programs: * Data ingestion, processing & visualization available as disaggregated products, not packaged into a solution * Lack of visibility at the granular level in the system & thus face challenges in defining actionable insights and relevant indicators for tracking * Limitation on hiring expert technical staff (architects, engineers) to smartly leverage a combination of available products together for end-to-end monitoring * Constraints related to budgets lead to limitations on building extensive digital infrastructure capabilities ### Solution cQube Ed is a pre-packaged solution for education with 120+ pre-defined actionable indicators & insights like attendance, assessment etc. It acts as an accelerator for states in fast tracking their Vidya Samiksha Kendra journeys. # Design Principles cQube Ed is based on the following design principles: 1. **Solution:** cQube is neither a tool nor a platform. It is a ready-to-use/pre-packaged, configurable, and extendable solution to enable observability and action towards effective policy implementation in education and other sectors, involving various stakeholders across govt, society, and private sectors. 2. **Education-specific:** A pre-packaged solution of cQube for education, *cQube Ed*, comes with a set of predefined actionable indicators and insights which are specific to education. For eg: the metrics could be related to attendance, enrolment, assessments etc. The schema for data ingestion will also be defined for edu-specific indicators. 3. **Based in Indian Context:** cQube is based in the Indian context, implying that the jurisdictions and hierarchies will be defined accordingly. For eg: The hierarchy for cQube Ed will be as follows - State > District > Block > Cluster > School > Class. # cQube adoptions Coming soon... # Discuss more about cQube Lets discuss here- [Link](https://github.com/orgs/Sunbird-cQube/discussions/792) # cQube on Gitpod Developers can try out cQube installation and ingestion on Gitpod. 1. Login to your github account. 2. Open sunbird cqube-devops repository and fork the repo into your personal github 3. Checkout to gitpod-branch 4. Update readme.md file as follows
Replace \ with your actual github username **Note:** * By default JH is selected as a state code. * If you want to change the state code, you can navigate to cqube-devops/config\_files/config.yml * Change the state code and commit the changes. 5. Commit the changes. 6. Click on Open in Gitpod 7. If you are a new user to gitpod, complete the signup process 8. Once the registration is done then the configuration page gets displayed. Click on continue
9. Automatically it sets a VM and start the cqube deployment. 10. Once the config file is generated, type no and hit enter

11. Select the default config file by typing no
12. Wait unit the installation is done. 13. Once installation is completed, you will be able to see the url to access the dashboard
14. Access the dashboard using the url in your browser.
15. Use keycloak credentials from config.yml file to login
16. Follow the [ingestion and processing ](https://cqube.sunbird.org/use-cqube/ingesting-the-data)steps to ingest the data. Once done you will be able to see the data in the dashboard.
Cqube deployment in gitpod video link: [gitpod\_recording.zip](https://drive.google.com/file/d/1lc6whjXGbxuPDpDqnyJs-n4Ev6GkXmdk/view?usp=sharing) Ingestion and processing video link:
# cQube Roadmap ### **How to read:** 1. The planned items are available on the JIRA board for cQube - [Link](https://project-sunbird.atlassian.net/jira/software/c/projects/CQ/boards/115/backlog?issueLimit=100) 2. For the items on the community discussions, we expect interested community users to go to the github discussion to suggest changes on existing items or add new items which will be added to the roadmap. {% embed url="" %}
# Architecture Details out the technical architecture of cQube cQube is envisioned as a ready-to-use / pre-packaged, configurable, and extendable DPG solution to enable observability and action towards effective policy implementation in education and other sectors, involving various stakeholders across govt, society, and private sectors. The architecture of cQube has been revised to enable flexibility in configuration: * Spec-based architecture has been created for cQube Ed with API-first design * There are 5 blocks: Installation > Ingestion > Processing > Visualisation > Actions * All the blocks can be used individually and as an end-to-end solution by cQube adopters The terminologies used in the architecture can be found [here](https://project-sunbird.atlassian.net/wiki/spaces/CQUB/pages/3258024044/cQube+Design+Document+Nov+2022#1.-Terminologies).
High-level architecture design can be found [here](https://project-sunbird.atlassian.net/wiki/spaces/CQUB/pages/3258024044/cQube+Design+Document+Nov+2022). Low-level architecture design can be found [here](https://project-sunbird.atlassian.net/wiki/spaces/CQUB/pages/3266740614/cQube+LLD+-+v12Dec2022). # Design Principles * **Low Tech/Minimal Coding:** Leverage existing tools and provide configuration options to minimize the need for coding. * **API First:** Design APIs for all functionality and blocks to enable independent evolution and integration. * **Progressive Enhancement:** Allow for module changes and tiered improvements in the user experience. * **Modularity:** Design modular blocks that can be selected and integrated based on specific needs. * **Integrations:** Enable integration of higher-order capabilities through clear boundaries and APIs. * **Ease of Deployment/Management:** Provide easy deployment options and support on commodity hardware or cloud platforms. * **Scalable:** * **Ingestion:** Design scalable ingestion processes for real-time data updates. * **SLA Driven:** Strive for a 30-minute SLA between data ingestion and insight generation. * **Performance:** Prioritize stable performance within specified hardware limitations. * **Data Privacy:** Store only aggregated data and insights, ensuring no personally identifiable information is stored. * **Accessibility:** Ensure accessibility on mobile and desktop devices. * **Data Security:** Implement specific measures to safeguard data (further details required). These design considerations aim to create a configurable, scalable, and user-friendly solution for effective data-driven governance in the education sector. The detailed design principles of cQube, according to the architecture, can be found [here](https://project-sunbird.atlassian.net/wiki/spaces/CQUB/pages/3258024044/cQube+Design+Document+Nov+2022#3.-Design-Considerations). # Key Components This page outlines the key components of cQube * **Installation:** The installation block enables easy deployment and setup of the cQube platform. It provides a streamlined process for getting cQube up and running quickly. * **Ingestion:** The ingestion block handles the collection and validation of data from various sources. It ensures that the data is accurately captured and standardized for further processing. * **Processing:** The processing block involves data transformation, cleansing, aggregation, and advanced analytics. It leverages the collected data to derive meaningful insights, identify patterns, and generate actionable intelligence. * **Visualization:** The visualization block focuses on presenting the analyzed data in a visually appealing and intuitive manner. It includes interactive dashboards, charts, graphs, and reports that allow users to explore and understand the data easily. * **Actions:** The actions block enables stakeholders to take informed actions based on the insights gained from the data. It supports evidence-based decision-making and provides a platform for implementing targeted interventions and policies. # Details of microservices * **Ingestion-ms:** The ingestion-ms is used to upload the data of the events, datasets, dimensions, transformers and pipeline. All these apis will be to ingesting the data into the cQube. * **Spec-ms:** The spec-ms is used to import schema of the events, datasets, dimensions. All these specs will be defined by the cQube platform prior to ingesting the data into the cQube. These specifications are derived by considering the KPIs as the Indicator. * **Generator-ms:** The generator-ms is used to create the adapters for converting the older input data format to newer format for VSK & NVSK programs. * **Nifi-ms:** Apache NiFi is used as a real-time integrated data logistics and simple event processing platform * **Postgres-ms:** Postgres microservice contains the schema and tables * **Nginx-ms:** It is commonly used as a reverse proxy and load balancer to manage incoming traffic and distribute it to slower upstream servers * **Kong-ms:** It is a lightweight API Gateway that secures, manages, and extends APIs and microservices. * **Dashboard-ms:** It consists of an angular app, it is used to visualize the datasets present in postgres-ms in the form of charts. On run time it requests query-builder to fetch data from postgres-ms and load it into the client side(Browser) * **Query\_builder-ms:** It consists of backend API, which consists of JWT,METRICS,QUERY apis **JWT** - it will generate a jwt token to restrict the other apis. **METRICS -** it consists of menus for the navigation bar and dashboard cards. **QUERY** - this api used for executing the SQL queries which integrated with Dashboard-ms. **LASTMODIFIED -** this api will use for the last modified data in the s3,azure and minio # Suggested Team Structure Shows you a list of things to help you get started with cQube V 5.0: cQube v5.0, the latest version of cQube, enables monitoring of 9 out-of-the-box state and national education programs for performance improvement. All of these programs will be tailored to the user role and jurisdiction. Any new program can also be added by the state for monitoring purposes. To get started with cQube, the following team composition is suggested: | Type | Language / Skills | Experience | Full-time people | | ----------------- | ------------------------------------- | ---------- | ---------------- | | Backend Developer | TypeScript, NestJS | 3+ years | 1 | | DevOps | Docker, Ansible | 3+ years | 1 | | QA | UAT | 3+ years | 1 | | Tech Lead | All of the above, & management skills | 5+ years | 1 | To enable any customizations/add new programs, following members are suggested in addition to the above: | Type | Language / Skills | Experience | Full-time people | | ------------------ | -------------------------- | ---------- | ---------------- | | Backend Developer | Nest.js, Python, NiFi, SQL | 4+ years | 1 | | Frontend Developer | Angular | 3+ years | 1 | \ The detailed skill sets required for each phase of the setting up process are mentioned below:
Configure
  • Knowledge of defining the grammars (dimensions, event) and verifying CSVs (data files for dimensions and events)
  • Basic knowledge of Pandas + Python to clean data if needed.
Deploy
  • Shell scripting, AWS, Ansible, Docker (build, deploy, update containers; Verify from logs connections)
  • Nginx, Networking, Ubuntu.
Monitor
  • Knowledge of defining the grammars (dimensions, event) and verifying CSVs (data files for dimensions and events) and KPI and their definitions
  • Basic of Database to verify SQL queries or construct new ones if needed
  • Ability to use GitHub to raise tickets, issues.
Maintain
  • Ansible, Docker (build, deploy, update containers; Verify from logs connections), Nginx, Networking, Ubuntu
Enhance & contribute (Frontend)
  • Angular 14
  • Good understanding of MVC
  • Understanding of javascript and typescript
  • Debugging skills; Basics of Dockerfile
Enhance & contribute (Backend)
  • Basic knowledge of Database to verify SQL queries or construct efficient new ones if needed
  • Knowledge of how datasets are created in cQube
  • NestJS, Typescript, Postgres
Enhance & contribute (Ingestion)
  • Should only be done by the core team
Enhance & contribute (DevOps)
## Supported Servers & Clouds 1. AWS - The steps to deploy cQube v5.0 on AWS have been mentioned [here](/deployment-process/aws-deployment) 2. On premise (State Data Centre, Local) - The steps to deploy cQube v5.0 on 'On premise' (State Data Centre, Local) have been mentioned [here](/deployment-process/sdc-deployment) # Hardware Requirements List of hardware requirements as pre-requisites for cQube V 5.0 ### For AWS : Create an AWS EC2 instance with the following configurations: * Ubuntu 22.04 (supported) * 16 GB of System RAM (minimum requirement) * 4 core CPU for AWS(minimum requirement) * 250GB HDD * S3 Storage Other Requirements: * Create a domain name * Configure cname of AWS ec2 to the domain name * Create a SSL certificate for the domain name.(which is applicable to configure with nginx ) **For Azure :** Create an AzureEC2 instance with the following configurations: * Ubuntu 22.04 (supported) * 16 GB of System RAM (minimum requirement) * 4 core CPU for Azure(minimum requirement) * 250GB HDD * Blob storage Other Requirements: * Create a domain name * Configure cname of Azure ec2 to the domain name * Create a SSL certificate for the domain name.(which is applicable to configure with nginx ) **For Oracle :** Create an Oracle EC2 instance with the following configurations: * Ubuntu 22.04 (supported) * 16 GB of System RAM (minimum requirement) * 4 core CPU for 2 OCPU for Oracle (minimum requirement) * 250GB HDD * Cloud Storage Other Requirements: * Create a domain name * Configure cname of Oracle ec2 to the domain name * Create a SSL certificate for the domain name.(which is applicable to configure with nginx ) ### For On-Premise (State Data Centre, Local): One server is required with the following configurations: * Ubuntu 22.04 (supported) \[[Link to install Ubuntu](https://ubuntu.com/tutorials/install-ubuntu-desktop#1-overview)] * 16 GB of System RAM (minimum requirement) * 4 core CPU (minimum requirement) * 250GB HDD * Minio Storage Other Requirements: * Create a domain name * Configure CNAME or A record of AWS ec2 to the domain name * Create a SSL certificate for the domain name.(which is applicable to configure with nginx) # Prerequisites Checklist | Item | Description | Automatic check/ manual check | Feedback to the user | | ----------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | OS | Check if the os is ubuntu 22.04 | Automatic | Please use ubuntu 22.04 or above for the best experience | | RAM | Check if the RAM is at least 16GB | Automatic | Highly recommended to maintain minimum of 16GB RAM | | CPU | Check if CPU is 4 core | Automatic | 2 core CPU might work but we suggest to use 4 core to avoid any server hangs | | Storage | Check if storage is minimum 250GB | Automatic | Please ensure that the storage is at least 250GB & is an SSD | | Domain Name test | Validate if the domain name is valid | Automatic |


| | SSL certs | Check if the user has provided the certs | Automatic | Certificates which are compatible with nginx | | Check ports ( 80,443,9000,9001,8000,4200,3000,3001,3002,3003,8096,5432,8080 ) | Check if the specified ports and running | Automatic |

Please kill the port by following the instructions below.
sudo netstat -ntlp
Sudo kill -15 \

| | Check for dimension files | Check if the required csv files are present in dimension folder | Automatic | Please upload the required dimension files | | SSL certificate verification | Check if the SSL certificates are valid | Manual | Make sure the certificate has a full chain of certificates to avoid handshake issues of SSL. | | SSL certificates upload | Copy the certificates into the server | Manual |

If you are opting for public mode of installation, Make sure to copy certificates to the server.(Navigate to the directory cqube-devops/ansible/ssl\_certificates.

Copy the certificate and private key inside this cqube- devops/ansible/ssl\_certificates folder )

| | Dimension files upload | Copy the dimension files to the server | Manual |

If you are opting for VSK and If you are willing to pull the data from NVSK, Then this step is required. ( Navigate to the directory cqube-devops/ansible/ssl\_certificates.cp state-dimension.data.csv grade-dimension.data.csv subject-dimension.data.csv etc../home/ubuntu/cqube-devops/ansible.dimension\_files

Path: cqube-devops/ansible/dimension\_files.)

| # Checking pre existing ports Make sure the following ports are not running. If running please stop the service or kill the port Security Group Configuration: * Port 80 inbound from 0.0.0.0/0 * Port 443 inbound from 0.0.0.0/0 * Port 3005 inbound from 0.0.0.0/0 * 5432 inbound to the particular ip which needs access Kong Configurations: ( already configured using one step deployment ) * 3000 will get routed to /api/ingestion * 3001 will get routed to /api/spec * 3002 will get routed to /api/queryBuilder * 3003 will get routed to /api/generator * 443 ( nginx is configured in cqube ) **Note:** we can check for the ports using the command **sudo netstat -ntlp** and can kill the particular port using its port id by command **sudo kill -15 \**
# Instance Creation Here the links for how to Create the instance in Azure , AWS and Oracle 1. [Azure ](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal?tabs=ubuntu) 2. [AWS ](https://www.techtarget.com/searchcloudcomputing/tutorial/How-to-create-an-EC2-instance-from-AWS-Console) 3. [Oracle](https://docs.oracle.com/en/cloud/cloud-at-customer/occ-get-started/create-oracle-linux-instance.html#GUID-77D03EFA-ABAA-4ED5-ABC8-9CA756B9D6BF) # Copying SSL Certificate Below are the steps to copy SSL certificates from a local source to Ubuntu and from Windows to Ubuntu. **From Local to Ubuntu:** 1. Please find the below command to copying ssl keys from local ubuntu to ubuntu server sudo scp -i poc\_key.pem \ ubuntu@\:\~/ 2. After copying the files from local to server then we need to move the SSL keys (.crt and.key) to the mentioned path. Path: cqube-devops/ansible/ssl\_certificates cp \ /home/ubuntu/cqube-devops/ansible/ssl\_certificates **From Windows to Ubuntu:** 1. Please find the below link to copying ssl keys from windows machine to ubuntu server(winscp,filezilla) 2. After copying the files from windows to server then we need to move the SSL keys (.crt and.key) to the mentioned path. Path: cqube-devops/ansible/ssl\_certificates cp \ /home/ubuntu/cqube-devops/ansible/ssl\_certificates # How can I install cQube? Under this topic we will get some understanding about High level Understanding of deployment. How to set the instance: 1. [Oracle Installation ](https://cqube.sunbird.org/use-cqube/getting-started/oracle-instance-setup) 2. [AWS Installation](https://cqube.sunbird.org/use-cqube/getting-started/aws-installation) 3. [SDC Installation ](https://cqube.sunbird.org/use-cqube/getting-started/sdc-installation) 4. [Azure Installation ](https://cqube.sunbird.org/use-cqube/getting-started/azure-installation) # High level understanding of deployment
# Oracle Installation Instance Setup in Oracle Cloud: **Step 1:** Create an instance with Ubuntu 22.04(Oracle Linux ) image, 2 core OCPU, and 16GB RAM
**Step 2:** Download the private key file associated with the instance.
**Step 3:** Create a Storage Bucket Go to the Oracle Cloud console and create a bucket with default configurations.
**Step 4:** Generate Config File in Oracle Cloud Login to the Oracle Cloud account and navigate to the profile section
Generate a new key pair in the API keys section and download the private key file
Copy the content of the configuration file for future use
**Upload the API Key File to the cQube Server:** * Connect to the cQube server using SSH. * Copy the Oracle API private key file to the server using the command: scp -i \ \ ubuntu@\:\~/ * Switch to the root user: sudo su * Move the private key file to the .oci directory: mkdir /root/.oci && cd /root/.oci mv /home/\/\ . * Verify the presence of the file in the current directory: ls -ltr * Exit from the root user and continue with the cQube installation. **Instructions to fill the Oracle config variables during installation:** * Copy the config file to the desired location (/root/.oci/config). * Open the config file and replace the following placeholders with actual values from the API key config file: User OCID Tenancy OCID Region (by index or name) * Decide whether to generate a new API Signing RSA key pair. If declining, provide the path to an existing key. * Specify the location for the API Signing private key file (/root/.oci/\). **(Note: Bucket should be created manually before installation process starts)** **Oracle bucket policy creation steps** **(Note : These policies are applicable for NVSK data pull)** **Step 1:** Initially, log in to the Oracle Cloud using the credentials, as illustrated in the screenshot below.
**Step 2**: Following the guidance from the above screenshot, please type “policies” into the search bar, as depicted in the screenshot below.
**Step 3:** Simply click on the policies option and proceed to create new policies, as shown in the screenshot below.
**Step 4**: We already have some existing policies. Please refer to the attached screenshot for more details.
**Primary Steps:** **Step 1:** A SSL certificate must be uploaded if you are opting for the public mode of installation where you want to access cQube using a public API endpoint. **Note :** You can ignore the above step if you are opting for localhost installation (single desktop). Connect to the cqube Oracle instance ssh -i \ \@\ Ex: ssh -i poc\_key.pem -o ServerAliveInterval=60 ubuntu\@13.200.12.31 **Step 2:** Clone the cqube-devops repository using following command **git clone** [**https://github.com/Sunbird-cQube/cqube-devops.git**](https://github.com/Sunbird-cQube/cqube-devops.git)
**Step 3:** Navigate to the directory where cQube is cloned or downloaded and checkout to the desired branch **cd cqube-devops/** **git checkout release-v5.0.5(latest release branch)**
**Step 4:** Navigate to the directory cqube-devops/ansible/ssl\_certificates. **Copy the certificate and private key inside this cqube- devops/ansible/ssl\_certificates folder.**
**Step 5 :** If you are opting to pull the data from the NVSK instance, then make sure to upload the dimension files. If you do not want to pull the data, then you can ignore this step. (Applicable if you choose access type VSK). If you are opting to pull data from an NVSK instance, then make sure to copy the dimension files inside cqube-devops/ansible/dimension\_files
Once all the primary steps of uploading SSL certificates and dimensions are done, we can check prerequisites using the script. **Navigate back to cqube-devops directory** **Step 6**: Give the following permissions to the pre\_requisites.sh file **sudo chmod u+x shell\_scripts/pre\_requisites.sh** **Step 7:** Run the pre\_requisites script with non root user with sudo privileges **sudo ./shell\_scripts/pre\_requisites.sh**
**Step 8 :** When steps 6 and 7 are done, you will see the feedback from status checks. **Step 9:** Go through the feedback provided, and if you find any issues in red resolve them by following the hints provided. If everything is good then proceed with cqube deployment. **Deployment Process:** **Step 1:**Refer to the prerequisites checklist section before starting the cqube deployment.-[ Link ](https://app.gitbook.com/o/-Mi9QwJlsfb7xuxTBc0J/s/-Miy6UN-HyTKGo2Sjcky/~/changes/326/get-started-on-cqube/prerequisites-checklist) **Step 2:** Navigate to cqube-devops directory and give the executable permissions to the install.sh file **sudo chmod u+x install.sh** **Step 3:** Install cQube with non root user with sudo privileges **sudo ./install.sh**
Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cQube **Step 4:** Generation of domain specific configuration file - Please refer to the Hints provided and enter the values accordingly. * access\_type: Enter VSK, NVSK, or others. a) VSK: To set up VSK for a state b) NVSK: To establish a nationwide cQube deployment c) Others: To set up cQube for other purposes. * state\_name: Enter the required state code by referring to the state list provided (this variable needs to be filled if you are opting for access\_type as VSK; otherwise, it will get filled as NA by default). * loginpage\_status: Enter true if you want to enable the login screen for the CQUBE instance. Otherwise, enter false. * data\_pull\_status: Enter true if you want to pull data from an NVSK instance, or else enter false. (You need to fill this variable if you have selected access\_type as VSK; otherwise, it will get auto-filled with NA.) * nvsk\_api\_endpoint: Enter the NVSK instance api endpoint if you selected true for data\_pull\_status; otherwise, it will get autofilled with NA. Once all the above variables are filled, you get prompted with a preview of the domain-specific config file. Review the entered variables. If everything is correct, then type ‘no’ and proceed further, or else enter ‘yes’ and re-enter the variables correctly.
**Step 5:** Generation of a generic config file: refer to the hints provided and enter the config file accordingly. * mode\_of\_installation: Enter public as you have opted for setting up cqube in AWS. (Possible values are public and localhost.) * storage\_type: Enter storage type as oracle. (Possible values are aws, azure, oracle, and local.) * api\_end\_point: Enter the url in which the clock is to be configured. * nginx\_cert\_file: Enter the name of the SSL certificate file that you uploaded in the SSL\_certificates folder. * nginx\_key\_file: Enter the name of the private key file that you uploaded in the ssl\_certificates folder. * Db\_credentials: You will be prompted with default database credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your database credentials. db\_user\_name ( Enter the postgres database username ) db\_name ( Enter the postgres database name ) db\_password ( Enter the postgres password )
* Readonly\_db\_credentials: You will be prompted with default read-only database credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your read-only database credentials. read\_only\_db\_user\_name ( Enter the postgres database username ) read\_only\_db\_password ( Enter the postgres password )
* Keycloak\_credentials: You will be prompted with the default keycloak admin dashboard credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your keycloak credentials. Keycloak\_adm\_name (Enter the keycloak admin name eg: admin) Keycloak\_adm\_password (enter the keycloak admin password eg: Admin\@123)
**Step 6:** Once the config file is generated, a preview of the config file is displayed, followed by a question where the user gets the option to re-enter the configuration values by choosing ‘yes’. If option ‘no’ is selected, then the install.sh moves to the next section.
* Enter a location for your config \[/home/basha/.oci/config]: /root/.oci/config (enter) * Enter a user OCID * Enter a tenancy OCID * Enter a region by index or name: ap-mumbai-1 * Do you want to generate a new API Signing RSA key pair? (If you decline you will be asked to supply the path to an existing key.) \[Y/n] : \ n * Enter the location of your API Signing private key file Ex: /root/.oci/itadmin\@tibilsolutions.com\_2023-04-18T10\_08\_35.916Z.pem
**Step 7:** A preview of the program\_selector.yml file is displayed, followed by a question where the user gets an option to enable or disable the programs by choosing 'yes'. If option 'no' is selected, then the install.sh moves to the next section. 1\. If access\_type is selected as VSK, the following programs will be displayed: by default, all programs are selected, and the programs are publicly visible.
**Note:** If you are willing to select the programs, type yes, select the programs you want, and enter public or private for each program to enable or disable the login screen for programs (follow the hints provided). 2. If access\_type is selected as NVSK, the following programs will be displayed: By default, all programs are selected, and the programs are publicly visible.
**Note:** If you are willing to select the programs, type yes, select the programs you want, and enter public or private for each program to enable or disable the login screen for programs (follow the hints provided). **Step 8:** Once all the configuration files are generated, the script moves further to clone the cQube microservice repositories and deploy cQube **Step 9:** If you selected data\_pull\_status as true, then you will see the data pulling APIs where the data gets pulled from the NVSK instance and processed in the VSK cQube Once installation is done, we need to wait 20 minutes to visualize the data on the cQube dashboard. **Step 10:** Once the installation is completed, you will be prompted with the following messages and required reference urls: **(Note: The installation process is expected to take approximately 30-40 minutes.)** **cQube Installed Successfully**


# AWS Installation IAM user and Role creation for S3 connectivity (AWS) An AWS Identity and Access Management (IAM) user is an entity that is created in AWS to represent the person or application that uses it to interact with AWS. A user in AWS contains a name and credentials. An IAM user with administrator permissions is different from the AWS account root user. One has to create an IAM user with a supported role to provide the connectivity between EC2 and S3. The role should have list, read, and write permissions. **Primary Steps:** **Step 1:** A SSL certificate must be uploaded if you are opting for the public mode of installation where you want to access cQube using a public API endpoint. **Note :** You can ignore the above step if you are opting for localhost installation (single desktop). **Step 2:** Clone the cqube-devops repository using following command **git clone** [**https://github.com/Sunbird-cQube/cqube-devops.git**](https://github.com/Sunbird-cQube/cqube-devops.git)
**Step 3:** Navigate to the directory where cQube is cloned or downloaded and checkout to the desired branch **cd cqube-devops/** **git checkout release-v5.0.5(latest release branch)**
**Step 4:** Navigate to the directory cqube-devops/ansible/ssl\_certificates. **Copy the certificate and private key inside this cqube- devops/ansible/ssl\_certificates folder.**
**Step 5 :** If you are opting to pull the data from the NVSK instance, then make sure to upload the dimension files. If you do not want to pull the data, then you can ignore this step. (Applicable if you choose access type VSK). If you are opting to pull data from an NVSK instance, then make sure to copy the dimension files inside cqube-devops/ansible/dimension\_files
Once all the primary steps of uploading SSL certificates and dimensions are done, we can check prerequisites using the script. **Navigate back to cqube-devops directory** **Step 6**: Give the following permissions to the pre\_requisites.sh file **sudo chmod u+x shell\_scripts/pre\_requisites.sh** **Step 7:** Run the pre\_requisites script with non root user with sudo privileges **sudo ./shell\_scripts/pre\_requisites.sh**
**Step 8 :** When steps 6 and 7 are done, you will see the feedback from status checks. **Step 9:** Go through the feedback provided, and if you find any issues in red resolve them by following the hints provided. If everything is good then proceed with cqube deployment. #### Deployment Process **Step 1:** Connect to the cqube AWS ec2 instance **For linux and macOS:** * Download the .pem file which is generated while creating the EC2 instance * Open the terminal and navigate to the folder where .pem file has been downloaded * Then give the read permission to the .pem file using following command **sudo chmod 400 \** * Use the following command to connect to the instance **ssh -i \ \@\** **For windows:** * Download the .pem file which is generated while creating the EC2 instance * Use puttygen to connect to the instance. * Refer following link to use puttyGen for connecting. **Step 2:** Refer to the prerequisites checklist section before starting the cqube deployment.-[ Link ](https://app.gitbook.com/o/-Mi9QwJlsfb7xuxTBc0J/s/-Miy6UN-HyTKGo2Sjcky/~/changes/326/get-started-on-cqube/prerequisites-checklist) **Step 3:** Navigate to cqube-devops directory and give the executable permissions to the install.sh file **sudo chmod u+x install.sh** **Step 4:** Install cQube with non root user with sudo privileges **sudo ./install.sh**
Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cQube **Step 5:** Generation of domain specific configuration file - Please refer to the Hints provided and enter the values accordingly. * access\_type: Enter VSK, NVSK, or others. a) VSK: To set up VSK for a state b) NVSK: To establish a nationwide cQube deployment c) Others: To set up cQube for other purposes. * state\_name: Enter the required state code by referring to the state list provided (this variable needs to be filled if you are opting for access\_type as VSK; otherwise, it will get filled as NA by default). * loginpage\_status: Enter true if you want to enable the login screen for the CQUBE instance. Otherwise, enter false. * data\_pull\_status: Enter true if you want to pull data from an NVSK instance, or else enter false. (You need to fill this variable if you have selected access\_type as VSK; otherwise, it will get auto-filled with NA.) * nvsk\_api\_endpoint: Enter the NVSK instance api endpoint if you selected true for data\_pull\_status; otherwise, it will get autofilled with NA. Once all the above variables are filled, you get prompted with a preview of the domain-specific config file. Review the entered variables. If everything is correct, then type ‘no’ and proceed further, or else enter ‘yes’ and re-enter the variables correctly.
**Step 6**: Generation of a generic config file: refer to the hints provided and enter the config file accordingly. * mode\_of\_installation: Enter public as you have opted for setting up cqube in AWS. (Possible values are public and localhost.) * storage\_type: Enter storage type as AWS. (Possible values are aws, azure, oracle, and local.) * api\_end\_point: Enter the url in which the clock is to be configured. * nginx\_cert\_file: Enter the name of the SSL certificate file that you uploaded in the SSL\_certificates folder. * nginx\_key\_file: Enter the name of the private key file that you uploaded in the ssl\_certificates folder. * Db\_credentials: You will be prompted with default database credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your database credentials. * db\_user\_name ( Enter the postgres database username ) * db\_name ( Enter the postgres database name ) * db\_password ( Enter the postgres password )
* Readonly\_db\_credentials: You will be prompted with default read-only database credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your read-only database credentials. 1. read\_only\_db\_user\_name ( Enter the postgres database username ) 2. read\_only\_db\_password ( Enter the postgres password )
* Keycloak\_credentials: You will be prompted with the default keycloak admin dashboard credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your keycloak credentials. 1. Keycloak\_adm\_name (Enter the keycloak admin name eg: admin) 2. Keycloak\_adm\_password (enter the keycloak admin password eg: Admin\@123)
**Step 7:** Once the config file is generated, a preview of the config file is displayed, followed by a question where the user gets the option to re-enter the configuration values by choosing ‘yes’. If option ‘no’ is selected, then the install.sh moves to the next section.
**Step 8:** Generation of storage config file: Refer to the hints provided below and enter the config file accordingly. * aws\_access\_key: Enter the aws s3 access key to access the s3 bucket. * aws\_secret\_key: Enter the aws s3 secret key to access the s3 bucket. * s3\_bucket: By default, an s3 bucket is selected. If you get an error because the bucket already exists, type a unique bucket name and proceed further. **Step 9:** A preview of the program\_selector.yml file is displayed, followed by a question where the user gets an option to enable or disable the programs by choosing 'yes'. If option 'no' is selected, then the install.sh moves to the next section. 1. If access\_type is selected as VSK, the following programs will be displayed: by default, all programs are selected, and the programs are publicly visible.
**Note:** If you are willing to select the programs, type yes, select the programs you want, and enter public or private for each program to enable or disable the login screen for programs (follow the hints provided). 2. If access\_type is selected as NVSK, the following programs will be displayed: By default, all programs are selected, and the programs are publicly visible.
**Note:** If you are willing to select the programs, type yes, select the programs you want, and enter public or private for each program to enable or disable the login screen for programs (follow the hints provided). **Step 10:** Once all the configuration files are generated, the script moves further to clone the cQube microservice repositories and deploy cQube **Step 11:** If you selected data\_pull\_status as true, then you will see the data pulling APIs where the data gets pulled from the NVSK instance and processed in the VSK cQube Once installation is done, we need to wait 20 minutes to visualize the data on the cQube dashboard. **Step 12:** Once the installation is completed, you will be prompted with the following messages and required reference urls: **(Note: The installation process is expected to take approximately 30-40 minutes.)** **cQube Installed Successfully**

\
# SDC Installation **Step 1:** Use the following command to connect to the On-premise instance **ssh -i \ \@\** **Ex: ssh -i poc\_key.pem -o ServerAliveInterval=60 ubuntu\@13.200.12.31** **Step 2:** Clone the cqube-devops repository using following command **git clone** [**https://github.com/Sunbird-cQube/cqube-devops.git**](https://github.com/Sunbird-cQube/cqube-devops.git)
**Step 3:** Navigate to the directory where cqube is cloned or downloaded and checkout to the desired branch(Release Branch) **cd cqube-devops/** **git checkout release-v5.0.5(latest release branch)**
**Step 4:** Give the following permissions to the install.sh file **sudo chmod u+x install.sh** **Step 5**: Install cqube with non root user with sudo privileges **sudo ./install.sh**
Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cQube **Step 6:** User Input Variables - These are the variables which need to be entered by the user by following the Hint provided * state\_name ( Enter the required state code by referring to the state list provided ) * api\_end\_point ( Enter the url in which cqube to be configured ) * Storage\_type : Local * Mode\_of\_installation:Public
**Step 7:** Optional\_variables- Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for yes to enter their credentials otherwise can opt for no when the question pops up * db\_user\_name ( Enter the postgres database username ) * db\_name ( Enter the postgres database name ) * db\_password ( Enter the postgres password )
**Step 8:** Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.
**Step 9:** A preview of the program\_selector.yml file is displayed followed by a question where the user gets an option to enable or disable the programs on choosing yes. If option no is selected then the install.sh moves to the next section.
**Step 10:** Once the installation is completed, You will be prompted with the following messages and required reference urls. **cQube Installed Successfully** **(Note: The installation process is expected to take approximately 30-40 minutes.)** cQube ingestion api can be accessible using \

# Azure Installation IAM user and Role creation for AZURE Storage Blob Container connectivity An AZURE Identity and Access Management (IAM) user is an entity that is created in AZURE to represent the person or application that uses it to interact with AZURE. A user in Azure contains a name and credentials. An IAM user with administrator permissions is different from the AWS account root user. One has to create an IAM user with a supported role to provide the connectivity between azure VM and blob container. The role should have list, read and write permissions. **Primary Steps:** **Step 1:** A SSL certificate must be uploaded if you are opting for the public mode of installation where you want to access cQube using a public API endpoint. **Note :** You can ignore the above step if you are opting for localhost installation (single desktop). **Step 2:** Clone the cqube-devops repository using following command **git clone** [**https://github.com/Sunbird-cQube/cqube-devops.git**](https://github.com/Sunbird-cQube/cqube-devops.git)
**Step 3:** Navigate to the directory where cQube is cloned or downloaded and checkout to the desired branch **cd cqube-devops/** **git checkout release-v5.0.5(latest release branch)**
**Step 4:** Navigate to the directory cqube-devops/ansible/ssl\_certificates. **Copy the certificate and private key inside this cqube- devops/ansible/ssl\_certificates folder.**
**Step 5 :** If you are opting to pull the data from the NVSK instance, then make sure to upload the dimension files. If you do not want to pull the data, then you can ignore this step. (Applicable if you choose access type VSK). If you are opting to pull data from an NVSK instance, then make sure to copy the dimension files inside cqube-devops/ansible/dimension\_files
Once all the primary steps of uploading SSL certificates and dimensions are done, we can check prerequisites using the script. **Navigate back to cqube-devops directory** **Step 6**: Give the following permissions to the pre\_requisites.sh file **sudo chmod u+x shell\_scripts/pre\_requisites.sh** **Step 7:** Run the pre\_requisites script with non root user with sudo privileges **sudo ./shell\_scripts/pre\_requisites.sh**
**Step 8 :** When steps 6 and 7 are done, you will see the feedback from status checks. **Step 9:** Go through the feedback provided, and if you find any issues in red resolve them by following the hints provided. If everything is good then proceed with cqube deployment. #### Deployment Process **Step 1:** Connect to the cqube Azure ec2 instance **For linux and macOS:** * Download the .pem file which is generated while creating the EC2 instance * Open the terminal and navigate to the folder where .pem file has been downloaded * Then give the read permission to the .pem file using following command **sudo chmod 400 \** * Use the following command to connect to the instance **ssh -i \ \@\** **Ex:** ssh -i poc\_key.pem -o ServerAliveInterval=60 ubuntu\@13.200.12.31 **For windows:** * Download the .pem file which is generated while creating the EC2 instance * Use puttygen to connect to the instance. * Refer following link to use puttyGen for connecting. **Step 2:** Refer to the prerequisites checklist section before starting the cqube deployment.-[ Link ](https://app.gitbook.com/o/-Mi9QwJlsfb7xuxTBc0J/s/-Miy6UN-HyTKGo2Sjcky/~/changes/326/get-started-on-cqube/prerequisites-checklist) **Step 3:** Navigate to cqube-devops directory and give the executable permissions to the install.sh file **sudo chmod u+x install.sh** **Step 4:** Install cQube with non root user with sudo privileges **sudo ./install.sh**
Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cQube **Step 5:** Generation of domain specific configuration file - Please refer to the Hints provided and enter the values accordingly. * access\_type: Enter VSK, NVSK, or others. a) VSK: To set up VSK for a state b) NVSK: To establish a nationwide cQube deployment c) Others: To set up cQube for other purposes. * state\_name: Enter the required state code by referring to the state list provided (this variable needs to be filled if you are opting for access\_type as VSK; otherwise, it will get filled as NA by default). * loginpage\_status: Enter true if you want to enable the login screen for the CQUBE instance. Otherwise, enter false. * data\_pull\_status: Enter true if you want to pull data from an NVSK instance, or else enter false. (You need to fill this variable if you have selected access\_type as VSK; otherwise, it will get auto-filled with NA.) * nvsk\_api\_endpoint: Enter the NVSK instance api endpoint if you selected true for data\_pull\_status; otherwise, it will get autofilled with NA. Once all the above variables are filled, you get prompted with a preview of the domain-specific config file. Review the entered variables. If everything is correct, then type ‘no’ and proceed further, or else enter ‘yes’ and re-enter the variables correctly.
**Step 6**: Generation of a generic config file: refer to the hints provided and enter the config file accordingly. * mode\_of\_installation: Enter public as you have opted for setting up cqube in AWS. (Possible values are public and localhost.) * storage\_type: Enter storage type as AWS. (Possible values are aws, azure, oracle, and local.) * api\_end\_point: Enter the url in which the clock is to be configured. * nginx\_cert\_file: Enter the name of the SSL certificate file that you uploaded in the SSL\_certificates folder. * nginx\_key\_file: Enter the name of the private key file that you uploaded in the ssl\_certificates folder.
* Db\_credentials: You will be prompted with default database credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your database credentials. * db\_user\_name ( Enter the postgres database username ) * db\_name ( Enter the postgres database name ) * db\_password ( Enter the postgres password )
* Readonly\_db\_credentials: You will be prompted with default read-only database credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your read-only database credentials. 1. read\_only\_db\_user\_name ( Enter the postgres database username ) 2. read\_only\_db\_password ( Enter the postgres password )
* Keycloak\_credentials: You will be prompted with the default keycloak admin dashboard credentials. If you want to continue with the same, enter no; otherwise, enter yes and proceed to enter your keycloak credentials. 1. Keycloak\_adm\_name (Enter the keycloak admin name eg: admin) 2. Keycloak\_adm\_password (enter the keycloak admin password eg: Admin\@123)
**Step 7:** Once the config file is generated, a preview of the config file is displayed, followed by a question where the user gets the option to re-enter the configuration values by choosing ‘yes’. If option ‘no’ is selected, then the install.sh moves to the next section. **Step 8:** Generation of storage config file: Refer to the hints provided below and enter the config file accordingly. * Azure Container String: Enter the container string * azure\_account\_key: Enter the azure account key to access the bucket. * azure\_account-name: Enter the azure account name to access the bucket. * azure blob container name : Enter unique azure blob container name. **Step 9:** A preview of the program\_selector.yml file is displayed, followed by a question where the user gets an option to enable or disable the programs by choosing 'yes'. If option 'no' is selected, then the install.sh moves to the next section. 1. If access\_type is selected as VSK, the following programs will be displayed: by default, all programs are selected, and the programs are publicly visible.
**Note:** If you are willing to select the programs, type yes, select the programs you want, and enter public or private for each program to enable or disable the login screen for programs (follow the hints provided). 2. If access\_type is selected as NVSK, the following programs will be displayed: By default, all programs are selected, and the programs are publicly visible.
**Note:** If you are willing to select the programs, type yes, select the programs you want, and enter public or private for each program to enable or disable the login screen for programs (follow the hints provided). **Step 10:** Once all the configuration files are generated, the script moves further to clone the cQube microservice repositories and deploy cQube **Step 11:** If you selected data\_pull\_status as true, then you will see the data pulling APIs where the data gets pulled from the NVSK instance and processed in the VSK cQube. Once installation is done, we need to wait 20 minutes to visualize the data on the cQube dashboard. **Step 12:** Once the installation is completed, you will be prompted with the following messages and required reference urls: **(Note: The installation process is expected to take approximately 30-40 minutes.)** **cQube Installed Successfully**
# How to prepare the data The steps given in the next few sections need to be followed as it is and in the same sequence * Defining dimensions & ingesting them * Defining event files & ingesting them **The ingestion will NOT be successful if event files are ingested before the dimension files.** # Schema The schema in cQube is a json object which needs to be defined in a specific format. cQube expects a schema to validate the data file provided by the state team. The current cQube schema should be defined as follows. {​​ "program": "string", "input": {​​ "type": "object", "properties": {​​ "column\_name": {​​ "type": "string || number", "shouldnotnull": true } }, "required": \[ "column\_name" ] } }
The properties consist of the column names present in the data file. We can specify the type of the column. It currently supports two types: string and number. The shouldnotnull property indicates that null values are not allowed. The required key indicates that the column names specified inside an array are mandatory and should be present in the data file. # How to prepare schemas for dimension files Dimensions are the columns that describe events. The dimensions are those values which do not change frequently and remain constant. Example: State name **CQube adopts the Dimension Specification as defined below** 
"program": "school",//Name of the dimension
  "input": {
    "type": "object",
    "properties": {
      "school_id": {
        "type": "string",
        "shouldNotNull": true
      },
      "udise_code": {
        "type": "string",
        "shouldNotNull": true
      },
      "school_name": {
        "type": "string",
        "shouldNotNull": true
      },
      "schoolcategory_id": {
        "type": "string",
        "shouldNotNull": false
      },
      "schoolmanagement_id": {
        "type": "string",
        "shouldNotNull": false
      },
      "school_statecategory_id": {
        "type": "string",
        "shouldNotNull": false
      },
      "cluster_id": {
        "type": "string",
        "shouldNotNull": true
      },
      "cluster_name": {
        "type": "string",
        "shouldNotNull": true
      },
      "block_id": {
        "type": "string",
        "shouldNotNull": true
      },
      "block_name": {
        "type": "string",
        "shouldNotNull": true
      },
      "district_id": {
        "type": "string",
        "shouldNotNull": true
      },
      "district_name": {
        "type": "string",
        "shouldNotNull": true
      },
      "latitude": {
        "type": "number",
        "shouldNotNull": false
      },
      "longitude": {
        "type": "number",
        "shouldNotNull": false
      }
    },
    "required": [
      "school_id",
      "udise_code",
      "school_name",
      "cluster_id",
      "cluster_name",
      "block_id",
      "block_name",
      "district_id",
      "district_name",
      "latitude",
      "longitude"
    ]
  }
}
# How to prepare schemas for event files An event is a data structure that records an occurrence at a particular time for an entity (eg: school, etc). It is a combination of simple data types (eg: integer, varchar, etc.). An event should always contain a column/set of columns that helps you calculate the Indicator. A table with a timestamp doesn’t necessarily mean that it is an event; it should contribute to either aggregation or filtering of the dataset. ``` { “program”: “string”, // Name of the event “input”: { "type": "object", "properties": { : { "type": "string || number", "shouldnotnull": true, “format”?: “date” } } } Example: school-attendance "program": "teacher-attendance", "input": { "type": "object", "properties": { "date": { "type": "string", "shouldNotNull": true, "format": "date" }, "school_id": { "type": "string", "shouldNotNull": true }, "grade": { "type": "string", "shouldNotNull": true }, "district_id": { "type": "string", "shouldNotNull": true }, "block_id": { "type": "string", "shouldNotNull": true }, "cluster_id": { "type": "string", "shouldNotNull": true }, "schoolcategory_id": { "type": "string", "shouldNotNull": true }, "total_teachers": { "type": "string", "shouldNotNull": true } }, "required": [ "date", "school_id", "grade", "district_id", "cluster_id", "block_id", "schoolcategory_id", "total_teachers" ] } } ``` # Examples of dimension & event files This page gives an example of cQube Dimension & Event schema for better understanding ### **Dimension Schema** **Ex : School Dimension** This is the data for all schools, clusters, blocks and districts in the state. **If CSV is being ingested, the file name should be -> school-dimension.data.csv** | # | Column | Data type | Description | Validation | | -- | -------------------- | --------- | --------------------------------------------------- | ----------- | | 1 | district\_id | string | Unique ID in the district | NA | | 2 | district\_name | string | Name of the district | NA | | 3 | block\_id | string | Unique ID in the block | NA | | 4 | block\_name | string | Name of the block | NA | | 5 | cluster\_id | string | Unique ID in the cluster | NA | | 6 | cluster\_name | string | Name of the cluster | NA | | 7 | school\_id | string | Unique ID in the school | NA | | 8 | school\_name | string | Name of the school | NA | | 9 | schoolcategory\_name | string | ID of the school category for the respective school | NA | | 10 | grade | string | ID of the grade present in the school | NA | | 11 | latitude | string | Latitude of the school | NA | | 12 | longitude | string | Longitude of the school | NA | | 13 | udise\_code | string | Unique ID |


| ### Event Schemas: **Ex : Teacher Attendance** This program entails indicators to monitor compliance and performance of teacher attendance. Following are the reports within this program: * **Teacher Attendance Compliance:** This report shows the compliance of attendance being marked by teachers. * **Teacher Attendance Summary:** This report shows the summary of teachers being present in the school (out of the ones marking their attendance). Following schema will be required to enable the teacher attendance program: **If CSV is being ingested, the file name should be -> teachersattendance-event.data.csv** #### **Note: The name for grammar file & event file should be same. e.g: teachersattendance-event.data.csv , teachersattendance-event.grammar.csv** | # | Column Name | Data Type | Description | Validation | | -- | ---------------------------- | --------- | ---------------------------------------------------------------------- | --------------------------------------- | | 1 | date | string | Date when the data was recorded |

DD/MM/YY

format to be used

| | 2 | district\_id | string | Unique ID of the district as per the dimension table | NA | | 3 | block\_id | string | Unique ID of the block as per the dimension table | NA | | 4 | cluster\_id | string | Unique ID of the cluster as per the dimension table | NA | | 5 | school\_id | string | Unique ID of the school as per the dimension table | NA | | 6 | schoolcategory\_name | string | Category\_id which the school ID belongs to as per the dimension table | NA | | 7 | grade | string | Grade for which the data is being entered | NA | | 8 | total\_teachers | string | Total number of teachers | NA | | 9 | teachers\_attendance\_marked | string | Total number of teachers whose attendance was marked | NA | | 10 | teachers\_marked\_present | string | Total number of teachers who were present | NA | #### NOTE : Each metric should be part of a separate event table. E.g. In teacher attendance event file, we will have 3 tables: one for total\_teachers, one for teachers\_attendance\_marked & one for teachers\_marked\_present # All cQube schemas used for VSK Please check [here](https://docs.google.com/spreadsheets/u/0/d/10yWmivrqtxdRLiJpX_ytzxbvoGG0WNWEI-7RLV8zelM/edit) for all the schema. # Ingesting the data # High-level understanding of ingestion The diagram that explains the ingestion process.
# Steps to ingest schema ## **Note:** To see the API documentation please go to this link and paste the contents of the github links provided below. The example data provided in the request body is for understanding purposes and it is not the actual data. The API endpoint specified in the below document is for public mode of installation. ## Steps to add the schema for events ### Step 1: Create a postman request and add details **API Endpoint:** \ /api/spec/event **HTTP Method:** POST
### **Step 2:** Build the request body with reference to YAML file. The request body for the above API is attached here for yaml:[ ](https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml) Provide valid input details for the parameters shown below.
### **Step 3:** Click on the send button for request and if the request is successful the user should see a response message. Please refer to the below screenshot.
Once the API response is successful then the schema is stored in the spec.”EventGrammar” table. ## Steps to add the schema for dimensions ### Step 1: Create a postman request and add details **API Endpoint:** \ /api/spec/dimension **HTTP Method:** POST
**Step 2:** Build the request body with reference to YAML file. The request body for the above API is attached here for yaml:[ ](https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml) Provide valid input details for the parameters shown below.
### Step 3: Click on the send button for request and if the request is successful the user should see a response message. Please refer to the below screenshot. Once the API response is successful then the schema is stored in the spec.”DimensionGrammar” table.

# Steps to ingest data files **Difference between State Specific programs and National programs API** | State Specific Programs API | National Programs API | | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | |
  1. Accepts file in .csv format
|
  1. Accepts file in .zip format
| |
  1. It is mainly used when the state team wants to onboard new programs apart from the starter pack.
| 2. This API can be used for starter pack programs of VSK and NVSK | |
  1. The uploaded file will be present in the process\_input folder in cloud storage given that there are no errors in the data.
| 3. The uploaded file will be present in the emission folder in cloud storage given if there are no errors in the data | ### **Note:** The ingestion API’s have prerequisites to work. Whenever any file is uploaded it should have the specified schema stored in the DB. This can be done by making use of spec API’s. ## Generate JWT token API ### Step 1: Create a postman request & add details. **API Endpoint:** \/api/ingestion/generatejwt **HTTP Method:** GET
### Step 2: Click on the send button for request and if the request is successful the user should see a jwt token generated. Please refer to the below screenshot.
## **Note:** The token generated using the above API is passed in Authorization headers in all the ingestion API’s. If the token is not passed then the API will not work. To pass the token in postman go to the Authorization tab in postman and select the bearer token option from the drop down present on the left menu. On the right side paste the jwt token generated through API. Please refer to the below screenshot.
## # Upload the .csv data file for state specific programs using ingestion API Events, Dimensions can be uploaded into respective cloud storage via the ingestion APIs. The event upload is done for only state programs using this API. The jwt token has to be passed in the API headers. **Step 1**: create a postman request and add details. API Endpoint: \/api/ingestion/new\_programs HTTP Method: POST
This API will import the dimension csv and upload it into the process\_input folder in the cloud if there are no errors. **Step 2:** Build the request body with reference to YAML file. The request body for the above API is attached here for yaml:[ ](https://github.com/Sunbird-cQube/spec-ms/blob/march-release/spec.yaml) Provide valid input details for the parameters shown below. * file : Attach the CSV file for the importing * ingestion\_type : Specify the type of ingestion (value should be event when uploading the event data file. If it is a dimension data file it should be dimension) * ingestion\_name : Name of the event or dimension * program\_name: Name of the state program (This key is required only for ingestion\_type = ‘event’) Note: The value for the key program\_name should be the same name as the folder name present inside ingest->VSK or NVSK->programs folder in processing-ms. Example: The above link has a list of programs array. **Example:** For a school attendance program the folder name is school-attendance so the same name should be given while passing the value for program\_name. "files": "./ingest/VSK/programs/school-attendance" => The folder name can be identified as school-attendance.
### Step 3: Click on the send button for request and if the request is successful the user should see a response message. Please refer to the below screenshot.
After successful execution of the csv import api we get the response and we can see the file status indicating if the file is uploaded or not using GET file status API. If the file is successfully uploaded we will get the response as uploaded and if there are any errors it will send us the response indicating there was an error in the file.
# API to upload starter pack data files for NVSK programs This focuses on ingestion of programs like UDISE, PGI, NISHTHA, DIKSHA, NAS, PM-POSHAN. The jwt token has to be passed in the API headers. **Note:** There could be some common errors in the files which could lead to unsuccessful data ingestion into cQube. Look at these errors here and resolve them before ingesting data into cQube. The already existing files will need to be ingested into cQube via the following process: This API accepts the data in a zip file format and adds it to the emission folder in the respective cloud storage / SDC.
**National Programs API:** ### Step 1: Open the specified request & add the details **API Endpoint:** \/api/ingestion/national\_programs **HTTP Method:** GET
### Step 2: Send the query parameters with reference to the YAML file. The query parameters for the above api is attached in Link for yaml: [ ](https://github.com/Sunbird-cQube/spec-ms/blob/dev/spec.yaml)[](https://github.com/Sunbird-cQube/spec-ms/blob/dev/spec.yaml)[spec-ms/spec.yaml at dev · Sunbird-cQube/spec-ms](https://github.com/Sunbird-cQube/spec-ms/blob/dev/spec.yaml) Provide the valid input details for the parameters shown below.
### Step 3: Click on the send button for the request and if the request is successful the user should see a response message which contains the status of the file.
# Get file status API #### Here are the steps for the get file status API **Step 1:** Open the specified request & add the details API Endpoint: \/ingestion/file-status HTTP Method: GET
**Step 2:** Send the query parameters with reference to the YAML file. The query parameters for the above api is attached in Link for yaml: [ spec-ms/spec.yaml at dev · Sunbird-cQube/spec-ms](https://github.com/Sunbird-cQube/ingestion-ms/blob/dev/spec.yaml) Provide the valid input details for the parameters shown below.
**Step 3:** Click on the send button for the request and if the request is successful the user should see a response message which contains the status of the file.
If the csv files are too large in size, the upload process of those files will take more time which will make users wait for longer duration to receive the response from the API. As a result, the upload process will be running asynchronously and we have developed this api to know the status of the file at any particular time.
# Scheduled API This API helps to schedule the processor group at any particular time. **Step 1:** Open the specified request & add the details API Endpoint: \/api/spec/schedule HTTP Method: POST
**Step 2:** Build the request body with reference to YAML file. The request body for the above api is attached in Link for yaml:[ spec-ms/spec.yaml at dev · Sunbird-cQube/spec-ms](https://github.com/Sunbird-cQube/spec-ms/blob/dev/spec.yaml) . Provide the valid input details for the Parameters shown below.
### Step 3: Click on the send button for the request and if the request is successful the user should see a response message. The schedule time will be updated in the processor group.
The schedule api helps to run the processor group in Nifi at a scheduled time. The schedule time can be updated by changing the cron expression for scheduled\_at property in the request body. **Processor Group state Change API:** This API helps to change the state of a processor group in Nifi. It will provide the functionality to start the processor group or stop the processor group. **Step 1:** Create a specified request in postman and add the details.
**Step 2:** Step 2: Build the request body with reference to YAML file. The request body for the above api is attached in Link for yaml:[ spec-ms/spec.yaml at dev · Sunbird-cQube/spec-ms](https://github.com/Sunbird-cQube/spec-ms/blob/dev/spec.yaml) . Provide the valid input details for the Parameters shown below.
**Step 3:** Click on the send button for the request and if the request is successful the user should see a response message.
In the above example as you can see the processor group onestep\_dataingestion\_aws is changed to running state.\ Note: 1\. This API is only used to change the state of a processor group. If you want to change the schedule time you will have to use the Schedule API. 2\. It is suggested to stop the processor groups once the data is ingested for all the programs using the above API \
# Error during ingestion Errors may occur due to state code mismatches, the inclusion of special characters, variations in column names between grammar and event files, etc. | ID | State / UT Name | | -- | ------------------------------------ | | 1 | Andaman and Nicobar Islands | | 2 | Andhra Pradesh | | 3 | Arunachal Pradesh | | 4 | Assam | | 5 | Bihar | | 6 | Chandigarh | | 7 | Chhattisgarh | | 8 | Dadra & Nagar Haveli and Daman & Diu | | 9 | Delhi | | 10 | Goa | | 11 | Gujarat | | 12 | Haryana | | 13 | Himachal Pradesh | | 14 | Jammu and Kashmir | | 15 | Jharkhand | | 16 | Karnataka | | 17 | Kerala | | 18 | Ladakh | | 19 | Lakshadweep | | 20 | Madhya Pradesh | | 21 | Maharashtra | | 22 | Manipur | | 23 | Meghalaya | | 24 | Mizoram | | 25 | Nagaland | | 26 | Odisha | | 27 | Puducherry | | 28 | Punjab | | 29 | Rajasthan | | 30 | Sikkim | | 31 | Tamil Nadu | | 32 | Telangana | | 33 | Tripura | | 34 | Uttar Pradesh | | 35 | Uttarakhand | | 36 | West Bengal | | 37 | CBSE | | 38 | CISCE | | 39 | NCERT | ### ## ## # Error Monitoring The error file captures all error records during the ingestion process and is uploaded to the relevant cloud storage. Users can log in to the respective cloud storage platform, navigate to the designated bucket created during installation, and follow these steps to access the error file: 1. For the new\_program API: a) Error files are stored in the "ingestion\_error" folder within the bucket. b) Within "ingestion\_error," there are folders for each program (named after the program\_name). Inside these program folders, there is a sub-folder with the current date as its name. c) Users can access the current date folder, review the error files within, and download them to examine CSV errors. 2. For the national\_programs API: a) Error files are stored in the "emission\_error" folder within the bucket. b) Inside "emission\_error," a folder is created with today's date, containing the error files. After identifying and rectifying errors, users should re-upload the entire file. Screenshots illustrating how to access error files are provided below for reference.
To get the count of processed records and error records GET /ingestion/file-status API can be used.
# Common errors in data files during ingestion ### Single Master of Districts, Blocks, Clusters and Schools within a state to be referenced: * State and national programs on cQube will require data for districts, blocks, clusters and schools to enable decentralised observability. Dimension masters for district, block, cluster and school will need to be created having a unique ID and name of the jurisdiction. A single master file for each jurisdiction should be created and that should be referenced for data of both - state as well as national programs. Any discrepancy will lead to unsuccessful data ingestion. * For example: If a state has 30 districts, a district dimension master will have data in[ this](https://cqube.sunbird.org/data-ingestion-and-processing/cqube-schemas#district-dimension) format. The table will contain IDs from 1 to 30 with names of the district against each ID. Each district will be mapped to the state ID as per the 1st point. Same process will be followed for dimension masters for blocks, clusters and schools. ### Individual values for grade, subject and medium instead of arrays: * State and national programs in cQube will require data for grade, subject and medium being followed in the state. Generally the states share values of subject in arrays like \['Political Science/Civics', 'Social Science']. This is an incorrect format. * The subjects should have individual values and different subjects should be in different rows. For example: Political Science / Civics will be Row 1 with relevant data in the event file and Social Science will be Row 2 with relevant data in the event file. Dimension for subject in the specified format linked here will also have 2 rows with unique IDs - Political Science / Civics and Social Science. ### No quotes - single (') or double (“) should be present in the data being ingested * In all the files being ingested, single or double quotes should not be present at all # Processor group name The programs are individually scheduled by using scheduled API please run the following program name space. 1. nishtha 2. pgi 3. pm\_poshan 4. diksha 5. nas 6. udise 7. sch\_att 8. ncf 9. micro\_improvements 10. nipun\_bharat 11. ncert\_quiz 12. pm\_shri # Adapter details **Overview** **Metric:** A metric is a column in an event data file on which the aggregations are done to derive the insights. CQube adapter is an ETL (Extract, Transform and Load) pipeline with processes used to move data from the adapter database to multiple CSVs after making the required transformations. A cQube adapter is needed because cQube expects data in a specific format and the output CSVs of the adapter can be ingested directly into cQube to get the programs, reports and indicators. ## Working of an Adapter \[Currently working for National Programs] 1. The adapter makes a connection with the state data source (ex: azure container/aws s3 bucket/oracle file system /minio bucket).Read the zipped data file from the emission folder emission/\/\.csv. 2. It then reads the raw data files from the datasource.It performs the transformation to generate the Dimension and Event (Fact) CSV files. The desired format and output columns list in the dimension and event file for each program can be found here. 3. Select the required column from the report(zip file). 4. Split the files according to the number of metrics in report 5. Output Event CSV files will be stored inside AWS S3 Bucket / Minio / Azure in the input-bucket process\_input/program/\/\-event.data.csv. process\_input/program/\/\-dimension.data.csv Format. 6. NiFi will run This adapter ETL pipeline will run at a specific frequency so that the output CSV data can be refreshed and the latest data will be ingested into the system. Example for illustration: 1. **Initial file**
datedistrict_idblock_idcluster_idschool_idschoolcategory_namegradegenderKPI-1KPI-2KPI-3











2. **Final files** | date | district\_id | block\_id | cluster\_id | school\_id | schoolcategory\_name | grade | gender | KPI-1 | | ---- | ------------ | --------- | ----------- | ---------- | -------------------- | ----- | ------ | ----- | | | | | | | | | | |
datedistrict_idblock_idcluster_idschool_idschoolcategory_namegradegenderKPI-2









| date | district\_id | block\_id | cluster\_id | school\_id | schoolcategory\_name | grade | gender | KPI-3 | | ----------- | ------------ | ----------- | ----------- | ----------- | -------------------- | ----------- | ----------- | ----------- | |


|


|


|


|


|


|


|


|


| ## Architecture
## Technology Used cQube adapter can use any system, programming language or ETL tool to develop the cQube adapter. For example: * Python scripts can be used to extract data from the source / state database, transform it and finally export the CSV files inside the AWS S3 bucket or cloud storage which is being used. Apache Airflow can be used to scheduling the python scripts. * Or, Apache NiFi can be used to create the end-to-end ETL Pipeline. * The only requirement is that the adapter-generated CSV files should have the same column names and the data format as per schema\ \ Refer this [link](https://docs.google.com/document/d/1F9ho_1y3sWDCzynPAzLScYT18VmuS4K-1dNOWs1z6rE/edit#heading=h.y16u3gcmp1l8) for detailed explanation # Postman details ## Setting up Postman * [Download](https://www.postman.com/downloads/) the postman application and import the collection. * Select the import option in postman to import the collection. Please refer to the screenshot.
## Postman collection files {% file src="/files/EcMHOLzrSyli3j8XNAPa" %} {% file src="/files/6L9pok5teROmunR5TvPl" %} # Processing of data The Following sections covered in this topic: Nifi data API details for Nifi-Rest # Data Processing using CLI command **Ingestion of Dimensions:** For the ingestion of dimension, we have to make sure, we have grammar and data files present in the below directory inside the server. `/ingest//dimensions` Run the below command from the /ingest/ folder to upload the dimensions in the database. `yarn cli ingest` * It will read the config.json file of a specific program type present inside the ingest folder. * Then it will process all the dimension grammar present in the dimensions folder. * The dimension grammars are stored in the “spec.dimensionGrammar” table and the dimensions tables are created in the dimensions schema. * It will also look for data files respective to each dimension grammar file name and ingest all the dimension data to the respective tables. * After the dimensions are ingested the programs array present in config.json is read and the event grammars are processed from the corresponding \ folder. The event grammars are stored in the spec.”EventGrammars” table. * The dataset grammars are also stored in the spec.”datasetGrammars” table and the dataset tables are created based on the combination of timeDimension, dimension and metric present in the event grammars. * In addition to the above combination of datasets created the user can also specify the combination of datasets that can be created in the whitelist array. The below screenshot shows the final output of the yarn cli ingest command when all the dimension,event and dataset grammars are processed. The dimension data is also ingested in this process.
**Note:** When you ingest dimensions everything gets deleted and re ingested. So to update the dimensions data, you should reupload the full dimensions data file and run the command (yarn cli ingest) to ingest dimensions **Ingestion of Program Data** For the ingestion of events data, we have to make sure, we have grammar and data files present in the below directory respective to each program inside the server. `/ingest//programs` The value for program type is read from the .env file and it can have two values VSK or NVSK. `yarn cli ingest-data` This command will ingest the data to the dataset tables for all the programs. It also provides an option to ingest the data for the particular program yarn cli ingest-data ---filter=\. The program name passed should be the same as the namespace specified in the config file. * It will read all the programs present in the config.json file. * It will check for the data files inside the programs folder and process the data. * Then the process dataset update request is created. * The transformer processes the request and updates the data to the dataset tables created. These datasets are used for visualization in the UI. Below command can be used to upload the data for PGI program. Here in the filter we are passing pgi. `yarn cli ingest-data --filter='pgi’`
The above figure shows the ingestion of data for the program PGI. The data ingested can then be used by the visualization to derive statistical data. **Delete the data in the database** `yarn cli nuke-db`: This command will delete all the data present in the database. # API Details for Nifi-Rest [Nifi-Rest-api’s code](https://github.com/Sunbird-cQube/generator-ms/blob/dev/static_processor_group/add_nifi_template.py) # Nifi section Nifi in cQube is used to create a data flow and automate the processing of data with the help of processor groups. These processor groups are responsible for accessing the data files from the cloud storage and moving it into the processing-ms codebase. Once the data files are moved the commands will be executed which will ingest the data into the database. Here we have different processor groups to process the data(Nifi canvas).\ 1\. To run adapters. 2\. To run all programs. 3\. To run program wise. The Processor groups in nifi can be scheduled using API called [scheduled API ](https://cqube.sunbird.org/use-cqube/ingesting-the-data/steps-to-ingest-data-files/scheduled-api)
**1. To run adapters:** This processor group is used to run the adapters shell script code.
**2. To run all programs:** This processor group is used to move files from process\_input folder to processing\_ms folder according to folder structures. .i.e., (dimensions/,programs/). Then it will run yarn cli ingest(where it will ingest all schemas and grammars to the database) and yarn cli ingest-data (where it will ingest all data files to the database) commands.
**3.To run program wise:** This processor group is used to move particular program files from process\_input folder to processing\_ms folder. Then it will run yarn cli ingest and yarn cli ingest-data --filter=’program\_name’.(where it will ingest particular program data files).
\ In order to automate these processor groups we have written REST-APIs. Please look into the below link to understand th Nifi codebase. \ [Nifi-Rest-api’s code](https://github.com/Sunbird-cQube/generator-ms/blob/dev/static_processor_group/add_nifi_template.py)
# Visualizing the data # High level understanding of how visualizations work in cQube
# Programs and reports out-of-the-box Details out the indicators in each report and program available in cQube Ed V 5.0 There are a total of 9 programs available in cQube v5.0: 1. [Student Attendance](#1.-student-attendance) 2. [NISHTHA](#2.-nishtha) 3. [DIKSHA](#3.-diksha) 4. [PM POSHAN](#4.-pm-poshan) 5. [NAS](#5.-nas) 6. [UDISE](#6.-udise) 7. [PGI](#7.-pgi) ### 1. Student Attendance This program entails indicators to monitor compliance and performance of student attendance. Following are the reports within this program. * **Schools Reporting Student Attendance:** This report shows the % of schools that report attendance within the selected time duration for more than 50% enrolled students on average. This report will have the Date Range filter. Other Filters like District, Block, Cluster, School will be picked up from RBAC. Following indicators will be available for specific roles as part of this report: For a State Officer:
IndicatorChart TypeApplicable Filters
Average % Schools Reporting Student AttendanceBig NumberDate Range
District-wise % Schools Reporting Student AttendanceTableDate Range
For a District Officer:
IndicatorChart TypeApplicable Filters
Average % Schools Reporting Student AttendanceBig NumberDistrict, Date Range
Block-wise % Schools Reporting Student AttendanceTableDistrict, Date Range
District-wise Rank in % Schools Reporting Student AttendanceTableDate Range
For a Block Officer: | Indicator | Chart Type | Applicable Filters | | --------------------------------------------------------- | ---------- | --------------------------- | | Average % Schools Reporting Student Attendance | Big Number | District, Block, Date Range | | Cluster-wise % Schools Reporting Student Attendance | Table | District, Block, Date Range | | Block-wise Rank in % Schools Reporting Student Attendance | Table | District, Date Range | For a Cluster Officer: | Indicator | Chart Type | Applicable Filters | | ----------------------------------------------------------- | ---------- | ------------------------------------ | | Average % Schools Reporting Student Attendance | Big Number | District, Block, Cluster, Date Range | | School-wise % Student attendance being reported | Table | District, Block, Cluster, Date Range | | Cluster-wise Rank in % Schools Reporting Student Attendance | Table | District, Block, Date Range | For a School Principal: | Indicator | Chart Type | Applicable Filters | | ------------------------------------------------------- | ---------- | -------------------------------------------- | | Average % Student attendance being reported | Big Number | District, Block, Cluster, School, Date Range | | Grade-wise % Student attendance being reported | Table | District, Block, Cluster, School, Date Range | | School-wise Rank in % Student attendance being reported | Table | District, Block, Cluster, Date Range | * **Average Students Present:** This report shows the % of students present (out of the ones for whom attendance is being reported) within the selected time duration on an average. This report will have the Date Range filter. Other Filters like District, Block, Cluster, School will be picked up from RBAC. Following indicators will be available for specific roles as part of this report: For a State Officer:
IndicatorChart TypeApplicable Filters
Average % Students PresentBig NumberDate Range
District-wise % Students PresentTableDate Range
Gender-wise % Students PresentBar ChartDate Range
Grade-wise % Students PresentBar ChartDate Range
For a District Officer: | Indicator | Chart Type | Applicable Filters | | ---------------------------------------- | ---------- | -------------------- | | Average % Students Present | Big Number | District, Date Range | | Block-wise % Students Present | Table | District, Date Range | | District-wise Rank in % Students Present | Table | Date Range | | Gender-wise % Students Present | Bar Chart | District, Date Range | | Grade-wise % Students Present | Bar Chart | District, Date Range | For a Block Officer: | Indicator | Chart Type | Applicable Filters | | ------------------------------------- | ---------- | --------------------------- | | Average % Students Present | Big Number | District, Block, Date Range | | Cluster-wise % Students Present | Table | District, Block, Date Range | | Block-wise Rank in % Students Present | Table | District, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Date Range | | Grade-wise % Students Present | Bar Chart | District, Block, Date Range | For a Cluster Officer: | Indicator | Chart Type | Applicable Filters | | --------------------------------------- | ---------- | ------------------------------------ | | Average % Students Present | Big Number | District, Block, Cluster, Date Range | | School-wise % Students Present | Table | District, Block, Cluster, Date Range | | Cluster-wise Rank in % Students Present | Table | District, Block, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Cluster, Date Range | | Grade-wise % Students Present | Bar Chart | District, Block, Cluster, Date Range | For a School Principal: | Indicator | Chart Type | Applicable Filters | | ------------------------------ | ---------- | -------------------------------------------- | | Average % Students Present | Big Number | District, Block, Cluster, School, Date Range | | Grade-wise % Students Present | Table | District, Block, Cluster, School, Date Range | | School-wise % Students Present | Table | District, Block, Cluster, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Cluster, School, Date Range | For a Class Teacher: | Indicator | Chart Type | Applicable Filters | | ------------------------------------- | ---------- | --------------------------------------------------- | | % Students Present | Big Number | District, Block, Cluster, School, Grade, Date Range | | Grade-wise Rank in % Students Present | Table | District, Block, Cluster, School, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Cluster, School, Grade, Date Range | * **Average Students Present:** This report shows the % of students present (out of the ones for whom attendance is being reported) within the selected time duration on an average. This report will have the Date Range filter. Other Filters like District, Block, Cluster, School will be picked up from RBAC. Following indicators will be available for specific roles as part of this report: For a State Officer:
IndicatorChart TypeApplicable Filters
Average % Students PresentBig NumberDate Range
District-wise % Students PresentTableDate Range
Gender-wise % Students PresentBar ChartDate Range
Grade-wise % Students PresentBar ChartDate Range
For a District Officer: | Indicator | Chart Type | Applicable Filters | | ---------------------------------------- | ---------- | -------------------- | | Average % Students Present | Big Number | District, Date Range | | Block-wise % Students Present | Table | District, Date Range | | District-wise Rank in % Students Present | Table | Date Range | | Gender-wise % Students Present | Bar Chart | District, Date Range | | Grade-wise % Students Present | Bar Chart | District, Date Range | For a Block Officer: | Indicator | Chart Type | Applicable Filters | | ------------------------------------- | ---------- | --------------------------- | | Average % Students Present | Big Number | District, Block, Date Range | | Cluster-wise % Students Present | Table | District, Block, Date Range | | Block-wise Rank in % Students Present | Table | District, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Date Range | | Grade-wise % Students Present | Bar Chart | District, Block, Date Range | For a Cluster Officer: | Indicator | Chart Type | Applicable Filters | | --------------------------------------- | ---------- | ------------------------------------ | | Average % Students Present | Big Number | District, Block, Cluster, Date Range | | School-wise % Students Present | Table | District, Block, Cluster, Date Range | | Cluster-wise Rank in % Students Present | Table | District, Block, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Cluster, Date Range | | Grade-wise % Students Present | Bar Chart | District, Block, Cluster, Date Range | For a School Principal: | Indicator | Chart Type | Applicable Filters | | ------------------------------ | ---------- | -------------------------------------------- | | Average % Students Present | Big Number | District, Block, Cluster, School, Date Range | | Grade-wise % Students Present | Table | District, Block, Cluster, School, Date Range | | School-wise % Students Present | Table | District, Block, Cluster, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Cluster, School, Date Range | For a Class Teacher: | Indicator | Chart Type | Applicable Filters | | ------------------------------------- | ---------- | --------------------------------------------------- | | % Students Present | Big Number | District, Block, Cluster, School, Grade, Date Range | | Grade-wise Rank in % Students Present | Table | District, Block, Cluster, School, Date Range | | Gender-wise % Students Present | Bar Chart | District, Block, Cluster, School, Grade, Date Range | ### 2. NISHTHA NISHTHA is a capacity building program for "Improving Quality of School Education through Integrated Teacher Training". It aims to build competencies among all the teachers and school principals at the elementary stage. NISHTHA is the world's largest teachers' training program. Following reports are available in this program: 1. Implementation Status 2. Courses and Medium status 3. % against Potential Base 4. District wise Status 5. Course wise Status ### **3. DIKSHA:** The Digital Infrastructure for Knowledge Sharing is an initiative of the National Council of Educational Research and Training (Ministry of Education, Govt. of India). This platform enables implementation of two key programs- NISHTHA and Energized Textbooks. Following reports are available in this program: 1. ETB Coverage Status 2. Content Coverage on QR 3. Learning Sessions on Potential Users ### **4. PM POSHAN:** Pradhan Mantri Poshan Shakti Nirman (PM POSHAN) earlier known as the National Programme of Mid-Day Meal in Schools is one of the foremost rights based Centrally Sponsored Schemes under the National Food Security Act, 2013 (NFSA). The primary objective of the scheme is to improve the nutritional status of children studying in classes I-VIII in eligible schools. Following report is available in this program: 1. Progress Status ### **5. NAS:** The National Achievement Survey (NAS) is a nationally representative large-scale survey of students' learning undertaken by the Ministry of Education, Government of India. Following reports are available in this program: 1. District Wise Performance 2. Grade & Subject performance ### **6. UDISE:** The Unified District Information System for Education (UDISE) is an educational management information system collected by school administrators on a yearly basis. Following reports are available in this program: 1. District Wise Performance 2. Correlation ### **7. PGI:** The Performance Grading Index (PGI) is a tool to provide insights on the status of school education in States and Union territories including key levers that drive their performance and critical areas for improvement. Following report is available in this program: 1. District Wise Performance # Enhance /Customize cQube # Available customizations Describes how customizations on the login (UI) layer of cQube Ed can be made cQube V 5.0 offers customizations on the UI layer of the login page. Post making these customizations, the UI layer of cQube will need to be redeployed. Please see the nested pages to view the available customizations in cQube UI # Changing Dashboard Logos and Headers There are 5 configurations enabled here for the dashboard pages. All of these configurations will be done in the UI configuration file, which can be found in the dashboard-ms micro service. You can access this file in this manner: dashboard-ms > src > assets > config > ui\_config.json. There are certain keys specified for each configuration to be changed as mentioned below. If there is a need to change any of these configurations, the users need to go to specified location directory (in ui\_config.json) as shown below and make the changes:
**dashboard\_header1\_title** : This key is used to change the title of the cQube dashboard page mentioned at the top. It is depicted in the image below:
**dashboard\_header1\_image** : This key is used to change the image of the cQube dashboard page present at the top. The size of the image should be 32px \* 32px. It is depicted in the image below:
dashboard\_header1\_image The user needs to upload the image into the specified directory: dashboard-ms > src > assets > images \ **dashboard\_header2\_title** : This key is used to change the state name in the title of the cQube dashboard. It is depicted in the image below:
dashboard\_header2\_title **dashboard\_header2\_image1** : This key is used to change the first image next to the title of the cQube dashboard. The size of the image should be 45px \* \. It is depicted in the image below:
dashboard\_header2\_image1 The user needs to upload the image into the specified directory: dashboard-ms > src > assets > images dashboard\_header2\_image2 : This key is used to change the second image next to the title of the cQube dashboard. The size of the image should be 45px \* 45px. It is depicted in the image below:
dashboard\_header2\_image2 The user needs to upload the image into the specified directory: dashboard-ms > src > assets > images Post making the changes, the user needs to redeploy the code again as explained below. ### Steps for redeploying cQube UI post changes Run the commands below in the given order to redeploy the UI config file changes for them to be reflected in the visualization layer: `sudo docker stop dashboard_app` `sudo docker rm dashboard_app` `sudo docker rmi dashboard_ms:1` `cd cqube-devops/microservices/dashboard-ms` `sudo git pull` `sudo docker build -t dashboard_ms:1 .` `sudo docker run -d -p 4200:80 --network cqube_net --name dashboard_app dashboard_ms:1` # Changing Program Name, Icon and Side Menu Sequence The name of the programs, icons and sequence of the side menubar can be changed as explained below.
This can be changed in the menu configuration file present in the query-builder micro service (query-builder > menus.csv).
The users can change the menu sequence and program name in this file. They need to add the required details mentioned in the CSV to add a new program in the menu bar. If they add / change any icon for the menu, they need to upload the icon images into the following directory dashboard-ms > src > assets > images The sample CSV filled is shown below:
Post making the changes, the user needs to redeploy the code again as explained below. ### Steps for redeploying cQube UI post changes Run the commands below in the given order to redeploy the menus.csv file changes for them to be reflected in the visualization layer: `sudo docker stop querybuilder_app` `sudo docker rm querybuilder_app` `sudo docker querybuilder_ms:1` `cd cqube-devops/microservices/querybuilder-ms` `sudo git pull` `sudo docker build -t querybuilder_ms:1` `sudo docker run -d -p 3002:3002 --network cqube_net --name querybuilder_app querybuilder_ms:1` \
# Adding a new KPI A shared BigNumber component is available in the below path src/app/shared/components/big-number * Create a new angular Component inside a Program following the below folder structure.
* Add the common component in the html file. * In the component typescript file add the getReportData Function which fetches the queries according to the configurations. For reference follow this file src/app/views/teacher-attendance/pages/teacher-attendance-summary/reports/tas-average-attendance-bignumber/tas-average-attendance-bignumber.component.ts * Add the configuration to the program config files found in the config folder.

* label is the tab/report name the current kpi belongs to * filter is an array of objects, where each object corresponds to the rbac level containing specific details for each levels 1. hierarchyLevel is the mapping of the current level to the logged in Level. 2. timeSeriesQueries hold key/object pairs where key is the chart type and value is the query for fetching the data for that chart. They are fetched when date time filter is involved 3. Similar to timeSeriesQueries queries hold the queries for non-date time filter scenarios 4. level inside the actions just specifies the name of next level in the given hierarchy. 5. The bigNumber object inside options contains extra options to be sent to that specific chart type. 6. title is the bigNumber titles to be displayed 7. valueSuffix is the suffix to the bigNumber Value to be displayed property is the column name/ property name in the output of the query to be displayed as the BigNumber. # Adding a Map KPI into dashboard ms A shared map component using leaflet is available in the below path `src/app/shared/components/maps/leaflet-map`
* Create a new report component and add the above shared component in the html files with any additional elements if needed \ For reference follow the below file: src/app/views/teacher-attendance/pages/teacher-attendance-summary/reports/teacher-attendance-map/teacher-attendance-map.component.html * In the component ts file add the required functions (ex: get reportData, apply criteria, drilldown) respective to each feature. **For reference follow the below file:** src/app/views/teacher-attendance/pages/teacher-attendance-summary/reports/teacher-attendance-map/teacher-attendance-map.component.ts * Options available in the configuration: * indicatorType specifies whether the kpi is of type percent or any general value type. * indicator represents the property name of the kpi on which map has to be based on. (property name is nothing but alias name in the query) * title specifies the title to be displayed in the legend * tooltipMetrics is an array of objects which is used to construct tooltip.It contains a property name/value which if present in the data will be added along with its prefix and suffixes to the tooltip element.
Above options are applicable for certain scenarios where metric filter is not needed. In a case where a metric filter is needed few extra options are available in the configuration. * metricFilterNeeded is a flag which can be set true if a metric filter is involved in the map report. * groupByColumn is a property by which the data from the queries can be grouped on a given level. * metricValueProp specifies the value property to identify the value of a given metric option. * metricLabelProp specifies the label of the metric option. Example for the above scenario can be found below:
Group by is used to group all metrics for a given district/block/cluster into a single row.
**Before Group By:**
\ **After Group By:**
To add drill down features just follow the steps in the table i.e to subscribe to the drilldown service in the map report component . For reference follow below report: src/app/views/teacher-attendance/pages/teacher-attendance-summary/reports/teacher-attendance-map/teacher-attendance-map.component.ts \
# Table Drill Down Customization A service named report-drill down-service has been created to maintain the state of the drilldown in a given program, which can be found in the below path of the dashboard ms codebase: `src/app/core/services/report-drilldown/report-drilldown.service.ts` In order to implement the drill down feature in a given table report, subscribe to the above service in the ngOnInit lifecycle function as shown below.
Add the drilldown function to the table report component file. (In future this function will be placed in the service itself). For reference the function can be found in the below file: src/app/views/teacher-attendance/pages/teacher-attendance-summary/reports/tas-average-attendance/tas-average-attendance.component.ts To enable clickable level columns in the table add the below configuration into the column object of the table report configuration.
Here are the steps: * dataProps is to specify what property name the column be and alias name of that property. * hierarchyLevel specifies which level the current column is in. * linkedReports represents which other reports will be affected by the drilldown of this table. * allowedLevels restricts the drill down to specified levels in the array To Link a report to drill down features of another report, subscribe to the service like above and add the drilldown function to the report component file.
For reference follow the implementation in the below file src/app/views/teacher-attendance/pages/teacher-attendance-summary/reports/average-attendance-school-table/average-attendance-school-table.component.ts
# Adding a Scatter Plot KPI into dashboard ms A sunbird dashlet library is being used to plot scatter charts across the application. * Create a new report component and add the above library selector in the html files with any (additional elements if needed) **For reference follow the below file:** src/app/views/udise/pages/correlation-tab/reports/correlation/correlation.component.html * In the component ts file add the required functions (ex: get reportData, apply criteria, drilldown) respective to each feature. **For reference follow the below file:** src/app/views/udise/pages/correlation-tab/reports/correlation/correlation.component.ts A common function getScatterChartReportData() is used to get the data based on the configuration which will be then passed to the dashlet library. The chart configurations present are based on the chartJS requirements which is used within the sb-dashlet library. **The above function can be found in the below file:** src/app/core/services/data.service.ts \ **Options available in the specifically for the scatter plot configuration:** * groupByNeeded specifies whether the group by is needed upon the data received from the query * groupByLabel specifies property name on which group by will be done * metricLabelProp specifies the property which will be converted as key after group by * metricValueProp specifies the property which will be converted as value after group by * valueSuffix is a suffix string which can be attached after the value in the tooltip, EX: % * tooltipMetrics is the configuration required to develop custom tooltips. It is an array of objects, in which each object represents a line in the tooltip (except the x-axis and y-axis values). **Each object consists of 3 properties as follows:** * valuePrefix is the prefix string before the value. * value is the property name of the value in the query result * valueSuffix is the suffix string after the value.
* Filters for the Scatter Plot (X-axis and Y-axis) can be configured as shown below.
**Options for filter configuration:** * label is the tab name under which the filter will be included * name is the placeholder name that will be displayed above the filter in the UI. * labelProp specifies the property in the query result for dropdown names. * valueProp specifies the property which will be mapped with the dropdown values. * id is just an identifier for each filter. * axis specifies which axis (out of x and y) this filter corresponds to. * query as the name suggests, holds the query which fetches the filter data. # Configure default date range across app/specific report Currently In cQube dashboard default date range is set to 7 days across app. If needed one can change the number of days for default selection across the app as shown below.
The above defaultSelectedDays can be set in the time-series-filter-panel in the shared folder. However if one wants to override and have custom default range for a specific report they can follow the below steps:
One can pass an input as shown above to the time-series-filter-panel html selector in a specific report html file.
# How to add a New Indicator Note: These steps are for adding a new indicator to an existing program 1. Navigate to the component created for the given report and into the "reports" folder. 2. Create a child component using the following CLI command: `ng g c `. 3. Once the component is created, fill the HTML and TypeScript files with references to previous indicators or with Angular/implementation knowledge. 4. Later, in the configuration file of that program, add an object with a unique key for that indicator. 5. Fill the configurations according to the reference again with the older indicators and fill the queries using the datasets ingested for this indicator for every level that this indicator is required for.
6. Based on the chart type, you can also provide options in the same configurations, such as titles, color of the columns, legend of the map report, etc.
# How to add a new report in an existing program Note: These steps are for adding a new report to an existing program 1. Navigate to the respective program in the client-side code base where the report has to be added. 2. Inside the "pages" folder, create an Angular component named after the report using the following CLI command: `ng g c `
3. Once the component is created, it will contain four files - `ts`, `spec-ts`, `html`, and `css`. 4. Fill the TypeScript and HTML files with the reference of other programs. You may need to have knowledge about Angular and how functions have been designed. 5. Include this component in the program HTML file to be rendered. 6. Under the same program name, there will be a config folder that contains a configuration file for the entire program (a JSON file). 7. If any filters are required for the specific report, add an object under the "filter" key in the above configuration file which specifies the filter and its values required for the report. You may need Angular knowledge to fill the object with the appropriate structure.

# How to add a new program (end to end) Note: These steps are for adding a new program ### Create event and dimension Grammar files: Design the event and dimension grammar files for your program. This grammar defines the structure and relationships between different data elements, such as event attributes and dimensions. * Event grammar: The event grammar defines the structure of the events to be ingested into the system. It includes information such as the event name, its attributes (e.g. event timestamp, user ID, etc.), and data types. * Dimension grammar: The dimension grammar defines the dimensions, or attributes, that will be used to group and filter events. It includes information such as the dimension name, its attributes (e.g. dimension value, category, etc.), and data types. ### Create a Config.json Config.json provides directive to processing engine on how to process the grammar and data files, where they are stored, etc. it should be placed in /ingest/\/ globals : global settings dimensions : It includes, file format, location and namespace programs : program specific settings like namespace, description, input location, output location, dimensions which should be considered etc. \ Only programs mentioned under config.json will be considered for ingestion.
### 15.3 Upload the event and dimension grammar spec: * Convert the Event and Dimension Grammar to JSON format as given in the below example Example: if there is a event with two columns school\_id and school\_name Below would be the JSON object { "program": "school-attendance", "input": { "type": "object", "properties": { "school\_id": { "type": "string", "shouldNotNull": true }, "school\_name": { "type": "string", "shouldNotNull": true } } } * Dimension grammar : Use the /spec/dimension API to upload the dimension grammar * Place the dimension grammar file as -dimension.grammar.csv in processing-ms/impl/c-qube/ingest/VSK or NVSK/dimensions/ folder as seen below or as per the config.json settings
**Contents of the district-dimension.grammar.csv**
* Event schema can be ingested in 2 ways: * Though API Method **Event spec API: spec/event**
* Through YARN CLI YARN CLI INGEST This command is used to ingest Event and Dimension Spec and Dimension Data based on the configuration we have * Create a \ folder in processing-ms/impl/c-qube/ingest/\/programs\\ * Place the event grammar file as \-event.grammar.csv in processing-ms/impl/c-qube/ingest/\/programs/\ folder as seen below (example for diksha program) This would be based on default config.json or else it should be as per config.json
Contents of the district-dimension.grammar.csv
### Data Ingestion: This process may involve uploading the CSV file or connecting to the database directly. Following links will provide more details on the process. * [Uploading through CSV](https://github.com/Sunbird-cQube/community/blob/master/ui-configurations/broken-reference/README.md) * [Connecting to Database](https://github.com/Sunbird-cQube/community/blob/master/ui-configurations/broken-reference/README.md) **Add event data on a regular frequency:** To keep your program up-to-date, continuously add new event data to the platform. Determine an appropriate frequency for updating the data, such as daily, weekly, or monthly.w Ensure that the new data is compatible with the existing event and dimension schema. ### UI Program Addition * Create a new module manually or using cli common ng g m \ in the below folder in the angular code base: src/app/views * Add the folders config and pages, where config holds a configuration file for the reports in this program and pages contains the code of all the tabs/reports. * Add the routing to the program component in the app-routing module using lazy-loading
* Add the configuration of this program to the main\_matric.csv present in the root folder of query builder.( To add this program to side nav and summary statistics) as shown below.
* Next add the programId filled in the above file to another file Program Selector in the query Builder root folder itself. The Program Selector file takes input from the ansible during installation to update which programs to be visible in the dashboard. If a new program is added after installation manually fill the programId and set the status true for that program to be visible in the dashboard.
* Configuration file of the newly added program should be added to the below file: src/app/core/config/configMapping.ts All the configuration files are mapped at a place for easy access , This object will be used to map each row in main\_metric with program wise configuration files with the respective programIds.
Later one can add the desired KPI to the above program created following the below steps # Additional Features Here we can see some of the additional features of the cQube # Public/Private dashboards During the installation, based on the inputs of the installer, access to the access for the program will be set to either public or private. Public Access indicates that the program can be accessed via a public dashboard with a login as a user. Private Access indicates that the program can be accessed only via a private dashboard, which is once any private or admin user is logged in. Keycloak Admin can override the access from the keycloak admin console by following the below steps. 1. Navigate to the roles tab. Select whichever role admin wants to modify the program access to.
2. Once on the selected role page, select the attributes tab.
3. Modify the value of the attribute named key, which is a string consisting of comma separated program Ids and save to apply the changes.
4. Any role to be added can be appended to that string with a comma between each programId, or any program ID removed will no longer be accessible to the selected role. 5. Currently configured Program ID’s can be found in a table of spec schema: **spec.DatasetGrammar.** Also following are the IDs that are currently available and configured program IDs within cQube: * nishtha * pm\_poshan * nas * diksha * pgi * udise * sch\_att * nipun\_bharat * ncert\_quiz * micro\_improvements * telemetry # Role based access control During the installation of cQube, three roles will be created within keycloak. 1. Private\_user 2. Guest (Default) 3. Admin Any roles created will be associated with guest roles by default. For accessing any private programs, users should be assigned private\_user. Any user who needs access to the admin panel as well as the telemetry dashboard will have to be assigned the admin role. One can be assigned both private, guest, and admin roles, through which he will have access to every program within the dashboard. During the installation, two users, one guest and an admin—will be created and assigned appropriate roles. Any user once created can navigate to the role mapping tab, select the required role from the available roles list, and add it so it will be associated with the user.
# Saving geographical preferences Any private user can have their priorities and preferences set for the first time, and whenever he logs into the dashboard, preferences will be automatically selected and the respective dashboards associated with the previously selected role will be shown. Any user who wants his preference to be saved for the next login can check the save my preferences checkbox before submitting the role and level details.
Once selected, the options for a particular role will be assigned to the user, and when the same user logs in again, they will be redirected directly to the dashboard with preselected values saved with the user. If any user wants to override the preference, they can click on the home button and reselect the options that will be saved for the next login. # Admin Panel It is a web application that allows gives user essential and important tools for managing& using cQube Features of the admin console include the following: 1. System Monitoring 2. Data debugger 3. Schema creator It is accessible for an admin user only through a separate URL( Ex: https\://\/admin). \
# Data Debugger It analyzes the data and reports the user about the errored records in the raw data file of an event or dimension. User has to select an event or dimension schema which is needed to validate the records of uploaded events or dimension files. After that debugger will analyze the uploaded data against the selected schema and it reports back to the user with the errored records. This feature helps the user to analyze the data before ingesting the data for data processing. Data Debugger analyzes the fed data and reports and identifies the errors in them, if any, in an event or dimension files. **How it works:** To validate the raw data of either dimension or an event, first: 1. Upload the grammar file for the dimension or event that you want to validate. 2. Validate the grammar file. 3. Validate the raw data against the uploaded grammar file. **Steps to validate the Raw Data file:** (Please note - The raw data file should be in CSV format.) 1. Check 'Dimension Grammar' is present in the dropdown menu. 2. Upload the dimension file. 3. Click on the 'Validate Grammar' Tab. Now, the system will show errors if any, in the screen for the uploaded file. If the file does not contain any errors, the user will be directed to the next screen. Here are more information about each step above. 1. **Upload the grammar file for the dimension or event that you want to validate.** The grammar file is a CSV file that defines the structure of the data that you want to validate. It contains information about the different fields in the data, such as the field name, data type, and constraints. Following screenshot of validate grammar of data debugger
Following is the sample grammar file block dimension:
**Note**: Include primary key if it is a dimension grammar file. 2. **Validate the grammar file.** Once you have uploaded the grammar file, you need to validate it to make sure that it is correct. The validation process will check for errors in the grammar file, such as missing fields or invalid data types. Here is a sample screenshot of a table that might be displayed to the user:
If the uploaded grammar file is valid, then a Validate Data button will be enabled for the user. Using the Validate Data button, the user can go into the validation step. Following screenshot will be displayed to the user
3. **Validate the raw data against the uploaded grammar file.** Once the grammar file has been validated, you can validate the raw data against it. The validation process will check for errors in the raw data, such as missing fields or invalid values. If there are no errors in the raw data file, then the validation will be successful. Otherwise, the validation will fail, and you will be given a report of the errors. Following screenshot for validate the data
If there are any errors in the uploaded data, then the errors will be reported to the user in the form of a table. Following is a screenshot of the error screen.
\ If the file is valid, then a alert message will be displayed to the user, which means that the data is ready to be ingested. Following is the screenshot:

**Steps:** * **Grammar Validation** The user must first choose the grammar file type to validate, either dimension or event. Then, the user must upload the grammar file. After uploading the file, the debugger analyzes the grammar and reports any errors to the user. If there are no errors, the debugger shows a message indicating that the schema is valid. * **Data Validation** The user must first choose the data file type to validate, either dimension or event. Then, the user must upload the raw data file. After uploading the file, the debugger analyzes the data against the grammar uploaded in the previous step and reports any errors to the user. If there are no errors, the debugger shows a message indicating that the schema is valid.
# Schema Generator Schema generator is a tool that can generate a schema for given set of data. This schema can then be used to validate the data and to generate code to work with the data. In the context of raw event data, the schema generator can be used to generate a grammar file and a dimension file. The grammar file defines the structure of the event data, while the dimension file defines the different dimensions of the event data. These files could then be used to validate the event raw data and to generate code to work with the data. For example, the grammar file could be used to validate the event data against a schema. The dimension file could be used to generate a database schema or to generate code to load the event data into a data warehouse. Here is an example of how a schema generator could be used to generate a grammar file and a dimension file for event raw data: 1. The user provides the schema generator with a sample of the event raw data. 2. The schema generator analyzes the data and generates a schema. 3. The user reviews the schema and makes any necessary changes. 4. The user generates the grammar file and the dimension file. 5. The user validates the event raw data against the grammar file. 6. The user generates code to load the event data into a data warehouse using the dimension file. Schema generators are a valuable tool for working with event raw data. They can help to ensure that the data is well-structured and easy to use. Steps : 1. Upload the sample event data CSV file. This file will be used by the schema generator to extract the schema and derive the dimensions. 2. Review the schema list of fields and field types. The schema generator will display a list of fields and their types as either dimensions or metrics. You can update the field types if needed. 3. Click the update button to generate the grammar and dimensions for the given event. Once the schema and dimensions have been generated, you can download them in a zip file. This file will contain two files: a grammar file and a dimension file. The grammar file defines the structure of the event data. The dimensions file defines the different dimensions of the event data. You can then use these files to validate your event raw data and generate code to work with the data. For example, you could use the grammar file to validate your event data against a schema. You could use the dimensions file to generate a database schema or to generate code to load your event data into a data warehouse. Here are some additional tips for using a schema generator: * Make sure that the sample event data file that you upload is representative of the entire event data set. This will help to ensure that the schema and dimensions that are generated are accurate. * Review the schema list of fields and field types carefully before clicking the update button. This will help to ensure that the schema and dimensions are generated correctly. * Once you have generated the schema and dimensions, test them with a small sample of your event raw data to make sure that they are working correctly. **Example:** 1. Upload the sample event data CSV file. This file will be used by the schema generator to extract the schema and derive the dimensions. Following screenshot of the schema generator.
Following is the sample data for the ETB coverage status data of the DIKSHA program.
Once the user uploads the file, the extracted schema will be shown to the user in the format of the table. The following screenshot represents the sample screen.
The user needs to be able to enter a program name and description, and choose whether the field should be treated as a metric or not. The field types are also shown in the table for every field. Users can edit the name of the field if they want and those field names are editable in the table. Along with the above, users are also able to select dimensions as well. Here’s the screenshot of another part of the screen that contains adding Dimensions. Note: Automatically date time fields are considered as time dimensions.
Users will be able to add Dimensions as shown in the above screenshot. By clicking on the “Add Dimension” button, users can add a new dimension for the event. After adding the dimension, the user needs to add at least one field from the event to the dimension. If the user adds more than one field to the dimension, then the user has to define at least one field as “Index”. Indexes are useful in removing the duplicates from the records while generating the dimension files. Note: If the user only selects one field for the dimension, then no need to define the Index field as the selected field would be considered as the Index for that dimension. After providing the required inputs, users can be able to generate dimensions and events by clicking on the above “Update” button.
2. After generating the dimension and event grammars for the provided file, the following screen will be displayed to the user.
The screen has navigation buttons at the top that the user can use to switch between the dimensions and events tabs. The list tab contains information about the dimensions and metrics list. The dimensions tab contains a drop-down menu that the user can use to select the dimension they want to see.
The events tab contains a dropdown menu that the user can use to select the event they want to see.
The user will be able to download all the generated files in zip format by using the “Download Ingest” button. # System Monitoring System monitoring contains information about the system resource consumption and status about docker container services of cQube. * Once the user is logged in to the admin panel. Click on monitoring, It redirects to another window where you will be able to see the grafana dashboard.
* By default the username and password will be the following Username: admin Password: admin * Click on login and the change the password as prompted
* Login to the grafana dashboard and click on the Toggle Menu bar and then click on Dashboard
* Once you click on dashboards, you will be able to see the dashboards. Click on General to expand the dashboard.
* **Docker containers:** This dashboard contains the general statistics about the docker container performances.
CPU used: This module gives you the total number of CPU used by the containers CPU cores: This module gives the information about the number of cpu cores the VM is running on Memory Load: It gives the total percentage of memory load by the docker containers Used Memory: It gives the information about the total GB of memory in use by the docker containers Storage Load: It gives the total percentage of storage load by the docker containers Used Storage: It gives the information about the total GB of storage in use by the docker containers Running Containers: It shows the total number of containers which are up and running Container CPU Usage: It shows the usage of CPU consumed by each container Container Memory Usage: It shows the usage of memory consumed by each container * Infra Metrics: This dashboard contains the general statistics about the docker infrastructure
Uptime: It shows the total uptime of the server from the last shutdown. CPU idle: It shows the percentage of CPU idle at that particular time CPU cores: It shows the number of cpu cores the VM is running on Available Memory: It shows the total GB of free RAM in the VM Free Storage: It shows the total number of free storage available CPU Usage: It shows the usage cycle of cpu Memory Usage: It shows the usage cycle of RAM Network Usage: It shows the usage of input/output network usage * **Monitor Services:** This dashboard contains the statistics about the monitoring services.
Prometheus uptime: It shows the number of minutes prometheus is up and running Memory Usage: It shows the memory usage of monitoring containers Container CPU usage: It shows the percentage of cpu used by monitoring containers Container memory usage: It shows the percentage of memory used by monitoring containers.
# Adding Users There are 2 ways to add users to cQube V 5.0 via Keycloak: 1. Adding individual users 2. Adding bulk users # Adding an individual user **Step 1 :** Post configuring Keycloak as explained here, navigate to the Users tab. **Step 2 :** Click on the add User button.
**Step 3 :** Enter the username for the user being created and save.
**Step 4 :** Once the user is created, navigate to the Credentials tab and enter the required password. Turn off the temporary password option and click on 'Set Password'.
Step 5: Navigate to the Role-Mapping, under Realm-roles select private\_user and click on ‘Add selected’ button. This will assign the private\_user role to the created user.
✅ The required user has been added successfully! **Steps to manually enable the flag to store Role-Info** **Step 1 :** Navigate client scopes on the left hand navigation. **Step 2 :** Select roles under the client scopes in the table.
**Step 3 :** Navigate to the Mappers tab on the top and select realm roles.
**Step 4 :** Toggle Add to UserInfo button and save.

####
\ \ \
# Adding bulk users **Step 1 :** Navigate to 'Import' tab in the 'Manage' section.
**Step 2 :** Click on 'Select file' and upload the .json file containing information of users to be added. (.json file should be in the format as shown below)
**Step 3 :** After selecting the.json file, turn the import users option on. Then, in the dropdown, select 'Overwrite' if you wish to overwrite existing users with the same information, or select 'Skip' if you want to skip them. If you select 'Fail', the process will fail in case there are users with duplicate information.
**Step 4 :** Post selecting the dropdown, click on 'Import'.
✅ The bulk users have been added successfully and will be able to login to the application using respective credentials! # Infra health monitoring * This dashboard contains the general statistics about the docker infrastructure
| Metrics | Description | | ---------------- | --------------------------------------------------------------- | | Uptime | It shows the total uptime of the server from the last shutdown. | | CPU idle | It shows the percentage of CPU idle at that particular time | | CPU cores | It shows the number of cpu cores the VM is running on | | Available Memory | It shows the total GB of free RAM in the VM | | Free Storage | It shows the total number of free storage available | | CPU Usage | It shows the usage cycle of cpu | | Memory Usage | It shows the usage cycle of RAM | | Network Usage | It shows the usage of input/output network usage |
# Usage monitoring **Telemetry Process** Telemetry is a database which provides the website traffic information to the users. This tool provides datapoints like number of visitors, their access style (devices) and the report type they have accessed, etc. 1. KPIs/Metrics captured: a. Most Visited Landing Pages b. Users Count c. Average time spent on each page d. Unique Users across Browser & Device Types e. Roll-Up of users using (Daily, Weekly) f. Click, Select, Download 2. Define Events, Dimensions & Schema eventName : telemetry-event.data
datetimestampuserIdpageNamepageEventpageEventNamedeviceTypeuserLocationbrowserTypetimeSpenttimeIntimeOut
24/04/2313:00:11cf797e0a-b857-47f1-b0d7-10bd83231421teacher-attendanceclickdownloadwebBengaluruchrome


24/04/2313:00:11cf797e0a-b857-47f1-b0d7-10bd83231421teacher-attendanceonloadpageLoadmobileBengalurufirefox321313:10:0213:13:02
24/04/2313:00:11cf797e0a-b857-47f1-b0d7-10bd83231421pgiclickapplywebBengaluruchrome


24/04/2313:00:11cf797e0a-b857-47f1-b0d7-10bd83231421teacher-attendanceclickcancelmobileBengaluruchrome


24/04/2313:00:11cf797e0a-b857-47f1-b0d7-10bd83231421loginlogin






3\. Workflow The diagram explains about the flow of the telemetry works.
4. Visualize the reports. 5. Steps done for enabling telemetry a. Create a python script for splitting up multi metrics into single metric (). b. Add the telemetry details in config.json file (). c. Create nifi processor group to schedule for moving data and processing d. Add new UI report e. Add the logics in UI to capture the events f. Create the grammars # Testing approaches & activities Two types of testing approaches are followed in cQube: 1. **Functional Testing:** * Smoke Testing * Regression Testing * System E2E Testing 2. **Non Functional Testing:** * Performance Testing Testing activities of cQube are divided into 3 phases: 1. **Installation Testing** - AWS SDC (On-premise) , Oracle 2. **API’s Testing** - Spec , Ingestion ,Dimension and Scheduler 3. **Visualization** - UI Application Testing # Manual & Automated testing **Testing activities mentioned below are done manually:** * **Installation of cQube** - AWS * **Installation of cQube** - SDC / On-premise ( localhost and Public ) * **Installation of cQube** - Oracle **Testing activities mentioned below are automated:** * **Using Karate Framework** - CuCumber with BDD approach automated all the event/spec, Ingestion/, /dimensions, /scheduler, etc., covered with all the required validations to the API responses. * **For Visualization** - Using Selenium with a Python programming language and Pytest framework for running the automation test suites - almost 80% of the UI application can be automated, and the rest of other test cases are executed as a manual process # Functional Testing # Smoke Testing When the build is ready for QA then, we will take up the release build version code and start testing the major features of the application to ensure that all the functionality is working as expected or not by following the phases * One-step Installation - Positive Scenarios to be executed * API karate Framework - Calling Spec , dimension , Ingestion, and Schedule APIs for each program wise to verify able to add schemas to db , input files to buckets, and verify the ability to schedule the Nifi processor and validate the processing of the ingested input files * Verify the after Nifi Processing of the files whether datasets of all the programs have been created tables or not in the database * Verifying the Visualization * Login Page * Dashboard - Summary Statistics * All Program Dashboard - Data is showing up or not, and each web feature functionality is working as expected or not # Functional tests When Smoke Testing is completed with 0 Blockers and 0 Majors, then we proceed with functional testing by following the phases * One-step Installation - Positive and Negative Scenarios to be executed * API Karate Framework - Validation of spec , Ingestion , Dimension , and Schedule API, both positive and negative scenarios using karate. * Verify the after Nifi Processing of the input files whether datasets of all the programs have been created tables or not in the database * Verifying the Visualization. * Login Page * Dashboard - Summary Statistics * All Program Dashboard - Data is showing up or not, and each web feature functionality is working as expected or not # Regression Testing This kind of testing activity can be followed up when new features or new enhancements are made to the existing UI - Visualization report. Then we will go through the Completed end to end regression testing of the Application, which includes * Login Page * Dashboard Screen * Summary Statistics - All Navigations * Each Program Dashboard and tab, along with web features to be tested * Session Out and Resolutions testing ( 2K & 4K ) with Chrome and Firefox compatibility browsers # System Testing This can be done when there is no blocker and major defects are on the defect list. Then it would be the final round of testing activity before moving to production/sign-off from Quality Assurance of the Application * One-Step Installation - AWS , SDC/On-premise , Oracle * API Testing - Spec , Ingestion , Dimension , Scheduler * Nifi Processing Validation of ingested program files * Database validation - verification of all the programs datasets are created or not * Visualization Testing - Login Page , Dashboard Page , Program Dashboards and their each tab functionalities to be tested * Verification of Logout , Release date , Release version and Resolutions tests # Non Functional Testing # Performance Testing The practice of evaluating how a system performs in terms of responsiveness and stability under a particular workload 1. Load Testing 2. Volume Testing # Load Testing It is a type of Performance Testing that determines the performance of a system, software product, or software application under real-life based load conditions. Basically, load testing determines the behavior of the application when multiple users use it at the same time Example: Scenarios are covered with minimum and maximum number of concurrent users are accessing the application on same time to verify that how the application can handle the pressure and responding to the user request # Volume Testing It is a type of software testing which is carried out to test a software application with a certain amount of data. The amount used in volume testing could be a database size or it could also be the size of an interface file that is the subject of volume Exmple: In cQube application we have ingestion of program wise input file which is in the csv file format here we are tested the sending the larger amount of data from 10L to 3M of records with csv file size approximately 1.5GB through the API to bucket
# Performance testing results Please access the link [here](https://docs.google.com/spreadsheets/d/1ec7CrZ07BP0lc8DbYkyMJHa5oHJii3GorGhCa0qMUHk/edit#gid=159280127) # Test for One-Step Installation Executing all the prepared manual test cases of one-step installation by manual by providing positive and negative inputs while doing installation and capturing the errors/defects, and updating into Jira board along with reproducible steps and screenshots for the understanding defect smoothly and avoid difficulty to understanding the defect to developers. # Test for Ingestion * By using Karate Framework - developed the cucumber scripts to read the schemes provided by the DB developers and based on the ingestion data to be generated according to predefined columns and data types of each file * Running the karate scripts to verify whether the ingestion process is working as expected or not # Test for nifi processing * Once all the Ingestion and Dimensions files are uploaded to the process\_input folder, then we will schedule the Nifi processor group to continue further processing of uploaded input files and store the final result in the database by creating datasets. * Verifying whether each program-level dataset is created or not in the database. # Test for UI Application * For Visualization - Automated the scripts using selenium python using pytest framework that has covered all the UI automation up to 80%. Most of the features are to be automated with both positive and negative validation in the automation scripts and will run the scripts once Nifi processing is completed * Rest other, like 20% of manual test cases like alignments , tooltip information, and resolution testing ,can be covered in the manual testing # Test for KPIs By using Pytest Framework : we followed to get the output data from the database and implemented the business logic, and verifying the final result matched to the expected outcome Step 1: Prepare input JSON files for each program(student\_attendance, teacher\_attendance, etc..) which have all the dimensions needed for a particular program along with its values and along with its metric with min and max values. Step 2: Use the grammar file required for the program (student\_attendance, teacher\_attendance, etc..) in order to extract the time dimension, dimensions, and metrics. Step 3: Get the required time dimension and dimensions from the grammar file Step 4: Get the required metrics from the config file Step 5: For each metric in the program, generate the test data along with the required dimension columns specified in the config file.
Step 6: Split the generated test data of each metric according to different time\_lines (daily, weekly, monthly, yearly) for each dimension key-metric aggregate value combination, to use it as the expected data (which should be filled up through manual calculation).
Step 7: Ingest the test data into the cube server and get the processed data in the database, which is the actual data file. Step 8: Fill in the aggregate values for expected data manually. Step 9: Compare the actual data with the expected data and generate the HTML test results. # State List Here the list of state name and state code | State Code | State name | | ---------- | --------------------------- | | AN | Andaman and Nicobar Islands | | AP | Andhra Pradesh | | AR | Arunachal Pradesh | | AS | Assam | | BR | Bihar | | CH | Chandigarh | | CT | Chhattisgarh | | DN | Dadra and Nagar Haveli | | DD | Daman and Diu | | DL | Delhi | | GA | Goa | | GJ | Gujarat | | HR | Haryana | | HP | Himachal Pradesh | | JK | Jammu and Kashmir | | JH | Jharkhand | | KA | Karnataka | | KL | Kerala | | LD | Lakshadweep | | MP | Madhya Pradesh | | MH | Maharashtra | | MN | Manipur | | ML | Meghalaya | | MZ | Mizoram | | NL | Nagaland | | OR | Odisha | | PY | Puducherry | | PB | Punjab | | RJ | Rajasthan | | SK | Sikkim | | TN | Tamil Nadu | | TG | Telangana | | TR | Tripura | | UP | Uttar Pradesh | | UT | Uttarakhand | | WB | West Bengal | # AWS Deployment Here the details for the complete deployment process **Prerequisites to install cQube on AWS machine** * Ubuntu 22.04 (supported) * 16 GB of System RAM (minimum requirement) * 4 core CPU (minimum requirement) * Domain name ex: * SSl certificate keys ex: certificate.crt and private.key * AWS access\_key and secret\_key * 250 GB Storage **Step 1:** Use the following command to connect to the AWS instance ssh -i \ \@\ **Ex: ssh -i poc\_key.pem -o ServerAliveInterval=60 ubuntu\@13.200.12.31** **Step 2**: Clone the cqube-devops repository using following command git clone
**Step 3:** Navigate to the directory where cqube is cloned or downloaded and checkout to the desired branch(Release Branch) cd cqube-devops/ git checkout release-v5.0.5(latest release branch)
**Step 4:** After checkout of the latest branch, we need to move the SSL keys (.crt and.key) to the mentioned path, as shown in the below screenshot. Path: cqube-devops/ansible/ssl\_certificates **Command for moving ssl keys:** cp certificate.crt private.key /home/ubuntu/cqube-devops/ansible/ssl\_certificates
**Step 5:** After copying the ssl keys similarly, we can move the VSK dimension to (state,district,grade,subject,medium) below the mentioned path, as shown in the screenshot below. **Command for moving dimensions:** **(Note: This Step 5 is applicable for pulling the data from NVSK )** cp state-dimension.data.csv grade-dimension.data.csv subject-dimension.data.csv etc../home/ubuntu/cqube-devops/ansible.dimension\_files Path: cqube-devops/ansible/dimension\_files.
**Step 6:** Give the following permissions to the install.sh file sudo chmod u+x install.sh **Step 7:** Install cqube with non root user with sudo privileges sudo ./install.sh
* Access\_type(Enter NVSK or VSK or Others) * state\_code ( Enter the required state code by referring to the state list provided ) * Do want to enable login screen for cqube instance(Enter true or false) * Do you want to pull the data from the NVSK server?(Enter true or false) * Please enter the end point to pull the data (Ex: cqube-demo-nvsk.tibilprojects.com) **Step 8:** User Input Variables - These are the variables which need to be entered by the user by following the Hint provided Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cQube
* Mode of installation: Public * Storage\_type : aws * API\_Endpoint(Enter the domain name : (ex: cqube-demo-cert.tibilprojects.com) * Please enter the name of cert file: ( ex:certificate.crt) * Please enter the name of key file: (ex: private.key) **Step 9:** Once you enter the above user input it will create one config file.please preview the config file and confirm if everything is correct.if it is correct type “no” and proceed and else type “yes” then correct it.
* db\_user\_name ( Enter the postgres database username ) * db\_name ( Enter the postgres database name ) * db\_password ( Enter the postgres password ) * read\_only\_db\_user( Enter the read only db user) * read\_only\_db\_password( Enter the read only db password) * keycloak\_adm\_name( Enter the keycloak admin name) * keycloak\_adm\_password( Enter the keycloak password) **Step 10:** Optional\_variables- Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for ‘yes’ to enter their credentials otherwise can opt for ‘no’ when the question pops up
**Step 11:** Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.
**Step 12:** A preview of the program\_selector.yml file is displayed followed by a question where the user gets an option to enable or disable the programs on choosing ‘yes’. If option ‘no’ is selected then the install.sh moves to the next section.
**Step 13:** Once the installation is completed, You will be prompted with the following messages and required reference urls. **cQube Installed Successfully** cQube ingestion api can be accessible using \
* After installation check the docker containers are up or not * Sudo docker ps -a
**Ingestion Flow:** **Setting up Postman:** * [Download](https://www.postman.com/downloads/) the postman application and import the collection. * Select the import option in the postman then select the upload files to import the collection. Please refer to the below screenshot.

* Upload the cQube\_latest.postman\_collection .json file and VSK\_Schema.postman\_collection.json, * After installations first need to create the jwt token in postman as shown below screenshot. API\_Endpoint:
* After creating the JWT token then upload the vsk\_schema and dimension schema as shown below screenshot API Endpoint:[ ](https://cqube-ssl-test.tibilprojects.com/api/ingestion/national_programs)[htpps://cqube-ssl-test.tibilprojects.com/api/spec/event](https://cqube-apr27-demo.tibilprojects.com/api/spec/event)
Run the both schema at a time (click on 3 dots and select Run collection) * After uploading the grammar files,Just copy the generate token and add into national\_programs and new programs * Go to authorization and select Bearer token, then paste the token * Then I need to upload the VSK data(NCERT) as shown in the below screenshot. API Endpoint:[ ]()[https://cqube-ssl-test.tibilprojects.com/api/ingestion/national\_programs
]()
* We need to ingest only three programs (pgi,diksha,nishtha)data through API. The other three programs automatically pull the data from the NVSK server. * For above point we need to upload (PGI, DIKSHA, NISHTHA) three programs schema and ingest the raw files to the aws emission bucket. * Then we can schedule the below mentioned processor groups one by one by using schedule API as shown in the below screenshot * Run\_adapters * data\_moving\_aws
* After the schedule, the programs should one by one by using schedule API as mentioned in the below screenshot. Bbody: { "processor\_group\_name": "ingest\_data", "scheduled\_at": "0 31 14 \* \* ?", "program\_name": "nishtha" }

* Check the visualization in the UI dashboard. As shown in the below screenshot.
\ \ \ \
\ \
# SDC Deployment **Prerequisites to install cQube on local machine** * Ubuntu 22.04 (supported) * 16 GB of System RAM (minimum requirement) * 4 core CPU (minimum requirement) * Domain name (with SSL), ex: * 250 GB Storage **Step 1:** Use the following command to connect to the On-premise instance ssh -i \ \@\ **Ex: ssh -i poc\_key.pem -o ServerAliveInterval=60 ubuntu\@13.200.12.31** **Step 2:** Clone the cqube-devops repository using following command git clone
**Step 3:** Navigate to the directory where cqube is cloned or downloaded and checkout to the desired branch(Release Branch) cd cqube-devops/ git checkout release-v5.0.5(Latest release branch )
**Step 4:** After checkout of the latest branch, we need to move the SSL keys (.crt and.key) to the mentioned path, as shown in the below screenshot. **Command for moving ssl keys:** cp certificate.crt private.key /home/ubuntu/cqube-devops/ansible/ssl\_certificates Path: cqube-devops/ansible/ssl\_certificates
**Step 5:** After copying the ssl keys similarly, we can move the VSK dimension to (state,district,grade,subject,medium) below the mentioned path, as shown in the screenshot below. **Command for moving dimensions:** **(Note: This Step 5 is applicable for pulling the data from NVSK )** cp state-dimension.data.csv grade-dimension.data.csv subject-dimension.data.csv etc../home/ubuntu/cqube-devops/ansible.dimension\_files Path: cqube-devops/ansible/dimension\_files.
**Step 6:** Give the following permissions to the install.sh file sudo chmod u+x install.sh **Step 7:** Install cqube with non root user with sudo privileges sudo ./install.sh
Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cqube **Step 8:** User Input Variables - These are the variables which need to be entered by the user by following the Hint provided * Access\_type(Enter NVSK or VSK or Others) * state\_code ( Enter the required state code by referring to the state list provided ) * Do want to enable login screen for cqube instance(Enter true or false) * Do you want to pull the data from the NVSK server?(Enter true or false) * Please enter the end point to pull the data (Ex: cqube-demo-nvsk.tibilprojects.com)
**Step 9:** Once you enter the above user input it will create one config file.please preview the config file and confirm if everything is correct.if it is correct type “no” and proceed and else type “yes” then correct it. * Mode of installation: Public * Storage\_type : local * API\_Endpoint(Enter the domain name ex: cqube-demo-cert.tibilprojects.com) * Please enter the name of cert file( ex:certificate.crt) * Please enter the name of key file (ex: private.key)
**Step 10:** Optional\_variables- Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for ‘yes’ to enter their credentials otherwise can opt for ‘no’ when the question pops up * db\_user\_name ( Enter the postgres database username ) * db\_name ( Enter the postgres database name ) * db\_password ( Enter the postgres password ) * read\_only\_db\_user( Enter the read only db user) * read\_only\_db\_password( Enter the read only db password) * keycloak\_adm\_name( Enter the keycloak admin name) * keycloak\_adm\_password( Enter the keycloak password)
**Step 11:** Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.
**Step 12:** A preview of the program\_selector.yml file is displayed followed by a question where the user gets an option to enable or disable the programs on choosing ‘yes’. If option ‘no’ is selected then the install.sh moves to the next section.
**Step 13:** Once the installation is completed, You will be prompted with the following messages and required reference urls. **cQube Installed Successfully** cQube ingestion api can be accessible using \
* After installation check the docker containers are up or not * Sudo docker ps -a

**Ingestion Flow:** Setting up Postman: * [Download](https://www.postman.com/downloads/) the postman application and import the collection. * Select the import option in the postman then select the upload files to import the collection. Please refer to the below screenshot.

* Upload the cQube\_latest.postman\_collection .json file and VSK\_Schema.postman\_collection.json, * After installations first need to create the jwt token in postman as shown below screenshot. API\_Endpoint: [https://cqube-ssl-test.tibilprojects.com/api/ingestion/generatejwt](http://vsk.ap.gov.in/api/ingestion/generatejwt)
* After creating the JWT token then upload the vsk\_chema and dimension schema as shown below screenshot API Endpoint: [htpps://cqube-ssl-test.tibilprojects.com/api/spec/event](https://cqube-apr27-demo.tibilprojects.com/api/spec/event)
Run the both schema at a time (click on 3 dots and select Run collection) * After uploading the grammar files,Just copy the generate token and add into national\_programs and new programs api * Go to authorization and select Bearer token, then paste the token * Then I need to upload the VSK data(NCERT) as shown in the below screenshot.ii API Endpoint: [https://cqube-ssl-test.tibilprojects.com/api/ingestion/national\_programs](https://ap.vsk.gov.in/api/ingestion/national_programs)
* We need to ingest only three programs (pgi,diksha,nishtha)data through API. The other three programs automatically pull the data from the NVSK server. * For above point we need to upload (PGI, DIKSHA, NISHTHA) three programs schema and ingest the raw files to the aws emission bucket. * Then we can schedule the below mentioned processor groups one by one by using schedule API as shown in the below screenshot * Run\_adapters * data\_moving\_local

* After that you can schedule the one by one program by using schedule API as mentioned in the below screenshot. Body: { "processor\_group\_name": "ingest\_data", "scheduled\_at": "0 31 14 \* \* ?", "program\_name": "nishtha" }
* Check the visualization in the UI dashboard. As shown in the below screenshot.

# Adapter Details During the Processing we can schedule processor groups one by one by using schedule API for the different storage types. * data\_moving\_local * data\_moving\_aws * data\_moving\_azure * data\_moving\_oracle # How can I upgrade cQube to the latest release **AWS** Connect to the cQube AWS EC2 Instance For Linux and macOS: 1. Download the .pem file generated during EC2 instance creation. 2. Open the terminal and navigate to the folder where the .pem file is downloaded. 3. Provide read permission to the .pem file: sudo chmod 400 \ Connect to the instance using the following command: 4. ssh -i \ \@\ For Windows: 1. Download the .pem file generated during EC2 instance creation. Use Puttygen to connect to the instance. Refer to this link for instructions:[ PuttyGen Instructions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html) # Deployment & ingestion related issues & their solutions ## Deployment Issues * **Process lock issue** When starting the deployment we might face an issue which states “Waiting for cache Lock: Could not get lock /var/lib/dpkg/lock-frontend” **Resolution:** Check for the process which is using sudo and stop the process using process id or wait till the sudo gets de-locked and restart the deployment process * **Redeployment of dashboard-ms** sudo docker stop dashboard\_app sudo docker rm dashboard\_app sudo docker dashboard\_ms:1 cd cqube-devops/microservices/dashboard-ms sudo git pull sudo docker build -t dashboard\_ms:1 . sudo docker run -d -p 4200:80 --network cqube\_net --name dashboard\_app dashboard\_ms:1 * **Redeployment of Querybuilder-ms** sudo docker stop querybuilder\_app sudo docker rm querybuilder\_app sudo docker querybuilder\_ms:1 cqube-devops/microservices/querybuilder-ms sudo git pull sudo docker build -t querybuilder\_ms:1 sudo docker run -d -p 3002:3002 --network cqube\_net --name querybuilder\_app querybuilder\_ms:1 * **Port in use error while running docker-compose** If this error arises, please check for the port which is affecting the deployment and kill the\ the port using the port ID. sudo netstat -ntlp | grep \ sudo kill -15 \\ Once the port is killed follow the installation script. ## Related to ingestion * **Single Master of Districts, Blocks, Clusters, and Schools**: To enable decentralized observability, a single master file should be created for each jurisdiction (district, block, cluster, and school) with a unique ID and name. Both state and national programs on cQube will reference these masters. Any discrepancy in the master data will result in unsuccessful data ingestion. * **Individual Values for Grade, Subject, and Medium:** Grade, subject, and medium data should have individual values in separate rows, instead of being stored as arrays. Each subject should have a unique ID, and different subjects should be in different rows. * **Change in Column Name for NISHTHA Program:** In the diksha\_nishtha\_percentage-enrollment-certification.zip file for the NISHTHA program, the first "State" column (Column B) should be renamed to "State Name\_Correct" to avoid duplication. * **No Quotes in the Data:** Quotes, whether single (') or double (“), should not be present in the data being ingested. * **Numerical Values instead of Strings:** Certain files (udise\_program-started.zip, nas\_program-started.zip, diksha\_nishtha\_program-started.zip) should use numerical values (1 and 0) instead of strings (Yes and No) in relevant columns. * **No Commas in Large Numerical Values:** Large numerical values in CSVs should not have commas (,) in between the digits. These guidelines ensure data consistency, proper formatting, and accurate ingestion in the cQube system.
# Reporting a Bug Describes the process of reporting a bug to the cQube Product Team There is a bug if something for the user isn’t working as intended. The cQube Product Team will be responsible to fix the bug that has been reported in the DPG. ### Communication Tool All the bugs and need for support should be posted as issues on the cQube Community on GitHub - [Link](https://github.com/Sunbird-cQube/community/issues). ### Reporting a Bug A new issue should be created in the GitHub Community of cQube **only** if any similar issue doesn't already exist or has been resolved by the cQube Product Team. The reported issues for cQube can be seen [here](https://github.com/Sunbird-cQube/community/issues?q=is%3Aopen+is%3Aissue) and the resolved issues can be seen [here](https://github.com/Sunbird-cQube/community/issues?q=is%3Aissue+is%3Aclosed). **Process to report a bug:** 1. Go to the issues tab of cQube Community on Github -
2. Click on 'New Issue'. Then select 'Get Started' in front of 'Bug report'.
3. Detail out the bug that you are facing with a title, description, steps to reproduce, expected behaviour, relevant screenshots and desktop / mobile configurations.
4. Attach relevant label(s) to the bug being reported.
5. Click on 'Submit new issue'. ### Response from the cQube Product Team The cQube Product Team will respond within 2-3 working days post the issue has been submitted with a timeline by when the bug can be fixed. It will be included in the cQube roadmap. The adopter who has raised the issue will be able to track the bug on a JIRA ticket. Ideally, P0 bugs will be prioritised and taken up in the ongoing sprint and P1 bugs will be taken up in the next sprint based on when the issue is reported. --- [Next Page](/llms-full.txt/1)