Sunbird cQube
  • 📖KNOW ABOUT CQUBE
    • What is cQube & what does it solve
      • Business case
      • cQube ed
      • Design Principles
    • cQube adoptions
    • Discuss more about cQube
  • 👍TRY CQUBE
    • cQube on Gitpod
  • 🌅WHAT IS NEXT IN CQUBE
    • cQube Roadmap
  • 💻TECHANICAL OVERVIEW
    • Architecture
    • Design Principles
    • Key Components
    • Details of microservices
  • 👩‍💻Get started on cQube
    • Suggested Team Structure
    • Hardware Requirements
    • Prerequisites Checklist
    • Checking pre existing ports
    • Instance Creation
    • Copying SSL Certificate
  • 🛃USE CQUBE
    • How can I install cQube?
      • High level understanding of deployment
      • Oracle Installation
      • AWS Installation
      • SDC Installation
      • Azure Installation
    • How to prepare the data
      • Schema
      • How to prepare schemas for dimension files
      • How to prepare schemas for event files
      • Examples of dimension & event files
      • All cQube schemas used for VSK
    • Ingesting the data
      • High-level understanding of ingestion
      • Steps to ingest schema
      • Steps to ingest data files
        • Upload the .csv data file for state specific programs using ingestion API
        • API to upload starter pack data files for NVSK programs
        • Get file status API
        • Scheduled API
      • Error during ingestion
        • Error Monitoring
        • Common errors in data files during ingestion
      • Processor group name
    • Adapter details
    • Postman details
    • Processing of data
      • Data Processing using CLI command
      • API Details for Nifi-Rest
      • Nifi section
    • Visualizing the data
      • High level understanding of how visualizations work in cQube
      • Programs and reports out-of-the-box
      • Enhance /Customize cQube
        • Available customizations
          • Changing Dashboard Logos and Headers
          • Changing Program Name, Icon and Side Menu Sequence
          • Adding a new KPI
          • Adding a Map KPI into dashboard ms
          • Table Drill Down Customization
          • Adding a Scatter Plot KPI into dashboard ms
          • Configure default date range across app/specific report
        • How to add a New Indicator
        • How to add a new report in an existing program
        • How to add a new program (end to end)
    • Additional Features
      • Public/Private dashboards
      • Role based access control
      • Saving geographical preferences
      • Admin Panel
        • Data Debugger
        • Schema Generator
        • System Monitoring
    • Adding Users
      • Adding an individual user
      • Adding bulk users
  • 🖥️MONITOR cQUBE
    • Infra health monitoring
    • Usage monitoring
  • 🔎QA testing
    • Testing approaches & activities
    • Manual & Automated testing
    • Functional Testing
      • Smoke Testing
      • Functional tests
      • Regression Testing
      • System Testing
    • Non Functional Testing
      • Performance Testing
        • Load Testing
        • Volume Testing
        • Performance testing results
    • Test for One-Step Installation
    • Test for Ingestion
    • Test for nifi processing
    • Test for UI Application
    • Test for KPIs
  • ☀️DEPLOYMENT PROCESS
    • State List
    • AWS Deployment
    • SDC Deployment
    • Adapter Details During the Processing
  • 🈴UPGRADING TO LATEST VERSION
    • How can I upgrade cQube to the latest release
  • 🆘Common issues and their solutions
    • Deployment & ingestion related issues & their solutions
  • ⏱️Standard Operating Procedure
    • Reporting a Bug
    • Protocol for issue reporting & resolution
    • Suggesting Enhancements
    • Raising a PR
  • ❓Frequently Asked Questions
    • Running List
  • 🧑‍🏫🧑🏫 Recording of trainings
    • Link to the training videos
  • 🧠Key Terms & Concepts
    • Definitions
  • 🚀cQube Release Notes
    • cQube - Release V 5.0.5
    • cQube - Release V 5.0.3
    • cQube - Release V 5.0.2
    • cQube - Release V 5.0.1
    • cQube - Release V 5.0
    • cQube - Release V 4.1-beta
    • cQube - Release V 4.0-beta
    • cQube - Release V 4.0-alpha
    • cQube - Release V 3.7
    • cQube - Release V 3.6
    • cQube - Release V 3.5
    • cQube - Release V 3.4
    • cQube - Release V 3.3
    • cQube - Release V 3.2
    • cQube - Release V 3.1
    • cQube - Release V 3
    • cQube - Release V 2
    • cQube - Release V 1.13 and V 1.13.1
    • cQube - Release V 1.12 and V 1.12.1
    • cQube - Release V 1.11
    • cQube - Release V 10 and V 10.1
    • cQube - Release V 1.9
    • cQube - Release V 1.8 and V 1.8.1
    • cQube - Release Notes V 1.7
    • cQube - Release Notes V 1.6 and V 1.6.1
    • cQube - Release Notes V 1.5
    • cQube - Release Notes V 1.4
    • cQube - Release Notes V 1.3
    • cQube - Release Notes V 1.2 and V 1.2.1
    • cQube - Release Notes V 1.1
    • cQube - Release Notes V 1.0
  • 📂cQube V 4.1 - Beta
    • Sunbird cQube Overview
    • cQube Product Description
    • Listen to Experts (Youtube)
    • Software Requirements
    • Acronyms
    • cQube Software Architecture
    • AWS - Network Architecture
      • Hardware requirements
      • Data Storage Locations
    • Security Implementations
    • Prerequisites for Installation process
    • New Use-Case Creation
    • cQube Setup & configuration
    • Base Installation steps
    • Base Upgradation steps
    • Workflow Installation steps
    • Workflow Upgradation steps
    • Laptop/Desktop Installation
      • Base Installation
      • Workflow Installation
      • Mock Data Processing
    • Ad-hoc analysis
    • Workflow process
    • Emission Process
    • cQube ER Diagrams
    • Data Validation after Ingestion
    • User Authentication Process
    • Admin Login Process
    • Admin Features
    • cQube Datasource Configuration
    • cQube data replay process
    • S3 Partitioning
    • Reports
    • Troubleshooting Issues
      • Data Processing-NIFI Issues
      • Data Processing-PostgreSQL Issues
      • Data Emission Issues
      • Angular & Node Issues
    • FAQs
    • Discuss
    • Report
    • Source Code
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub
  1. DEPLOYMENT PROCESS

SDC Deployment

PreviousAWS DeploymentNextAdapter Details During the Processing

Last updated 1 year ago

Was this helpful?

Prerequisites to install cQube on local machine

  • Ubuntu 22.04 (supported)

  • 16 GB of System RAM (minimum requirement)

  • 4 core CPU (minimum requirement)

  • Domain name (with SSL), ex: https://vsk.scertchd.edu.in/

  • 250 GB Storage

Step 1: Use the following command to connect to the On-premise instance

ssh -i <path_to_the_pem_file> <user_name>@<public_ip_of_the_instance>

Ex: ssh -i poc_key.pem -o ServerAliveInterval=60 ubuntu@13.200.12.31

Step 2: Clone the cqube-devops repository using following command

git clone

Step 3: Navigate to the directory where cqube is cloned or downloaded and checkout to the desired branch(Release Branch)

cd cqube-devops/

git checkout release-v5.0.5(Latest release branch )

Step 4: After checkout of the latest branch, we need to move the SSL keys (.crt and.key) to the mentioned path, as shown in the below screenshot.

Command for moving ssl keys:

cp certificate.crt private.key /home/ubuntu/cqube-devops/ansible/ssl_certificates

Path: cqube-devops/ansible/ssl_certificates

Step 5: After copying the ssl keys similarly, we can move the VSK dimension to (state,district,grade,subject,medium) below the mentioned path, as shown in the screenshot below.

Command for moving dimensions:

(Note: This Step 5 is applicable for pulling the data from NVSK )

cp state-dimension.data.csv grade-dimension.data.csv subject-dimension.data.csv etc../home/ubuntu/cqube-devops/ansible.dimension_files

Path: cqube-devops/ansible/dimension_files.

Step 6: Give the following permissions to the install.sh file

sudo chmod u+x install.sh

Step 7: Install cqube with non root user with sudo privileges

sudo ./install.sh

Install.sh file contains a shell script where it will run shell scripts and ansible-playbook to setup the cqube

Step 8: User Input Variables - These are the variables which need to be entered by the user by following the Hint provided

  • Access_type(Enter NVSK or VSK or Others)

  • state_code ( Enter the required state code by referring to the state list provided )

  • Do want to enable login screen for cqube instance(Enter true or false)

  • Do you want to pull the data from the NVSK server?(Enter true or false)

  • Please enter the end point to pull the data (Ex: cqube-demo-nvsk.tibilprojects.com)

Step 9: Once you enter the above user input it will create one config file.please preview the config file and confirm if everything is correct.if it is correct type “no” and proceed and else type “yes” then correct it.

  • Mode of installation: Public

  • Storage_type : local

  • API_Endpoint(Enter the domain name ex: cqube-demo-cert.tibilprojects.com)

  • Please enter the name of cert file( ex:certificate.crt)

  • Please enter the name of key file (ex: private.key)

Step 10: Optional_variables- Database credentials contain default values. If the user wishes to enter their own credentials then the user should opt for ‘yes’ to enter their credentials otherwise can opt for ‘no’ when the question pops up

  • db_user_name ( Enter the postgres database username )

  • db_name ( Enter the postgres database name )

  • db_password ( Enter the postgres password )

  • read_only_db_user( Enter the read only db user)

  • read_only_db_password( Enter the read only db password)

  • keycloak_adm_name( Enter the keycloak admin name)

  • keycloak_adm_password( Enter the keycloak password)

Step 11: Once the config file is generated, A preview of the config file is displayed followed by a question where the user gets an option to re enter the configuration values on choosing yes. If option no is selected then the install.sh moves to the next section.

Step 12: A preview of the program_selector.yml file is displayed followed by a question where the user gets an option to enable or disable the programs on choosing ‘yes’. If option ‘no’ is selected then the install.sh moves to the next section.

Step 13: Once the installation is completed, You will be prompted with the following messages and required reference urls.

cQube Installed Successfully

cQube ingestion api can be accessible using <domain_name>

  • After installation check the docker containers are up or not

  • Sudo docker ps -a

Ingestion Flow:

Setting up Postman:

  • Select the import option in the postman then select the upload files to import the collection. Please refer to the below screenshot.

  • Upload the cQube_latest.postman_collection .json file and VSK_Schema.postman_collection.json,

  • After installations first need to create the jwt token in postman as shown below screenshot.

  • After creating the JWT token then upload the vsk_chema and dimension schema as shown below screenshot

Run the both schema at a time (click on 3 dots and select Run collection)

  • After uploading the grammar files,Just copy the generate token and add into national_programs and new programs api

  • Go to authorization and select Bearer token, then paste the token

  • Then I need to upload the VSK data(NCERT) as shown in the below screenshot.ii

  • We need to ingest only three programs (pgi,diksha,nishtha)data through API. The other three programs automatically pull the data from the NVSK server.

  • For above point we need to upload (PGI, DIKSHA, NISHTHA) three programs schema and ingest the raw files to the aws emission bucket.

  • Then we can schedule the below mentioned processor groups one by one by using schedule API as shown in the below screenshot

    • Run_adapters

    • data_moving_local

  • After that you can schedule the one by one program by using schedule API as mentioned in the below screenshot.

Body:

{

"processor_group_name": "ingest_data",

"scheduled_at": "0 31 14 * * ?",

"program_name": "nishtha"

}

  • Check the visualization in the UI dashboard. As shown in the below screenshot.

the postman application and import the collection.

API_Endpoint:

API Endpoint:

API Endpoint:

☀️
Download
https://drive.google.com/drive/u/0/folders/12Wn7UIHgUhq6U3GzlRN-zdrJMnj1hrPO
https://cqube-ssl-test.tibilprojects.com/api/ingestion/generatejwt
htpps://cqube-ssl-test.tibilprojects.com/api/spec/event
https://cqube-ssl-test.tibilprojects.com/api/ingestion/national_programs
https://github.com/Sunbird-cQube/cqube-devops.git