Sunbird cQube
  • ๐Ÿ“–KNOW ABOUT CQUBE
    • What is cQube & what does it solve
      • Business case
      • cQube ed
      • Design Principles
    • cQube adoptions
    • Discuss more about cQube
  • ๐Ÿ‘TRY CQUBE
    • cQube on Gitpod
  • ๐ŸŒ…WHAT IS NEXT IN CQUBE
    • cQube Roadmap
  • ๐Ÿ’ปTECHANICAL OVERVIEW
    • Architecture
    • Design Principles
    • Key Components
    • Details of microservices
  • ๐Ÿ‘ฉโ€๐Ÿ’ปGet started on cQube
    • Suggested Team Structure
    • Hardware Requirements
    • Prerequisites Checklist
    • Checking pre existing ports
    • Instance Creation
    • Copying SSL Certificate
  • ๐Ÿ›ƒUSE CQUBE
    • How can I install cQube?
      • High level understanding of deployment
      • Oracle Installation
      • AWS Installation
      • SDC Installation
      • Azure Installation
    • How to prepare the data
      • Schema
      • How to prepare schemas for dimension files
      • How to prepare schemas for event files
      • Examples of dimension & event files
      • All cQube schemas used for VSK
    • Ingesting the data
      • High-level understanding of ingestion
      • Steps to ingest schema
      • Steps to ingest data files
        • Upload the .csv data file for state specific programs using ingestion API
        • API to upload starter pack data files for NVSK programs
        • Get file status API
        • Scheduled API
      • Error during ingestion
        • Error Monitoring
        • Common errors in data files during ingestion
      • Processor group name
    • Adapter details
    • Postman details
    • Processing of data
      • Data Processing using CLI command
      • API Details for Nifi-Rest
      • Nifi section
    • Visualizing the data
      • High level understanding of how visualizations work in cQube
      • Programs and reports out-of-the-box
      • Enhance /Customize cQube
        • Available customizations
          • Changing Dashboard Logos and Headers
          • Changing Program Name, Icon and Side Menu Sequence
          • Adding a new KPI
          • Adding a Map KPI into dashboard ms
          • Table Drill Down Customization
          • Adding a Scatter Plot KPI into dashboard ms
          • Configure default date range across app/specific report
        • How to add a New Indicator
        • How to add a new report in an existing program
        • How to add a new program (end to end)
    • Additional Features
      • Public/Private dashboards
      • Role based access control
      • Saving geographical preferences
      • Admin Panel
        • Data Debugger
        • Schema Generator
        • System Monitoring
    • Adding Users
      • Adding an individual user
      • Adding bulk users
  • ๐Ÿ–ฅ๏ธMONITOR cQUBE
    • Infra health monitoring
    • Usage monitoring
  • ๐Ÿ”ŽQA testing
    • Testing approaches & activities
    • Manual & Automated testing
    • Functional Testing
      • Smoke Testing
      • Functional tests
      • Regression Testing
      • System Testing
    • Non Functional Testing
      • Performance Testing
        • Load Testing
        • Volume Testing
        • Performance testing results
    • Test for One-Step Installation
    • Test for Ingestion
    • Test for nifi processing
    • Test for UI Application
    • Test for KPIs
  • โ˜€๏ธDEPLOYMENT PROCESS
    • State List
    • AWS Deployment
    • SDC Deployment
    • Adapter Details During the Processing
  • ๐ŸˆดUPGRADING TO LATEST VERSION
    • How can I upgrade cQube to the latest release
  • ๐Ÿ†˜Common issues and their solutions
    • Deployment & ingestion related issues & their solutions
  • โฑ๏ธStandard Operating Procedure
    • Reporting a Bug
    • Protocol for issue reporting & resolution
    • Suggesting Enhancements
    • Raising a PR
  • โ“Frequently Asked Questions
    • Running List
  • ๐Ÿง‘โ€๐Ÿซ๐Ÿง‘๐Ÿซ Recording of trainings
    • Link to the training videos
  • ๐Ÿง Key Terms & Concepts
    • Definitions
  • ๐Ÿš€cQube Release Notes
    • cQube - Release V 5.0.5
    • cQube - Release V 5.0.3
    • cQube - Release V 5.0.2
    • cQube - Release V 5.0.1
    • cQube - Release V 5.0
    • cQube - Release V 4.1-beta
    • cQube - Release V 4.0-beta
    • cQube - Release V 4.0-alpha
    • cQube - Release V 3.7
    • cQube - Release V 3.6
    • cQube - Release V 3.5
    • cQube - Release V 3.4
    • cQube - Release V 3.3
    • cQube - Release V 3.2
    • cQube - Release V 3.1
    • cQube - Release V 3
    • cQube - Release V 2
    • cQube - Release V 1.13 and V 1.13.1
    • cQube - Release V 1.12 and V 1.12.1
    • cQube - Release V 1.11
    • cQube - Release V 10 and V 10.1
    • cQube - Release V 1.9
    • cQube - Release V 1.8 and V 1.8.1
    • cQube - Release Notes V 1.7
    • cQube - Release Notes V 1.6 and V 1.6.1
    • cQube - Release Notes V 1.5
    • cQube - Release Notes V 1.4
    • cQube - Release Notes V 1.3
    • cQube - Release Notes V 1.2 and V 1.2.1
    • cQube - Release Notes V 1.1
    • cQube - Release Notes V 1.0
  • ๐Ÿ“‚cQube V 4.1 - Beta
    • Sunbird cQube Overview
    • cQube Product Description
    • Listen to Experts (Youtube)
    • Software Requirements
    • Acronyms
    • cQube Software Architecture
    • AWS - Network Architecture
      • Hardware requirements
      • Data Storage Locations
    • Security Implementations
    • Prerequisites for Installation process
    • New Use-Case Creation
    • cQube Setup & configuration
    • Base Installation steps
    • Base Upgradation steps
    • Workflow Installation steps
    • Workflow Upgradation steps
    • Laptop/Desktop Installation
      • Base Installation
      • Workflow Installation
      • Mock Data Processing
    • Ad-hoc analysis
    • Workflow process
    • Emission Process
    • cQube ER Diagrams
    • Data Validation after Ingestion
    • User Authentication Process
    • Admin Login Process
    • Admin Features
    • cQube Datasource Configuration
    • cQube data replay process
    • S3 Partitioning
    • Reports
    • Troubleshooting Issues
      • Data Processing-NIFI Issues
      • Data Processing-PostgreSQL Issues
      • Data Emission Issues
      • Angular & Node Issues
    • FAQs
    • Discuss
    • Report
    • Source Code
Powered by GitBook
On this page
  • cQube Architecture describes about three main layers of the application.
  • Data Input
  • Data Cleansing, Data Transformation and Data Processing
  • Data visualization

Was this helpful?

Edit on GitHub
  1. cQube V 4.1 - Beta

cQube Software Architecture

PreviousAcronymsNextAWS - Network Architecture

Last updated 2 years ago

Was this helpful?

cQube Architecture describes about three main layers of the application.

  1. Data Input

  2. Data Cleansing, Data Transformation and Data Processing

  3. Data visualization

Data Input

Once the raw data reaches to emission storage location, Nifi starts fetching the specified data sources related csv files at the scheduled time of each data source. For the future reference, Nifi keeps a copy of raw data into the data input storage location.

Data Cleansing, Data Transformation and Data Processing

Once the raw data file copy is saved into the data input storage location, NIFI inserts the raw data into the staging tables (Referred as temporary tables) of PostgreSQL where the data is ready for the data cleaning process. Data cleaning includes the data validations as well as duplicate records handlings.

The cleaned and transformed data will be inserted to the transaction tables of the PostgreSQL.

NIFI invoke the PostgreSQL quires to perform the metrics calculations on the cleaned and transformed data. Resulted data will be saved into the Aggregation tables of PostgreSQL.

The below images describes the data flow steps happening in cQube

Data visualization

As earlier said cQube is a โ€œdecision making toolโ€.

In explanation, cQubeโ€™s every report in Analytics is made up of dimensions and metrics. cQube reports provide good predictive insight and anticipates upcoming correlations at different components of the data and activities of the educational system. These reports help cQube admins to make appropriate decisions based on the insights.

cQube visualizations can be customized according to the userโ€™s views. cQube allows to create different cQube themes to create different look & feel designs into reports

Data emission will be performed periodically as per the specified time interval from different data sources with the help of an automated extraction process. Once the emission process extracts the data fields which are used by cQube, it is converted into csv formatted, pipe delimited files. The CSV data files will then be placed into the state data center by the automated process. will invoke cQube data-ingestion APIs to emit the data.

๐Ÿ“‚
Emission automated processes
cQube software architecture
cQube Dataflow