Sunbird cQube
  • 📖KNOW ABOUT CQUBE
    • What is cQube & what does it solve
      • Business case
      • cQube ed
      • Design Principles
    • cQube adoptions
    • Discuss more about cQube
  • 👍TRY CQUBE
    • cQube on Gitpod
  • 🌅WHAT IS NEXT IN CQUBE
    • cQube Roadmap
  • 💻TECHANICAL OVERVIEW
    • Architecture
    • Design Principles
    • Key Components
    • Details of microservices
  • 👩‍💻Get started on cQube
    • Suggested Team Structure
    • Hardware Requirements
    • Prerequisites Checklist
    • Checking pre existing ports
    • Instance Creation
    • Copying SSL Certificate
  • 🛃USE CQUBE
    • How can I install cQube?
      • High level understanding of deployment
      • Oracle Installation
      • AWS Installation
      • SDC Installation
      • Azure Installation
    • How to prepare the data
      • Schema
      • How to prepare schemas for dimension files
      • How to prepare schemas for event files
      • Examples of dimension & event files
      • All cQube schemas used for VSK
    • Ingesting the data
      • High-level understanding of ingestion
      • Steps to ingest schema
      • Steps to ingest data files
        • Upload the .csv data file for state specific programs using ingestion API
        • API to upload starter pack data files for NVSK programs
        • Get file status API
        • Scheduled API
      • Error during ingestion
        • Error Monitoring
        • Common errors in data files during ingestion
      • Processor group name
    • Adapter details
    • Postman details
    • Processing of data
      • Data Processing using CLI command
      • API Details for Nifi-Rest
      • Nifi section
    • Visualizing the data
      • High level understanding of how visualizations work in cQube
      • Programs and reports out-of-the-box
      • Enhance /Customize cQube
        • Available customizations
          • Changing Dashboard Logos and Headers
          • Changing Program Name, Icon and Side Menu Sequence
          • Adding a new KPI
          • Adding a Map KPI into dashboard ms
          • Table Drill Down Customization
          • Adding a Scatter Plot KPI into dashboard ms
          • Configure default date range across app/specific report
        • How to add a New Indicator
        • How to add a new report in an existing program
        • How to add a new program (end to end)
    • Additional Features
      • Public/Private dashboards
      • Role based access control
      • Saving geographical preferences
      • Admin Panel
        • Data Debugger
        • Schema Generator
        • System Monitoring
    • Adding Users
      • Adding an individual user
      • Adding bulk users
  • 🖥️MONITOR cQUBE
    • Infra health monitoring
    • Usage monitoring
  • 🔎QA testing
    • Testing approaches & activities
    • Manual & Automated testing
    • Functional Testing
      • Smoke Testing
      • Functional tests
      • Regression Testing
      • System Testing
    • Non Functional Testing
      • Performance Testing
        • Load Testing
        • Volume Testing
        • Performance testing results
    • Test for One-Step Installation
    • Test for Ingestion
    • Test for nifi processing
    • Test for UI Application
    • Test for KPIs
  • ☀️DEPLOYMENT PROCESS
    • State List
    • AWS Deployment
    • SDC Deployment
    • Adapter Details During the Processing
  • 🈴UPGRADING TO LATEST VERSION
    • How can I upgrade cQube to the latest release
  • 🆘Common issues and their solutions
    • Deployment & ingestion related issues & their solutions
  • ⏱️Standard Operating Procedure
    • Reporting a Bug
    • Protocol for issue reporting & resolution
    • Suggesting Enhancements
    • Raising a PR
  • ❓Frequently Asked Questions
    • Running List
  • 🧑‍🏫🧑🏫 Recording of trainings
    • Link to the training videos
  • 🧠Key Terms & Concepts
    • Definitions
  • 🚀cQube Release Notes
    • cQube - Release V 5.0.5
    • cQube - Release V 5.0.3
    • cQube - Release V 5.0.2
    • cQube - Release V 5.0.1
    • cQube - Release V 5.0
    • cQube - Release V 4.1-beta
    • cQube - Release V 4.0-beta
    • cQube - Release V 4.0-alpha
    • cQube - Release V 3.7
    • cQube - Release V 3.6
    • cQube - Release V 3.5
    • cQube - Release V 3.4
    • cQube - Release V 3.3
    • cQube - Release V 3.2
    • cQube - Release V 3.1
    • cQube - Release V 3
    • cQube - Release V 2
    • cQube - Release V 1.13 and V 1.13.1
    • cQube - Release V 1.12 and V 1.12.1
    • cQube - Release V 1.11
    • cQube - Release V 10 and V 10.1
    • cQube - Release V 1.9
    • cQube - Release V 1.8 and V 1.8.1
    • cQube - Release Notes V 1.7
    • cQube - Release Notes V 1.6 and V 1.6.1
    • cQube - Release Notes V 1.5
    • cQube - Release Notes V 1.4
    • cQube - Release Notes V 1.3
    • cQube - Release Notes V 1.2 and V 1.2.1
    • cQube - Release Notes V 1.1
    • cQube - Release Notes V 1.0
  • 📂cQube V 4.1 - Beta
    • Sunbird cQube Overview
    • cQube Product Description
    • Listen to Experts (Youtube)
    • Software Requirements
    • Acronyms
    • cQube Software Architecture
    • AWS - Network Architecture
      • Hardware requirements
      • Data Storage Locations
    • Security Implementations
    • Prerequisites for Installation process
    • New Use-Case Creation
    • cQube Setup & configuration
    • Base Installation steps
    • Base Upgradation steps
    • Workflow Installation steps
    • Workflow Upgradation steps
    • Laptop/Desktop Installation
      • Base Installation
      • Workflow Installation
      • Mock Data Processing
    • Ad-hoc analysis
    • Workflow process
    • Emission Process
    • cQube ER Diagrams
    • Data Validation after Ingestion
    • User Authentication Process
    • Admin Login Process
    • Admin Features
    • cQube Datasource Configuration
    • cQube data replay process
    • S3 Partitioning
    • Reports
    • Troubleshooting Issues
      • Data Processing-NIFI Issues
      • Data Processing-PostgreSQL Issues
      • Data Emission Issues
      • Angular & Node Issues
    • FAQs
    • Discuss
    • Report
    • Source Code
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub
  1. cQube V 4.1 - Beta

cQube Datasource Configuration

PreviousAdmin FeaturesNextcQube data replay process

Last updated 2 years ago

Was this helpful?

cQube supports creating new data sources using configuration files. Through which User can enter the required details like the name of the new data source, columns , data types, Metric type and type of visualization reports.

Through configuration files, Users can also create new reports with the existing data sources.(coming soon)

List of report types supported through configuration

  • Map report

  • Table report

  • Heat chart (coming soon)

  • Bar chart (coming soon)

  • Multi Bar chart (coming soon)

  • Pie chart (coming soon)

  • Line chart (coming soon)

cQube latest Software architecture

NOTE: Python files for new metrics (in the above architecture) is not be supported now, will be enabled in future.

Below is the link for sample configuration file format

steps to create the configuration file

secion-1 []

  • Table_name: The Table/datasource name

  • Columns: The column names in the raw data file.

  • Data_Type: The postgres supported data types.

  • Constraints: The postgres supported contraints

  • Ref_Table: If constraint is foreign key, then specify the referenced table name.

  • Ref_column: If constraint is foreign key, then specify the referenced column name.

  • Null_validation_required: column to be included in null validation.

  • check_same_id: Column to be included in same id duplicate validation.

secion-2 [table_1_trans]

  • level_of_data: Type/level of raw data. Example: school or student level.

  • filters_required: dimensions of data required in UI Example: District/block/cluster/ etc..

  • time_selections: supported time selections

    • daily (maximum 30 days)

    • weekly (maximum 15 weeks)

    • year_and_month

    • last_30_days

    • last_7_days

    • overall

  • date_column_to_filter: date column to consider for metrics generation.

  • nifi_select_columns: columns to display on UI.

secion-2 [type_of_data]

  • metric_type: supported metric_type

    • sum.

    • count.

Steps to configure the new data source

  • Prepare the configuration file according to the above link, save as csv and compress to .zip format.

  • Schedule the configure_datasource processor group under admin console --> NiFi scheduler.

  • Emit the zipped configuration file to the emission directory in the below directory structure. Filename should be the datasource name.

├── configure_datasource
│   └── datasourcename.zip
│       └── datasourcename.csv
  • Navigate to the admin console/Logs and check the NiFi app log

On successful file processing, you should see the below message

**** Succesfully Executed! ****
file: <filename> executed succesfully.
  • Navigate to the admin console page, Click on configuration, Under configurable Data Source section click the drop drown and choose the new data source and click Submit.

  • Wait until the timer stops. Then go to the scheduler page, On successful build, you should see the new data source name under the list and schedule the newly added data source.

  • Emit the file into new_datasource_name/ directory into emission directory.

  • After data processing,the configured reports will be available with data in cQube Dashboard.

📂
sample configuration file
cQube latest software architecture