Sunbird cQube
  • 📖KNOW ABOUT CQUBE
    • What is cQube & what does it solve
      • Business case
      • cQube ed
      • Design Principles
    • cQube adoptions
    • Discuss more about cQube
  • 👍TRY CQUBE
    • cQube on Gitpod
  • 🌅WHAT IS NEXT IN CQUBE
    • cQube Roadmap
  • 💻TECHANICAL OVERVIEW
    • Architecture
    • Design Principles
    • Key Components
    • Details of microservices
  • 👩‍💻Get started on cQube
    • Suggested Team Structure
    • Hardware Requirements
    • Prerequisites Checklist
    • Checking pre existing ports
    • Instance Creation
    • Copying SSL Certificate
  • 🛃USE CQUBE
    • How can I install cQube?
      • High level understanding of deployment
      • Oracle Installation
      • AWS Installation
      • SDC Installation
      • Azure Installation
    • How to prepare the data
      • Schema
      • How to prepare schemas for dimension files
      • How to prepare schemas for event files
      • Examples of dimension & event files
      • All cQube schemas used for VSK
    • Ingesting the data
      • High-level understanding of ingestion
      • Steps to ingest schema
      • Steps to ingest data files
        • Upload the .csv data file for state specific programs using ingestion API
        • API to upload starter pack data files for NVSK programs
        • Get file status API
        • Scheduled API
      • Error during ingestion
        • Error Monitoring
        • Common errors in data files during ingestion
      • Processor group name
    • Adapter details
    • Postman details
    • Processing of data
      • Data Processing using CLI command
      • API Details for Nifi-Rest
      • Nifi section
    • Visualizing the data
      • High level understanding of how visualizations work in cQube
      • Programs and reports out-of-the-box
      • Enhance /Customize cQube
        • Available customizations
          • Changing Dashboard Logos and Headers
          • Changing Program Name, Icon and Side Menu Sequence
          • Adding a new KPI
          • Adding a Map KPI into dashboard ms
          • Table Drill Down Customization
          • Adding a Scatter Plot KPI into dashboard ms
          • Configure default date range across app/specific report
        • How to add a New Indicator
        • How to add a new report in an existing program
        • How to add a new program (end to end)
    • Additional Features
      • Public/Private dashboards
      • Role based access control
      • Saving geographical preferences
      • Admin Panel
        • Data Debugger
        • Schema Generator
        • System Monitoring
    • Adding Users
      • Adding an individual user
      • Adding bulk users
  • 🖥️MONITOR cQUBE
    • Infra health monitoring
    • Usage monitoring
  • 🔎QA testing
    • Testing approaches & activities
    • Manual & Automated testing
    • Functional Testing
      • Smoke Testing
      • Functional tests
      • Regression Testing
      • System Testing
    • Non Functional Testing
      • Performance Testing
        • Load Testing
        • Volume Testing
        • Performance testing results
    • Test for One-Step Installation
    • Test for Ingestion
    • Test for nifi processing
    • Test for UI Application
    • Test for KPIs
  • ☀️DEPLOYMENT PROCESS
    • State List
    • AWS Deployment
    • SDC Deployment
    • Adapter Details During the Processing
  • 🈴UPGRADING TO LATEST VERSION
    • How can I upgrade cQube to the latest release
  • 🆘Common issues and their solutions
    • Deployment & ingestion related issues & their solutions
  • ⏱️Standard Operating Procedure
    • Reporting a Bug
    • Protocol for issue reporting & resolution
    • Suggesting Enhancements
    • Raising a PR
  • ❓Frequently Asked Questions
    • Running List
  • 🧑‍🏫🧑🏫 Recording of trainings
    • Link to the training videos
  • 🧠Key Terms & Concepts
    • Definitions
  • 🚀cQube Release Notes
    • cQube - Release V 5.0.5
    • cQube - Release V 5.0.3
    • cQube - Release V 5.0.2
    • cQube - Release V 5.0.1
    • cQube - Release V 5.0
    • cQube - Release V 4.1-beta
    • cQube - Release V 4.0-beta
    • cQube - Release V 4.0-alpha
    • cQube - Release V 3.7
    • cQube - Release V 3.6
    • cQube - Release V 3.5
    • cQube - Release V 3.4
    • cQube - Release V 3.3
    • cQube - Release V 3.2
    • cQube - Release V 3.1
    • cQube - Release V 3
    • cQube - Release V 2
    • cQube - Release V 1.13 and V 1.13.1
    • cQube - Release V 1.12 and V 1.12.1
    • cQube - Release V 1.11
    • cQube - Release V 10 and V 10.1
    • cQube - Release V 1.9
    • cQube - Release V 1.8 and V 1.8.1
    • cQube - Release Notes V 1.7
    • cQube - Release Notes V 1.6 and V 1.6.1
    • cQube - Release Notes V 1.5
    • cQube - Release Notes V 1.4
    • cQube - Release Notes V 1.3
    • cQube - Release Notes V 1.2 and V 1.2.1
    • cQube - Release Notes V 1.1
    • cQube - Release Notes V 1.0
  • 📂cQube V 4.1 - Beta
    • Sunbird cQube Overview
    • cQube Product Description
    • Listen to Experts (Youtube)
    • Software Requirements
    • Acronyms
    • cQube Software Architecture
    • AWS - Network Architecture
      • Hardware requirements
      • Data Storage Locations
    • Security Implementations
    • Prerequisites for Installation process
    • New Use-Case Creation
    • cQube Setup & configuration
    • Base Installation steps
    • Base Upgradation steps
    • Workflow Installation steps
    • Workflow Upgradation steps
    • Laptop/Desktop Installation
      • Base Installation
      • Workflow Installation
      • Mock Data Processing
    • Ad-hoc analysis
    • Workflow process
    • Emission Process
    • cQube ER Diagrams
    • Data Validation after Ingestion
    • User Authentication Process
    • Admin Login Process
    • Admin Features
    • cQube Datasource Configuration
    • cQube data replay process
    • S3 Partitioning
    • Reports
    • Troubleshooting Issues
      • Data Processing-NIFI Issues
      • Data Processing-PostgreSQL Issues
      • Data Emission Issues
      • Angular & Node Issues
    • FAQs
    • Discuss
    • Report
    • Source Code
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub
  1. USE CQUBE
  2. Additional Features
  3. Admin Panel

Data Debugger

It analyzes the data and reports the user about the errored records in the raw data file of an event or dimension. User has to select an event or dimension schema which is needed to validate the records of uploaded events or dimension files. After that debugger will analyze the uploaded data against the selected schema and it reports back to the user with the errored records. This feature helps the user to analyze the data before ingesting the data for data processing.

Data Debugger analyzes the fed data and reports and identifies the errors in them, if any, in an event or dimension files.

How it works:

To validate the raw data of either dimension or an event, first:

  1. Upload the grammar file for the dimension or event that you want to validate.

  2. Validate the grammar file.

  3. Validate the raw data against the uploaded grammar file.

Steps to validate the Raw Data file:

(Please note - The raw data file should be in CSV format.)

  1. Check 'Dimension Grammar' is present in the dropdown menu.

  2. Upload the dimension file.

  3. Click on the 'Validate Grammar' Tab.

Now, the system will show errors if any, in the screen for the uploaded file. If the file does not contain any errors, the user will be directed to the next screen.

Here are more information about each step above.

  1. Upload the grammar file for the dimension or event that you want to validate.

The grammar file is a CSV file that defines the structure of the data that you want to validate. It contains information about the different fields in the data, such as the field name, data type, and constraints.

Following screenshot of validate grammar of data debugger

Following is the sample grammar file block dimension:

Note: Include primary key if it is a dimension grammar file.

  1. Validate the grammar file.

Once you have uploaded the grammar file, you need to validate it to make sure that it is correct. The validation process will check for errors in the grammar file, such as missing fields or invalid data types.

Here is a sample screenshot of a table that might be displayed to the user:

If the uploaded grammar file is valid, then a Validate Data button will be enabled for the user. Using the Validate Data button, the user can go into the validation step.

Following screenshot will be displayed to the user

  1. Validate the raw data against the uploaded grammar file.

Once the grammar file has been validated, you can validate the raw data against it. The validation process will check for errors in the raw data, such as missing fields or invalid values.

If there are no errors in the raw data file, then the validation will be successful. Otherwise, the validation will fail, and you will be given a report of the errors.

Following screenshot for validate the data

If there are any errors in the uploaded data, then the errors will be reported to the user in the form of a table. Following is a screenshot of the error screen.

If the file is valid, then a alert message will be displayed to the user, which means that the data is ready to be ingested. Following is the screenshot:

Steps:

  • Grammar Validation

The user must first choose the grammar file type to validate, either dimension or event. Then, the user must upload the grammar file. After uploading the file, the debugger analyzes the grammar and reports any errors to the user. If there are no errors, the debugger shows a message indicating that the schema is valid.

  • Data Validation

The user must first choose the data file type to validate, either dimension or event. Then, the user must upload the raw data file. After uploading the file, the debugger analyzes the data against the grammar uploaded in the previous step and reports any errors to the user. If there are no errors, the debugger shows a message indicating that the schema is valid.

PreviousAdmin PanelNextSchema Generator

Last updated 1 year ago

Was this helpful?

🛃