Sunbird cQube
  • πŸ“–KNOW ABOUT CQUBE
    • What is cQube & what does it solve
      • Business case
      • cQube ed
      • Design Principles
    • cQube adoptions
    • Discuss more about cQube
  • πŸ‘TRY CQUBE
    • cQube on Gitpod
  • πŸŒ…WHAT IS NEXT IN CQUBE
    • cQube Roadmap
  • πŸ’»TECHANICAL OVERVIEW
    • Architecture
    • Design Principles
    • Key Components
    • Details of microservices
  • πŸ‘©β€πŸ’»Get started on cQube
    • Suggested Team Structure
    • Hardware Requirements
    • Prerequisites Checklist
    • Checking pre existing ports
    • Instance Creation
    • Copying SSL Certificate
  • πŸ›ƒUSE CQUBE
    • How can I install cQube?
      • High level understanding of deployment
      • Oracle Installation
      • AWS Installation
      • SDC Installation
      • Azure Installation
    • How to prepare the data
      • Schema
      • How to prepare schemas for dimension files
      • How to prepare schemas for event files
      • Examples of dimension & event files
      • All cQube schemas used for VSK
    • Ingesting the data
      • High-level understanding of ingestion
      • Steps to ingest schema
      • Steps to ingest data files
        • Upload the .csv data file for state specific programs using ingestion API
        • API to upload starter pack data files for NVSK programs
        • Get file status API
        • Scheduled API
      • Error during ingestion
        • Error Monitoring
        • Common errors in data files during ingestion
      • Processor group name
    • Adapter details
    • Postman details
    • Processing of data
      • Data Processing using CLI command
      • API Details for Nifi-Rest
      • Nifi section
    • Visualizing the data
      • High level understanding of how visualizations work in cQube
      • Programs and reports out-of-the-box
      • Enhance /Customize cQube
        • Available customizations
          • Changing Dashboard Logos and Headers
          • Changing Program Name, Icon and Side Menu Sequence
          • Adding a new KPI
          • Adding a Map KPI into dashboard ms
          • Table Drill Down Customization
          • Adding a Scatter Plot KPI into dashboard ms
          • Configure default date range across app/specific report
        • How to add a New Indicator
        • How to add a new report in an existing program
        • How to add a new program (end to end)
    • Additional Features
      • Public/Private dashboards
      • Role based access control
      • Saving geographical preferences
      • Admin Panel
        • Data Debugger
        • Schema Generator
        • System Monitoring
    • Adding Users
      • Adding an individual user
      • Adding bulk users
  • πŸ–₯️MONITOR cQUBE
    • Infra health monitoring
    • Usage monitoring
  • πŸ”ŽQA testing
    • Testing approaches & activities
    • Manual & Automated testing
    • Functional Testing
      • Smoke Testing
      • Functional tests
      • Regression Testing
      • System Testing
    • Non Functional Testing
      • Performance Testing
        • Load Testing
        • Volume Testing
        • Performance testing results
    • Test for One-Step Installation
    • Test for Ingestion
    • Test for nifi processing
    • Test for UI Application
    • Test for KPIs
  • β˜€οΈDEPLOYMENT PROCESS
    • State List
    • AWS Deployment
    • SDC Deployment
    • Adapter Details During the Processing
  • 🈴UPGRADING TO LATEST VERSION
    • How can I upgrade cQube to the latest release
  • πŸ†˜Common issues and their solutions
    • Deployment & ingestion related issues & their solutions
  • ⏱️Standard Operating Procedure
    • Reporting a Bug
    • Protocol for issue reporting & resolution
    • Suggesting Enhancements
    • Raising a PR
  • ❓Frequently Asked Questions
    • Running List
  • πŸ§‘β€πŸ«πŸ§‘πŸ« Recording of trainings
    • Link to the training videos
  • 🧠Key Terms & Concepts
    • Definitions
  • πŸš€cQube Release Notes
    • cQube - Release V 5.0.5
    • cQube - Release V 5.0.3
    • cQube - Release V 5.0.2
    • cQube - Release V 5.0.1
    • cQube - Release V 5.0
    • cQube - Release V 4.1-beta
    • cQube - Release V 4.0-beta
    • cQube - Release V 4.0-alpha
    • cQube - Release V 3.7
    • cQube - Release V 3.6
    • cQube - Release V 3.5
    • cQube - Release V 3.4
    • cQube - Release V 3.3
    • cQube - Release V 3.2
    • cQube - Release V 3.1
    • cQube - Release V 3
    • cQube - Release V 2
    • cQube - Release V 1.13 and V 1.13.1
    • cQube - Release V 1.12 and V 1.12.1
    • cQube - Release V 1.11
    • cQube - Release V 10 and V 10.1
    • cQube - Release V 1.9
    • cQube - Release V 1.8 and V 1.8.1
    • cQube - Release Notes V 1.7
    • cQube - Release Notes V 1.6 and V 1.6.1
    • cQube - Release Notes V 1.5
    • cQube - Release Notes V 1.4
    • cQube - Release Notes V 1.3
    • cQube - Release Notes V 1.2 and V 1.2.1
    • cQube - Release Notes V 1.1
    • cQube - Release Notes V 1.0
  • πŸ“‚cQube V 4.1 - Beta
    • Sunbird cQube Overview
    • cQube Product Description
    • Listen to Experts (Youtube)
    • Software Requirements
    • Acronyms
    • cQube Software Architecture
    • AWS - Network Architecture
      • Hardware requirements
      • Data Storage Locations
    • Security Implementations
    • Prerequisites for Installation process
    • New Use-Case Creation
    • cQube Setup & configuration
    • Base Installation steps
    • Base Upgradation steps
    • Workflow Installation steps
    • Workflow Upgradation steps
    • Laptop/Desktop Installation
      • Base Installation
      • Workflow Installation
      • Mock Data Processing
    • Ad-hoc analysis
    • Workflow process
    • Emission Process
    • cQube ER Diagrams
    • Data Validation after Ingestion
    • User Authentication Process
    • Admin Login Process
    • Admin Features
    • cQube Datasource Configuration
    • cQube data replay process
    • S3 Partitioning
    • Reports
    • Troubleshooting Issues
      • Data Processing-NIFI Issues
      • Data Processing-PostgreSQL Issues
      • Data Emission Issues
      • Angular & Node Issues
    • FAQs
    • Discuss
    • Report
    • Source Code
Powered by GitBook
On this page

Was this helpful?

Edit on GitHub
  1. USE CQUBE
  2. Ingesting the data
  3. Steps to ingest data files

Upload the .csv data file for state specific programs using ingestion API

PreviousSteps to ingest data filesNextAPI to upload starter pack data files for NVSK programs

Last updated 1 year ago

Was this helpful?

Events, Dimensions can be uploaded into respective cloud storage via the ingestion APIs. The event upload is done for only state programs using this API.

The jwt token has to be passed in the API headers.

Step 1: create a postman request and add details.

API Endpoint: <domain_name>/api/ingestion/new_programs

HTTP Method: POST

This API will import the dimension csv and upload it into the process_input folder in the cloud if there are no errors.

  • file : Attach the CSV file for the importing

  • ingestion_type : Specify the type of ingestion (value should be event when uploading the event data file. If it is a dimension data file it should be dimension)

  • ingestion_name : Name of the event or dimension

  • program_name: Name of the state program (This key is required only for ingestion_type = β€˜event’)

Note: The value for the key program_name should be the same name as the folder name present inside ingest->VSK or NVSK->programs folder in processing-ms.

The above link has a list of programs array.

Example: For a school attendance program the folder name is school-attendance so the same name should be given while passing the value for program_name.

"files": "./ingest/VSK/programs/school-attendance" => The folder name can be identified as school-attendance.

Step 3:

Click on the send button for request and if the request is successful the user should see a response message. Please refer to the below screenshot.

After successful execution of the csv import api we get the response and we can see the file status indicating if the file is uploaded or not using GET file status API. If the file is successfully uploaded we will get the response as uploaded and if there are any errors it will send us the response indicating there was an error in the file.

Step 2: Build the request body with reference to YAML file. The request body for the above API is attached here for yaml: Provide valid input details for the parameters shown below.

Example:

πŸ›ƒ
https://github.com/Sunbird-cQube/spec-ms/blob/dev/spec.yaml
https://github.com/Sunbird-cQube/processing-ms/blob/dev/impl/c-qube/ingest/VSK/config.json