How to add a new program (end to end)

Note: These steps are for adding a new program

Create event and dimension Grammar files:

Design the event and dimension grammar files for your program. This grammar defines the structure and relationships between different data elements, such as event attributes and dimensions.

• Event grammar: The event grammar defines the structure of the events to be ingested into the system. It includes information such as the event name, its attributes (e.g. event timestamp, user ID, etc.), and data types.

• Dimension grammar: The dimension grammar defines the dimensions, or attributes, that will be used to group and filter events. It includes information such as the dimension name, its attributes (e.g. dimension value, category, etc.), and data types.

Create a Config.json

Config.json provides directive to processing engine on how to process the grammar and data files, where they are stored, etc. it should be placed in /ingest/<VSK or NVSK>/

globals : global settings

dimensions : It includes, file format, location and namespace

programs : program specific settings like namespace, description, input location, output location, dimensions which should be considered etc.

Only programs mentioned under config.json will be considered for ingestion.

15.3 Upload the event and dimension grammar spec:

• Convert the Event and Dimension Grammar to JSON format as given in the below example

Example: if there is a event with two columns

school_id and school_name

Below would be the JSON object

{

"program": "school-attendance",

"input": {

"type": "object",

"properties": {

"school_id": {

"type": "string",

"shouldNotNull": true

},

"school_name": {

"type": "string",

"shouldNotNull": true

}

}

}

• Dimension grammar : Use the /spec/dimension API to upload the dimension grammar

• Place the dimension grammar file as -dimension.grammar.csv in processing-ms/impl/c-qube/ingest/VSK or NVSK/dimensions/ folder as seen below or as per the config.json settings

Contents of the district-dimension.grammar.csv

• Event schema can be ingested in 2 ways:

  • Though API Method

Event spec API: spec/event

• Through YARN CLI

YARN CLI INGEST

This command is used to ingest Event and Dimension Spec and Dimension Data based on the configuration we have

• Create a <new-program> folder in processing-ms/impl/c-qube/ingest/<state_code>/programs\

• Place the event grammar file as <xyz>-event.grammar.csv in processing-ms/impl/c-qube/ingest/<state_code>/programs/<new-program> folder as seen below (example for diksha program)

This would be based on default config.json or else it should be as per config.json

Contents of the district-dimension.grammar.csv

Data Ingestion:

This process may involve uploading the CSV file or connecting to the database directly. Following links will provide more details on the process.

• Uploading through CSV
• Connecting to Database

Add event data on a regular frequency: To keep your program up-to-date, continuously add new event data to the platform. Determine an appropriate frequency for updating the data, such as daily, weekly, or monthly.w Ensure that the new data is compatible with the existing event and dimension schema.

UI Program Addition

• Create a new module manually or using cli common ng g m <module-name> in the below folder in the angular code base: src/app/views

• Add the folders config and pages, where config holds a configuration file for the reports in this program and pages contains the code of all the tabs/reports.

• Add the routing to the program component in the app-routing module using lazy-loading

• Add the configuration of this program to the main_matric.csv present in the root folder of query builder.( To add this program to side nav and summary statistics) as shown below.

• Next add the programId filled in the above file to another file Program Selector in the query Builder root folder itself. The Program Selector file takes input from the ansible during installation to update which programs to be visible in the dashboard. If a new program is added after installation manually fill the programId and set the status true for that program to be visible in the dashboard.

• Configuration file of the newly added program should be added to the below file:

src/app/core/config/configMapping.ts

All the configuration files are mapped at a place for easy access , This object will be used to map each row in main_metric with program wise configuration files with the respective programIds.

Later one can add the desired KPI to the above program created following the below steps