> For the complete documentation index, see [llms.txt](https://cqube.sunbird.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cqube.sunbird.org/use-cqube/processing-of-data/nifi-section.md).

# Nifi section

 Nifi in cQube is used to create a data flow and automate the processing of data with the help of processor groups. These processor groups are responsible for accessing the data files from the cloud storage and moving it into the processing-ms codebase. Once the data files are moved the commands will be executed which will ingest the data into the database. 

Here we have different processor groups to process the data(Nifi canvas).\
1\. To run adapters.

2\. To run all programs.

3\. To run program wise.

The Processor groups in nifi can be scheduled using API called [scheduled API ](https://cqube.sunbird.org/use-cqube/ingesting-the-data/steps-to-ingest-data-files/scheduled-api)

<figure><img src="https://lh7-us.googleusercontent.com/RTAA79E4m4LW4kNxJ-x9lqS7LHwui1rMc-m-w4Pjx4UwURcFH9xhzn0xW6H4_CJDE_eyxI-RZNJIYcVLeJzsCOiroK2PXlI8Cdpoq5sDKdW3j6V5KxXoARgNBTzwA72tYx6M4FIr4i1CzhtAkDLn6cw" alt=""><figcaption></figcaption></figure>

**1. To run adapters:** This processor group is used to run the adapters shell script code.

<figure><img src="https://lh7-us.googleusercontent.com/vYOhhEZhqKmGv3WYCUQaPsnNHsHu4JuHhK5IRCXxrNf_O8-KfidyQElnuQb6BJizTMXo0moKHTwQ5KxMWSVGYnuJaLP_ATOw4wNBlhMTCdsz-ufJs-jyJr2QpwxENzKssec2EdeqRZGBHRQof4GRUdM" alt=""><figcaption></figcaption></figure>

<figure><img src="https://lh7-us.googleusercontent.com/1-B6z29UGf5kF5HNL6k9EQxJOJokP2IKIpIbmI6M-m9ERkP4xS1IZ9DtjNd1ccleV5Ig9QawB1oqUjIFfEOeACVQVGuTH4U-9270jHkTm0GsxwLZaB75u76P77-ulwre6OYiOMXKslcu2rE-WPZDdPY" alt=""><figcaption></figcaption></figure>

**2. To run all programs:** This processor group is used to move files from process\_input folder to processing\_ms folder according to folder structures. .i.e., (dimensions/,programs/). Then it will run yarn cli ingest(where it will ingest all schemas and grammars to the database) and yarn cli ingest-data (where it will ingest all data files to the database) commands.

<figure><img src="https://lh7-us.googleusercontent.com/F-AkuAxqcUxj4Lko-3lnAficTq2tDQh_x_GXKCdPsqy4ZJUvozqRMRS3fCtFplDC-R8qDtwdcvJ19phlJGY77UqIZ75cB0gi_DiuSYCvtIu8oH6lwTbO0K2O6UlcXbYM8MDEJ4SZg09S0dGBkbwuxbs" alt=""><figcaption></figcaption></figure>

**3.To run program wise:** This processor group is used to move particular program files from process\_input folder to processing\_ms folder. Then it will run yarn cli ingest and yarn cli ingest-data --filter=’program\_name’.(where it will ingest particular program data files).

<figure><img src="https://lh7-us.googleusercontent.com/WLJMhDjsBMTveEgWHBrLVTNqb62DdiRsrxyh2K_TRdnxWVb29xnKDv5r2fnKkV3bfAa9ekyK4kR-vLIRtPQXpNOtxdDEer7prp24KTK3NKkt-vMYn8Tu0EiBPivT_tjnsV63iKYHhZQ_2YsOXd5VKsQ" alt=""><figcaption></figcaption></figure>

\
In order to automate these processor groups we have written REST-APIs. Please look into the below link to understand th Nifi codebase.

\
[Nifi-Rest-api’s code](https://github.com/Sunbird-cQube/generator-ms/blob/dev/static_processor_group/add_nifi_template.py)

<br>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://cqube.sunbird.org/use-cqube/processing-of-data/nifi-section.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.