Data Processing-NIFI Issues

Where

Why

How to resolve

Error message

Error Location

Error type

Error identification (Cause)

Probable resolution

Error student_attendance.txt is not valid zip format.

Installation/logs/nifi-app.log

checking the extension

If file is not valid zip

"1. Check the file is emitted in the .Zip format or not 2. Check the file is in rar or any other then zip format. 3. If the above 2 are correct then contact cQube team."

Error while extracting the header/columns from csv file (error)

Installation/logs/nifi-app.log

Data validation

Error while extracting columns from file.

1. Check the delimiter of the csv file, delimiter should be "|" except for Diksha Datasource. Diksha delimiter " , "

Error - columns did not match

Installation/logs/nifi-app.log

Data validation

If the columns do not match with the requirement

"1. Check the column names are accoding to the cQube doc. 2. Rename the columns as per the cQube doc. 3. Clear the Nifi Queue [ Righ click on canvas , then click Empty all queues] 4. Re emit the file with correct columns."

Error - filename has invalid data type: invalid record=, Total records=

Installation/logs/nifi-app.log

Data validation

If the columns datatype doesn't match with the requirement

"1. Check the column datatypes are accoding to the cQube doc. 2. Rename the columns as per the cQube doc. 3. Clear the Nifi Queue [ Righ click on canvas , then click Empty all queues] 4. Re emit the file with correct columns."

Error while checking mirror data

Installation/logs/nifi-app.log

Data validation

when unexpected characters found in data

"1. Check for the special characters or invalid csv format file. 2. If special characters or invalid csv format exists, then clear the Nifi Queue [ Righ click on canvas , then click Empty all queues] and delete the staging / temp tables. 3. If the above steps doesn't solve the issue, then contact cQube team."

Error while selecting unique data

Installation/logs/nifi-app.log

Data validation

when unexpected characters found in data

"1. check for the special characters or invalid csv format file. 2. If special characters or invalid csv format exists, then clear the Nifi Queue [ Righ click on canvas , then click Empty all queues] and delete the staging / temp tables. 3. If the above steps doesn't solve the issue, then contact cQube team."

Error while getting null values

Installation/logs/nifi-app.log

Data validation

when unexpected characters found in data

"1. Check for the special characters or invalid csv format file .2. If special characters or invalid csv format exists, then clear the Nifi Queue [ Righ click on canvas , then click Empty all queues] and delete the staging / temp tables. 3. If the above steps doesn't solve the issue, then contact cQube team."

Error while fetching the year and month from temp table.

Installation/logs/nifi-app.log

Run time Error

When DB is down/ not in running state

"1. Check the Db status. 2. If it is stopped , start it . 3. If the problem persist after doing above steps , Contact cQube team."

Error while executing query.

Installation/logs/nifi-app.log

Run time Error

When DB is down/ not in running state

"1. Check the Db status. 2. If it is stopped , start it . 3. If the problem persist after doing above steps , Contact cQube team."

Failed to perform validation due to java.lang.OutOfMemoryError: Java heap space

Installation/logs/nifi-app.log

Run time Error - Out of Heap Memory

when Heap memory is consumed full.

1. Contact cQube team.

Amazon S3 due to Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown;

Installation/logs/nifi-app.log

Run time Error

When large number of file writes happens at same time

1. Contact cQube team.

Error while checking diksha summary rollup request status from table, Error=

Installation/logs/nifi-app.log

Run time Error

When DB is down/ not in running state

"1. Check the Db status. 2. If it is stopped , start it . 3. If the problem persist after doing above steps , Contact cQube team."

Invoke HTTP Routing to Failure due to exception: Failed to connect to :8082: java.net.ConnectException: Failed to connect to :8082

Installation/logs/nifi-app.log

Configuration Error

Diksha API Call

"1. Check whether API is working ,test it using curl or postman. 2. If it is not working , contact the respective API team. 3.If the above steps doesn't solve the issue, contact cQube team."

InvokeHTTP Error codes 4XX

Installation/logs/nifi-app.log

Configuration Error

Diksha API Call

"1. Check whether ur able make the api call using curl or postman. 2. If it is not working , contact the respective API team. 3.If the above steps doesn't solve the issue, contact cQube team."

"Error while downloading the file.File url = "

Installation/logs/nifi-app.log

Run time Error

Diksha API Call

"1. Check the file expiry time from diksha_api_meta table. 2. If file ur is expired , then clear the Nifi queues. 3. delete the today's request and re run the diksha processor. 4. If problem persist after performing above 3 steps, Contact cQube team."

NO DATA FOUND - DATASET (PROGRESS-EXHAUST) for Request:

Installation/logs/nifi-app.log

Configuration Error / API

Diksha API Call

"1. Check whether you have emitted batch ids file. 2. Check whether Diksha API is returning data using CURL / POSTMAN. 3. If problem persist after performing above 2 steps, Contact cQube team."

NO DATA FOUND - DATASET (SUMMARY-ROLLUP) for Request:

Installation/logs/nifi-app.log

Configuration Error / API

Diksha API Call

"1. Check whether Diksha API is returning data using CURL / POSTMAN. 2. If problem persist after performing above the step 1, Contact cQube team."

Routing to failure : unable to unmarshal JSON to an Object

Installation/logs/nifi-app.log

Run time Error

when unexpected characters found in output data

"1. check for the special characters or invalid csv format file. 2. If special characters or invalid csv format exists, then clear the Nifi Queue [ Righ click on canvas , then click Empty all queues] and delete the staging / temp tables. 3. If the above steps doesn't solve the issue, then contact cQube team."

Last updated