#Write and Marge CSV file in node

1 messages · Page 1 of 1 (latest)

grand widget
#

I have API end point over there I'm get data CSV data using axios.
It's give me 5-6 CSV file data , So I have to write all csv file data into one File and also have to filter over that csv file data write into other csv_filter file.

For that I have tried :

  1. csv-parser
  2. csv
  3. fast-csv
  4. json2csv
  5. write-csv

but this not write data proper in csv file.
Some time header multiple time and some time data not write in order I want.

And one more problem I have that on reading csv file if I add data into Array it's give me this problem

It's give me Last few GCs
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

whole needle
#

If it's not in the correct order, you have to control the operations to be synchronous instead of asynchronous. If there are duplicate headers, then something is wrong with the reducing function you are using. If you are running out of memory, then it's just that the amount of data you are trying to load into memory exceeds your system memory.

spiral mango
#

Working with clean data is the most difficult part of working with any data. From your description it sounds as though the files may be different from one of another.

My suggestion is to read all of the CSV files and get them in a common, workable data structure, like JSON. Then iterate through those JSON objects to create your one needed CSV file. This way you can write rules around missing or bad data and handle things better than trying to stack CSV files together.

grand widget
#

I am actually doing what use describe here @spiral mango
But when I am add data object into array that it's give me error : heap out of memory
And if I write single-single data into file it's give me error : too many files are open

spiral mango
#

That’s a hard one to solve without seeing the code.

grand widget
#

This is my code
I have used csv-parser for parse csv and make csv data into json format and csv-writer to write data into csv file

image.png
spiral mango
#

Based on your original error and the code there are multiple things to unpack here.

Some time header multiple time and some time data not write in order I want.
I assume you're writing the data out, so review that area of the code. Is it writing different results each time it's run?
if I add data into Array it's give me this problem

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
This one I am pretty sure the array your loading all the CSV data into (results) is running out of heap memory. In node the array and all the objects in the array are stored on the heap. I don't know where this code is being run, etc. to know what the allocated heap size is.
This is not uncommon when you have 1M+ row data sets your stuffing into memory.

whole needle
#

Handling large CSV files, especially when they are fetched from an API, can be tricky. Memory issues can arise when trying to process everything in-memory. Here's a general approach to handle this scenario efficiently:

Stream the API Data: Instead of fetching all the CSV data at once, use a streaming approach to read the CSV data from the API. This ensures that you're only processing a chunk of data at any given time, rather than loading everything into memory.

Write CSV Directly: Instead of collecting all CSV rows in memory, write them directly to the target CSV file as they're processed. This way, you're not storing large arrays in memory.

Filter and Write to a Second CSV: While writing to the main CSV file, you can also filter rows based on your criteria and write the filtered rows to a separate CSV file.

Handle Headers Properly: To avoid writing headers multiple times, check if the target CSV file already exists. If it doesn't, write the headers; otherwise, skip them.

Memory Issue: The "JavaScript heap out of memory" error indicates that your Node.js process is running out of memory. This can happen if you're trying to process a large amount of data all at once. The streaming approach should help mitigate this by processing data in chunks.

If you continue to encounter memory issues, consider increasing the heap size when starting your Node.js process\

spiral mango
#

Nice addition @whole needle

grand widget
#

Thank you guys @spiral mango and @whole needle for this valuable help
If you guy free and I get your knowledge related to JS and Node JS.

Thank you again.

grand widget
#

Hello @whole needle and @spiral mango
I have try all of you way and I am getting this result

First time it's write header to csv file but second time it add header as you can see in image

If you can help me that why it's happening
and here is my code.

image.png image.png
spiral mango
grand widget
#

new file @spiral mango

spiral mango
#

Well, you'll want to skip writing the header in subsequent files as you're outputting. By the way I went to npm to look at json2csv options and noticed a large warning which reads that the library is abandoned.

grand widget
#

Yes I know but I don't have any other option right now
I was tried everything from last 15 or more days

spiral mango
#

To confirm, you are using json2csv to output the data to csv?

grand widget
#

Yes I am

grand widget
#

Hello @spiral mango and @whole needle
I still not find solution

whole needle