-
Notifications
You must be signed in to change notification settings - Fork 216
Description
Describe the bug
The issue occurs when parsing from multiple streams and writing to a single stream.
The first line from the second stream is appended to some line from the first stream but should be put in a new line.
Sometimes it's appended in a new line but mostly not.
When I use csv-stringify to format stream then it works well, everything is in a new line
I read streams simultaneously then problem occurs, when I do it one by one then it works
Parsing or Formatting?
- Formatting
To Reproduce
Full code in the sandbox. Code is in src/my-processor.js
https://codesandbox.io/p/sandbox/streams-merge-ldw56f
To demonstrate the issue I've created in /temp-files two csv-s 'file1.csv' and file2.csv.
Formatting is done in two methods transformStreamFastCSV- problematic and transformStreamAndFormatStringify - working.
Results are merged into two files temp-files/merge-fast-csv.csv and temp-files/merge-stringify.csv.
Short version (more like pseudocode):
const filesToDownload = streams.length;
const writer = fs.createWriteStream('...');
formatToCSVStream() {
const options = isFirstResponse
? { headers: true, includeEndRowDelimiter: true }
: { headers: true, writeHeaders: false, includeEndRowDelimiter: true };
return csv.format(options);
};
fastCSVParse(index) {
return csv
.parse({ headers: true })
.transform((data) => {...data, file: 'file-'+index})
.on("error", (error) => {
console.error(error);
});
}
let streamsEndCounter = 0;
streams.forEach((stream, index) => {
stream
.pipe(fastCSVParse(index))
.pipe(formatToCSVStream())
.on("end", () => {
streamsEndCounter++;
if (streamsEndCounter === filesToDownload) {
writer.end();
}
})
.pipe(writer, { end: false });
isFirstResponse = false;
});
Expected behavior
Fast csv creates 1 row that has no newline and record 100,Riannon... is mostly appended to some line:
199,Britte,Stacy,Britte.Stacy@yopmail.com,dummyBritte.Stacy@gmail.com,worker,file-0100,Riannon,Gino,Riannon.Gino@yopmail.com,dummyRiannon.Gino@gmail.com,doctor,file-1
but should be placed in new line like result from csv-stringify
199,Britte,Stacy,Britte.Stacy@yopmail.com,dummyBritte.Stacy@gmail.com,worker,file-0 100,Riannon,Gino,Riannon.Gino@yopmail.com,dummyRiannon.Gino@gmail.com,doctor,file-1
Desktop (please complete the following information):
- Node Version 18.19.0 and in codesandbox 20.something
