Skip to content

[BUG] Format concats lines when reading from multiple streams to one file #934

@krystian-wolarek

Description

@krystian-wolarek

Describe the bug
The issue occurs when parsing from multiple streams and writing to a single stream.
The first line from the second stream is appended to some line from the first stream but should be put in a new line.
Sometimes it's appended in a new line but mostly not.
When I use csv-stringify to format stream then it works well, everything is in a new line
I read streams simultaneously then problem occurs, when I do it one by one then it works

Parsing or Formatting?

  • Formatting

To Reproduce
Full code in the sandbox. Code is in src/my-processor.js
https://codesandbox.io/p/sandbox/streams-merge-ldw56f

To demonstrate the issue I've created in /temp-files two csv-s 'file1.csv' and file2.csv.
Formatting is done in two methods transformStreamFastCSV- problematic and transformStreamAndFormatStringify - working.
Results are merged into two files temp-files/merge-fast-csv.csv and temp-files/merge-stringify.csv.

Short version (more like pseudocode):

const filesToDownload = streams.length;
const writer = fs.createWriteStream('...');

formatToCSVStream() {
    const options = isFirstResponse
        ? { headers: true, includeEndRowDelimiter: true }
        : { headers: true, writeHeaders: false, includeEndRowDelimiter: true };
    return csv.format(options);
};

fastCSVParse(index) {
    return csv
        .parse({ headers: true })
        .transform((data) => {...data, file: 'file-'+index})
        .on("error", (error) => {
            console.error(error);
        });
}
let streamsEndCounter = 0;
streams.forEach((stream, index) => {
    stream
        .pipe(fastCSVParse(index))
        .pipe(formatToCSVStream())
        .on("end", () => {
            streamsEndCounter++;

            if (streamsEndCounter === filesToDownload) {
              writer.end();
            }
        })
        .pipe(writer, { end: false });

    isFirstResponse = false;
});

Expected behavior
Fast csv creates 1 row that has no newline and record 100,Riannon... is mostly appended to some line:
199,Britte,Stacy,Britte.Stacy@yopmail.com,dummyBritte.Stacy@gmail.com,worker,file-0100,Riannon,Gino,Riannon.Gino@yopmail.com,dummyRiannon.Gino@gmail.com,doctor,file-1
but should be placed in new line like result from csv-stringify
199,Britte,Stacy,Britte.Stacy@yopmail.com,dummyBritte.Stacy@gmail.com,worker,file-0 100,Riannon,Gino,Riannon.Gino@yopmail.com,dummyRiannon.Gino@gmail.com,doctor,file-1

Screenshots
Zrzut ekranu 2024-04-5 o 12 55 15

Desktop (please complete the following information):

  • Node Version 18.19.0 and in codesandbox 20.something

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions