Concatenate data in a stream using a transform
npm install @voliware/node-concat-transform
What is it?
In Node streams you may get data in small chunks. This is rare from a file, unless set explicitly, but will happen in a throttled network system. Most often you cannot operate on pieces of data beause you do not have the entire chunk of data.
node-concat-transform is a Node Transform that you can use with the
pipe() method to concatenate (link or bundle) data into one bigger chunk. Then, you can operate on that larger chunk if needed, and continue sending it downstream.
Why do I need it?
If you are using a readable Node stream that receives chunks and you are not 100% sure you will receive all the data at once, you will probbably need to bundle it first. You can easily accomplish this with
How do I use it?
Once you have a readable Node stream, such as a file stream, TCP stream, or any other type of
Readable stream, you simply create a
ConcatTransform and pass it into the
pipe() method before you intend to do something with the stream data.
Suppose you have a readable stream receiving chunks of data that is only 1 character in length. You can create a
ConcatTransform to bundle the chunks into one big peice of data that can then be operated on. This would be essential for example when zipping stream data.
let readable = ;let concat_transform = ;let writable = ;// simulate a stream reading data 1 character/number at a timeforlet i = 0; i < datalength; i++readable;// stream endsreadable;// make use of the stream datareadable;
If you are receiving data that has a header, or a start delimiter and/or end delimiter, you can also set options for your
ConcatTransform that will bundle the data based on these options. This is common in network based streams where an EOF set of characters, such as
/r/n, indicate the end of a datagram, but you may actually get peices of the next or previous datagram in the same chunk. If your chunks have headers, than you will need to pass a custom function that parses the header and returns the total number of bytes expected in a fully formed chunk.
let concat_transform =// strings or Buffers are accepteddelimiter:start: "START"end: "END"// to parse a header, write a custom function// that will return the expected full chunk lengthheader: header_parsing_function;