TAR streaming utils
A set of utils for TAR streaming and processing, there is only tar-stream-chunker tool for now.
tar-stream-chunker can help to handle data stream of unknown size without hitting a disk. As long as creating TAR archive as a stream needs to know file size in advance, that is why the idea of the chunker is to split input stream onto chunks with known size.
$ npm install -g @rough/tar-stream-utils
$ pg_dump <options> | tar-stream-chunker --file-name db-dump.sql --chunk-size 100000000 | tarsnap -cvf db-dump.sql.$(date +daily.%Y%m%d.%H%M%S) @-
And resulting TAR archive could look like this:
$ cat file | tar-stream-chunker --file-name file --chunk-size 1000000 > archive.tar Chunks: 1, size: 1048576 B = 1 MiB. Chunks: 2, size: 2064384 B = 2 MiB. Chunks: 3, size: 3096576 B = 3 MiB. Chunks: 4, size: 4112384 B = 4 MiB. Chunks: 5, size: 5144576 B = 5 MiB. Chunks: 6, size: 5398528 B = 5 MiB. $ tar -tf archive.tar file.chunk00001 file.chunk00002 file.chunk00003 file.chunk00004 file.chunk00005 file.chunk00006 README $ tar -xf archive.tar -O README In order to assemble the original file: $ cat file.chunk* > file