pull-lend-stream

    1.2.2 • Public • Published

    Build Status

    pull-lend-stream

    A refinement of the paramap module for dynamic, unbounded, and fault-tolerant parallel processing.

    Continuously lends values to concurrent sub-streams. Transparent re-lending in case of failure of sub-streams.

    Each sub-stream continuously borrows (different) values from a lender and returns results back after processing.

    • Supports multiple concurrent sub-streams.
    • Produces results in the order in which the lender reads the values.
    • If a sub-stream closes before producing all its results, the original source values for missing results are lent transparently to other sub-streams.
    • Each sub-stream obtains values as fast as they read them.

    Useful for delegating processing to a dynamic number of concurrent, cooperative, connected, but unreliable clients.

    Quick Example

    var pull = require('pull-stream')
    var lendStream = require('pull-lend-stream')
    
    var lender = lendStream()
    
    function minus (x, cb) {
      setTimeout(function () {
        cb(null, -x)
      }, 201)
    }
    
    // Twice faster
    function addTen (x, cb) {
      setTimeout(function () {
        cb(null, 10 + x)
      }, 100)
    }
    
    function borrower (mapper) {
      return function (err, stream) {
        if (err) return console.log(err.message)
    
        pull(
          stream,
          pull.asyncMap(mapper),
          stream
        )
      }
    }
    
    // Prints -0,11,12,-3,14,15,-6,17,18,-9,20
    pull(
      pull.count(10),
      lender,
      pull.collect(function (err, results) {
        if (err) throw err
        console.log(results)
      })
    )
    
    lender.lendStream(borrower(minus))
    lender.lendStream(borrower(addTen))
    

    Signature

    The following signature follows the js module signature syntax and conventions. All callbacks ('cb') have the '(err, value)' signature.

    lendStream: () =>
    lender: {
        sink: (read: (abort, cb)),
        lendStream: (borrower: (
            err,
            subStream: {
                source: (abort, cb),
                sink: (read: (abort, resultCb)),
                close: (?err)
            })),
        source: (abort, resultCb)
    }
    

    Properties

    Italic names refer to the function signature above.

    1. If no subStream is read then read is never called.
    2. Multiple subStreams can be created by calling lendStream multiple times.
    3. Once lendStream has been called,
      3.1 the borrower will eventually be called either with a subStream or an err;
      3.2 if there is no err and values are read by calling subStream.source, subStream.source will eventually abort.
    4. lender.source closes after lender.sink has received an abort and all subStreams have closed.
    5. lender.source produces results in the order in which the values were read by the lender.sink.
    6. If a subStream ends before successfully sinking results computed for all values read, then the values for missing results are transparently migrated to other subStreams.
    7. Unfair: if a subStream reads values faster than other subStreams, it will obtain more values.
    8. When a borrower is called, err is truthy if and only if:
      8.1 lender.sink has not been called yet (lender is not connected to an upstream source);
      8.2 lender.source was aborted;
      8.3 all available values have been borrowed and all results have been sourced.
    9. subStream.close(?err) ends the corresponding subStream with the error err if present, or with true otherwise.

    Expectations on the sub-streams

    1. Sub-streams should correctly close when subStream.source aborts and the event should propagate to their sink. Otherwise it will indefinitely prevent the lender from closing (Prop. 4).

    Debugging

    You can obtain a trace of the internal events of the module by activating the logging using the DEBUG=pull-lend-stream environment variable (see debug).

    You can also obtain the internal state of the module at a specific point in time by calling the _state() method. It returns an object with the following properties:

        return {
          connected: Boolean, // The lender is connecter to an upstream source
          ended: Boolean, // Upstream is closed
          closed: Boolean, // lender is closed
          openedNb: Number, // Number of sub-streams opened
          lendState: Object // State of the internal pull-lend module
        }
    
    

    The output of the _state() method should not be relied on for regular operations because it depends on the implementation of the module and may change in the future.

    Random Testing

    A tool for random testing is available under ./test/random. The methodology used is explained in the README.

    Install

    npm i pull-lend-stream

    DownloadsWeekly Downloads

    19

    Version

    1.2.2

    License

    MIT

    Unpacked Size

    46.2 kB

    Total Files

    13

    Last publish

    Collaborators

    • elavoie