‚̧Numeric Production Mechanism
    Have ideas to improve npm?Join in the discussion! ¬Ľ

    mongo-oplog-reader

    3.2.0¬†‚Äʬ†Public¬†‚Äʬ†Published

    mongo-oplog-reader

    Read MongoDB oplogs from a sharded cluster.

    Build Status

    • Supports multiple processes for high availability
    • Emits event after document has been written to the majority of members of the shard's replica set
    • Resumes from the last emitted event of the replica set (in the case of process crash/stop/restart)
    • Ability to add/remove shards/hosts without restarting

    Notes:

    • The order of the oplog events is not guaranteed to be strictly chronological if there are multiple workers per oplog

    Requirements

    • Redis 3.2+

    Install

    npm i mongo-oplog-reader redis
    

    Usage

    import MongoOplogReader from 'mongo-oplog-reader';
    import redis from 'redis';
     
    const redisClient = redis.createClient();
     
    const connectionStrings = [
      'mongodb://shard0-primary/local',
      'mongodb://shard0-secondary0/local',
      'mongodb://shard0-secondary1/local',
      'mongodb://shard1-primary/local',
      'mongodb://shard1-secondary0/local',
      'mongodb://shard1-secondary1/local'
    ];
     
    const reader = new MongoOplogReader({
      redisClient,
      workersPerOplog: 1 // total # of redundant workers per oplog (respected across all processes)
    });
    reader.filter(oplogDocument => oplogDocument.op !== 'u'); // ignore 'update' operations
    reader.setConnectionStrings(connectionStrings);
    reader.onEvent(data => {
      // return a promise to apply backpressure on the oplog stream to prevent a 
      // build-up of in-memory stream buffering while performing slower async operations
      return somethingAsync(data);
    });
    reader.start();

    License

    MIT

    Install

    npm i mongo-oplog-reader

    DownloadsWeekly Downloads

    2

    Version

    3.2.0

    License

    MIT

    Unpacked Size

    23.8 kB

    Total Files

    7

    Last publish

    Collaborators

    • avatar
    • avatar