@uizaio/node-kafka-client

    0.0.6 • Public • Published

    node-kafka-client

    Build Status dependencies Status devDependencies Status

    A wrapper library of node-rdkafka with extended features

    Install

    Important: We are using node-rdkafka 2.7.4 which is not supporting Node v12 or later. Let's use Node version v8 > v10

    npm install @uiza/node-kafka-client

    Usage

    Consumer

    const { Consumer } = require("@uiza/node-kafka-client");
     
    const consumer = new Consumer({
      // Required options
      name: "test-consumer-name",
      groupId: "test-group-id",
      host: "localhost:9092",
      topic: "test-topic",
     
      // Optional options
      connectTimeout: 5000,
      mode: "non-flowing",
      intervalFetchMessage: 10, // Only affect for non-flowing mode
      numMsgFetchPerTime: 1, // Only affect for non-flowing mode
      logger: null,
    });
     
    consumer.listen(async (data) => {
      console.log("Listen data:", data);
    });

    The consumer inherit all node-rdkafka configuration options for you to set or override. Using snippet config below and reference original node-rdkafka lib for more detail of configuration.

    {
      ...
      rdKafkaConfig: {
        debug: "all",
      },
      rdKafkaTopicConfig: {
        "auto.offset.reset": "latest",
      },
    }
    consumer.on("event.log", console.log);

    Flowing mode:

    • Pros
      • Concurrent handler processing message.
    • Cons
      • Might cause OOM (out-of-memory) when start the consumer with the lag offset too much
      • Need to handler duplicated messages.
      • Each consumer will block a thread of libuv.
    • Suitable for:
      • Small processing time handler.
      • The result of processing messages is independent.

    Non-flowing:

    • Pros:
      • Handle one-by-one messages. The next message just consumed after the previous committed
      • It's not block the thread of libuv.
      • Fully control the interval and number of message to fetch
    • Cons:
      • Non-concurrency consume messages, higher latency compare to flowing mode.
    • Suitable for:
      • High demand of consistency handling messages or the result of processing a message depends on the previous one
      • Running multiple consumers in one application
      • Long-time-taking handler

    Producer

    [will be updated]

    Benchmark

    [will be updated]

    Contributing

    Feel free to dive in! Open an issue or submit PRs.

    Contributors

    License

    MIT © Uiza

    Install

    npm i @uizaio/node-kafka-client

    DownloadsWeekly Downloads

    14

    Version

    0.0.6

    License

    MIT

    Unpacked Size

    32 kB

    Total Files

    10

    Last publish

    Collaborators

    • uizadeveloper