Ratestate is a ratelimiter in the form of a Node.js module that can transmit states of different entities while avoiding transmitting the same state twice, and adhering to a global speed limit.
Let's say you purchased some intelligent lightbulbs and want to set new colors in near-realtime (e.g. based on color detection of camera input), however the central hub receiving the color commands has a rate limiter that only accepts 30 updates per second. Ratestate can help you spread & drip updates amongst the different lightbulbs, without forming queues (by forgetting about superseded colors).
npm install --save ratestate
Here's a little CoffeeScript example
ratestate =interval: 30:# Transmit the state to idcb nullratestatestartratestatesetState 1color: "purple"ratestatesetState 1color: "green"ratestatesetState 1color: "yellow"ratestatesetState 1color: "yellow"ratestatesetState 1color: "yellow"ratestatesetState 1color: "green"ratestatestop
In this example, entity
1 will reach
"green" and probably won't be set to any other intermediate state (color in this case), as we're setting the state much faster than our configured
interval could keep up with.
Behavior and Limitations
Ratestate is similar to Underscore's debounce, but it runs indefintely and assumes you want to update the state of different entities, but for all entities you are globally speed limited. For instance you might want to
- Continously update 20 different
.jsonfiles on S3, but your server/network only allows a few updates per second. The part of the program that sets the updates, should fire & forget, and not concern itself with environmental constraints like that.
- Flush the current status of visitors to disk for caching, but throttle the total throughput as to not wear out your harddisk or cause high load.
- Capture dominant colors from a video feed at 60 frames per second, and push those colors to Philips HUE lamps, but the combined throughput to them is capped by a rate-limiter on the central Bridge, allowing you to only pass through 30 colors per second total.
You can call
setState as much as you'd like, and Ratestate will
- Only transmit at a maximum speed every configured
- Take care of an even spread between the entities
- Not execute
workerif the state has not changed
- Consider the last pushed state for an entity leading, it will not attempt to transmit every state if more states are set than can be transmitted
- Avoid concurrently working on the same entity (last write wins)
By default, Ratestate detects if a state has changed by comparing hashes of set
state objects and it won't consider executing the
worker on entity states that have not changed.
If this built-in serializing & hashing is too heavy for your usecase (your states are huge - your interval low), you can supply your own function that will be executed on the
state object to determine its uniqueness. In the following example we'll supply our own
hashFunc to determine if the state is a candidate for passing to the
megabyte = 1024 * 1024 * 1024status =id : "foo-id"status : "UPLOADING"bytes_received: 2073741824client_agent : "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0"client_ip : "184.108.40.206"uploads :name: "tesla.jpg"results:original:name: "tesla.jpg"resized:name: "tesla-100px.jpg"ratestate =:returnstatestatusstatebytes_received - statebytes_received % megabytestateuploadslengthstateresultslengthjoin "-"ratestatestartratestatesetState "foo-id"statusratestatestop
This would internally be 'hashed' as
UPLOADING-653908770816-1-2, if we detect a change in our system and blindly call
setState for our entity, this only executes the
worker on it if
statushas changed, OR
- We have more than a new megabytes worth of
- The amount of
- The amount of
As that covers all the interesting changes for us, it's more efficient than serializing and hashing an entire object.
finalState is much like
setState (it's called under the hood), but requires a callback, which is called after the
worker successfully finished on it. Additionally, all data of the involved entity are removed from ratestate.
- Track errors, abort after x(?)
- Implement a forceful
start, so that intervals are ignored if we don't have a previous state on the entity yet.
- Fix concurrency test (last write does not win)
- Test concurrency
- Implement a lock per entity to avoid concurrent writes
- Optional callback for
setState. Useful for setting the last state of an entity. Otherwise: not recommended as there's no guarantee your
callbackwill be fired for anything other than the last write.
@_desiredStatesbookkeeping after worker executed on it without error
- Allow to use your own hashing function (currently only full hashing is implemented)
./lib. This is only used so that people can use this node module without a CoffeeScript dependency. If you want to work on the source, please do so in
./src and type:
make build or
I'd be happy to accept issues and pull requests. If you plan on working on something big, please first give a shout.
Run tests via
To single out a test use
make test GREP=foobar
- Updates the
- Saves a release commit with the updated version in Git
- Pushes to GitHub
- Publishes to npmjs.org
This project received invaluable contributions from:
Like this project? Consider a donation. You'd be surprised how rewarding it is for me see someone spend actual money on these efforts, even if just $1.