Pelias Blacklist Stream
This package provides a configuration-driven approach to removing specific records from an import stream.
It's particularly helpful where algorithmic deduplication fails or for when data is erroneous and needs to be replaced with an alternative.
$ npm install pelias-blacklist-stream
The blacklist stream is intended to be used by pipelines passing objects generated by pelias/model.
You can specify which records are omitted from the build by providing their globally-unique-id (GID).
GIDs can be found in the results served by pelias/api or by calling
getGid() on a Pelias Model object.
const blacklistStream = ;const blacklist ="openaddresses:address:us/tx/libery:377e64dd81884dbe": "1800 Mlk, Liberty, TX, USA""openaddresses:address:us/fl/statewide:bee100ffcc77c699": undefined;const stream = ;
The stream will now remove any documents which match either of the GIDs
The values are optional, you can specify a human-readable comment for debugging.
Using blacklist files specified from Pelias Config
If no arguments are provided when calling
blacklistStream(), it will load your local pelias/config.
If your config contains entries in the
imports.blacklist.files array then each file will be loaded from disk, merged and used as the blacklist.
const blacklistStream = ;const stream = ; // no arguments specified
The relevant parts of the pelias config file, usually located at
Blacklist file format
Blacklist files stored on disk can have any file extension (or none).
Each line of the file should contain one GID and optionally one comment, lines are separated by a '\n' newline character.
An example of a blacklist file without comments:
An example of a blacklist file with debugging comments:
openaddresses:address:us/tx/libery:377e64dd81884dbe # 1800 Mlk, Liberty, TX, USAopenaddresses:address:us/fl/statewide:bee100ffcc77c699
If the line contains a '#' symbol then anything after the '#' will be considered a comment. Using another '#' in your comment string is not supported.
The parser will
String.trim() whitespace but you must take care to provide the correct letter casing.
pelias-blacklist-stream npm module can be found here:
Please fork and pull request against upstream master on a feature branch.
Pretty please; provide unit tests and script fixtures in the
Running Unit Tests
$ npm test
Travis tests every release against all supported Node.js versions.
We rely on semantic-release and Greenkeeper to maintain our module and dependency versions.