A generic pluggable query-engine system (that supports indexes) for levelup/leveldb databases.
Using this architecture you can query your levelup database using your own query langauge with full index support.
Query languages currently supported
Here is a list of the query plugins that are currently written and work with
- jsonquery-engine - MongoDB query language implemented for levelup WITH indexing!
- fulltext-engine - Query your levelup/leveldb engine using full text search phrases with full text indexing.
Write YOUR own query language plugins to run on top of levelup, and get the full benefit if indexing for highly-performant lookups!
Why is this needed?
Because while it is relatively easy to programatically write your own stream filters to query levelup, making efficient use of indexes (particularly when multiple indexes may be available) is painful.
This module allows you to write your own index implementations and query languages and have the query engines work out what the best indexes are to use.
Query languages will generally have a declarative syntax that specifies what fields are to be retrieved, and this can be used to identify indexes, and then most efficiently retrieve your data.
Install via npm:
$ npm install level-queryengine
Example Usage with the jsonquery-engine:
var levelQuery =jsonqqueryEngine =pairs =levelup =db = ;dbquery;// index all the properties in pairsdb;// alternatively you could just index the properties you want:// db.ensureIndex('num');// db.ensureIndex('tags');db;
Example Usage with the fulltext-engine:
var levelQuery =fulltextEngine =levelup =db = ;dbquery;// index the properties you want (the 'doc' property on objects in this case):db;db;
Example Usage with the path-engine:
var levelQuery =pathEngine =levelup =db = ;dbquery;// index the properties you want:db;db;
Returns a function which you call on your levelup instance to add query and indexing capabilities. Eg:
var levelQuery =levelup =db = ;
Sets the query engine to use for subsequent
queryEngine plugin is simply an object that returns the following functions:
// examplequery: querymatch: matchfnplans:'property': propertyPlanFn'pairs': pairsPlanFn;
query(function) - will be called by
db#query, to execute the query. It must return an index stream (of indexes - see subindex) if indexes can be used OR
nullif no indexes can be used and a full levelup database "table" scan will commence instead.
match(objectToMatch, query)(function) - a function that will be used as a final filter on the object stream generated. This function must return
querypassed to it. If an index creates some false positives then this function is responsible for doing the FINAL filter to ensure that
objectToMatchmatches the query. This is also used in the event of table scan to do the bulk of the filtering.
plans- a map of index strategy types to "plan" functions which take a subquery of the query execution, and return an indexStream if indexes can be used for the subquery, or
nullif no index can be used (which will cause a full levelup database "table" scan. Plan functions are used to divide responsibility between generating complicated compound queries (which may need to AND or OR the resultantant stream of index keys), to a simple subquery which works out what indexes to use. See the implementations of jsonquery-engine and path-engine for more details. By exposing the internals of the query plans it allows for people to write their own implementations of the query engines to run over other indexing strategies.
query using the current query engine, and returns a stream of values.
This project is under active development. Here's a list of things I'm planning to add:
- add time taken to the statistics reporting
- make the
indexHitsreporting more accurate for compound queries that depend on ANDs and ORs of streams.
- allow for multiple query engines to be used at the same time
- Do a SQL based query engine reference implementation (just for kicks!).
- Add joins to some of the reference implementations.