Wondering what’s next for npm?Check out our public roadmap! »

    @naturalcycles/db-lib
    TypeScript icon, indicating that this package has built-in type declarations

    8.5.11 • Public • Published

    @naturalcycles/db-lib

    Lowest Common Denominator API to supported Databases

    npm code style: prettier loc Actions

    Defines 3 things:

    • CommonDB interface
    • CommonDao class
    • DBQuery class

    CommonDB serves as a Lowest Commond Denominator between different DB implementations (see further). So you can use same syntax, e.g getById<DBM>(id: string): Promise<DBM | undefined> across different DBs.

    DBQuery allows to use the same query syntax across different DBs! E.g:

    const q = DBQuery.create('table1')
      .filterEq('type', 'cat')
      .filter('updated', '>', '2019-01-17')
      .order('name', true)
    
    await db.runQuery(q)

    So, you can run it against Datastore, Firestore, Redis, MongoDB, Airtable, etc. Different DBs, same syntax!

    You can swap DB implementations without changing your application code. Migrate Datastore to Firestore? Easy.

    You can test your code against InMemoryDB (that implements full CommonDB interface, even with querying, streaming, etc). So, your unit tests can use exactly same querying syntax, or even exactly same services, DAOs. Just swap real DB with InMemoryDB in your setupJest.ts (for example).

    Supported databases

    • [x] InMemoryDB (with optional file persistence, Redis-like; implemented in this package)
    • [x] datastore-lib (GCP Datastore, or Firestore in Datastore mode)
    • [x] firestore-lib (Firestore in Native mode)
    • [x] mysql-lib (MySQL)
    • [x] redis-lib (Redis)
    • [x] mongo-lib (MongoDB)
    • [x] airtable-lib (Airtable)
    • [x] HttpDB (CommonDB exposed via REST API, implemented in backend-lib)
    • [ ] TODO: Google Spreadsheet DB

    Features

    • CommonDB, CommonDao, DBQuery
    • Streaming (Node.js streams with backpressure)
    • DBM / BM, validation, conversion (Joi-powered)
    • Conventions
      • String ids
      • created, updated (unix timestamps)
      • Dates as ISO strings, e.g 2019-06-21
      • Timestamps as unixtimestamps (seconds, not milliseconds; UTC)
      • Complex objects as JSON serialized to string (DBM), converted to object (BM)

    Concept

    CommonDB is a low-level API (no high-level sugar-syntax). CommonDao is the opposite - a high-level API (with convenience methods), built on top of CommonDB.

    Concerns of CommonDB:

    • Access to DB (all tables): CRUD (create, read, update, delete)
    • Batch methods (cause they can be more optimal if implemented "natively")
    • Querying
    • Streaming

    Concerns of CommonDao:

    • Access to one DB Table ("kind")
    • Transformation between DBM and BM, validation/conversion
    • Auto-generating id, created, updated fields
    • Anonymization hook to be able to plug your implementation (privacy by design)

    CommonDB API

    ping

    ping(): Promise<void>

    Call this to check that DB connection, credentials, configuration is working. Should throw an error if any of above is invalid.

    getByIds

    getByIds<DBM>(table: string, ids: string[]): Promise<DBM[]>

    await db.getByIds('table1', ['id1, 'id2'])
    // [ { id: 'id1', ... }, { id: 'id2', ... } ]

    Should return items in the same order as ids in the input.

    Only returns items that are found, does not return undefined (absent) items.

    runQuery

    runQuery<DBM>(q: DBQuery<DBM>): Promise<RunQueryResult<DBM>>

    const q = DBQuery.create('table1').filterEq('type', 'cat').order('name', true) // desc
    
    await db.runQuery(q)
    // { records: [ { ... }, { ... }, ... ] }
    runQueryCount

    runQueryCount(q: DBQuery): Promise<number>

    await db.runQuery(DBQuery.create('table1'))
    // 5
    streamQuery

    streamQuery<DBM>(q: DBQuery<DBM>): ReadableTyped<DBM>

    Returns ReadableTyped (typed wrapper of Node.js Readable).

    Streams in Node.js support back-pressure by default (if piped properly by the consumer).

    const q = DBQuery.create('table1') // "return all items" query
    
    await _pipeline([
      db.streamQuery(q),
      writableForEach(item => {
        console.log(item)
      }),
    ])
    
    // { item1 }
    // { item2 }
    // ...
    saveBatch

    saveBatch<DBM>(table: string, dbms: DBM[]): Promise<void>

    Since CommonDB is a "minimal API", there's no save method for a single item, only for multiple. Pass an array with single item to save just one item.

    const items = [
      { item1 },
      { item2 },
    ]
    
    await db.saveBatch('table1', items) // returns void
    await db.runQuery(DBQuery.create('table1') // "get all" query
    // [ { item1 }, { item2 } ]
    deleteByIds

    deleteByIds(table: string, ids: string[]): Promise<number>

    Returns number of deleted items (not all CommonDB implementations support that).

    await db.deleteByIds('table1', ['id1', 'id2'])
    // 2
    deleteByQuery

    deleteByQuery(q: DBQuery): Promise<number>

    Returns number of deleted items.

    await db.deleteByQuery(DBQuery.create('table1'))
    // 2
    getTables

    getTables(): Promise<string[]>

    await db.getTables()
    // [ 'table1', 'table2' ]
    getTableSchema

    getTableSchema(table: string): Promise<CommonSchema>

    await db.getTableSchema('table1')
    // todo: describe CommonSchema example
    createTable

    createTable(schema: CommonSchema): Promise<void>

    Applicable to Relational DBs, like MySQL. Will invoke smth like create table Table1 ... ;. Takes a CommonSchema as an argument.

    DBQuery

    Object that defines "DB Query".

    // Simplest query - "get all" query
    DBQuery.create('table1')
    
    // where type = "cat"
    DBQuery.create('table1').filter('type', '==', 'cat')
    
    // OR
    DBQuery.create('table1').filterEq('type', 'cat')
    
    // Where updated > 2019-01-17
    DBQuery.create('table1').filter('updated', '>', '2019-01-17')
    
    // order by 'name'
    DBQuery.create('table1').filter('updated', '>', '2019-01-17').order('name')
    
    // order by 'name' in descending order
    DBQuery.create('table1').filter('updated', '>', '2019-01-17').order('name', true)

    Features:

    .filter(key: string, operator: Operator, value: any)
    .filter('updatedDate', '>', '2019-01-17')
    .filterEq(key: string, value: any)
    .filterEq('updated', true)
    .order(key: string, descending: boolean = false)
    .order('updated') // asc
    .order('updated', true) // desc
    .limit(lim: number)
    .limit(1000)
    .limit(0) // no limit
    .select(fields: string[])

    Allows "projection queries" - queries that return subset of fields. Like select a,b,c from Table in SQL, as opposed to select * from Table.

    Passing empty array will actually return an array of empty objects (documented edge case).

    .select([]) // returns [ {}, {}, {} ]
    .select(['id']) //=> [ { id: 'id1' }, { id: 'id2' }, ... ]

    DEBUG namespaces

    • nc:db-lib:commondao
    • nc:db-lib:inmemorydb

    Exports

    • / root
    • /adapter/file
    • /adapter/cachedb
    • /testing
      • dbTest
      • daoTest
      • Test models, utils, etc
    • /validation
      • Joi validation schemas for DBQuery, CommonDBOptions, CommonSchema, etc

    Packaging

    • engines.node >= LTS
    • main: dist/index.js: commonjs, es2019
    • types: dist/index.d.ts: typescript types
    • /src folder with source *.ts files included

    Install

    npm i @naturalcycles/db-lib

    DownloadsWeekly Downloads

    203

    Version

    8.5.11

    License

    MIT

    Unpacked Size

    475 kB

    Total Files

    155

    Last publish

    Collaborators

    • avatar
    • avatar
    • avatar