s3-csv-to-json

    0.0.8 • Public • Published

    s3-csv-to-json

    Converts AWS S3 files from CSV to JSON lines via stream with support to gzip for both input and output. Ready to be used as a Node.js module, as a Lambda or via CLI.

    Currently it downloads the input file from S3 and upload the result back to it. A support to have one of the both ends pointing to a local file will be worked on.

    Goals

    • Work with files stored on AWS S3
    • Handle large files (stream: Input from S3 -> Convert -> Upload to S3)
    • Support to gzip for both input and output
    • AWS Lambda ready

    Installation

    Node.js Module

        npm install s3-csv-to-json
    

    Lambda

    Clone the repo and install its dependencies without optionals, as aws-sdk is already available on Lambda functions.

        npm install --no-optional
    

    After installing all dependencies, package it as a zip (e.g. zip -r s3-csv-to-json.zip s3-csv-to-json/*).

    Lambda config:

    • Code entry type: Upload the zip package.
    • Runtime: Node.js 8.10+
    • Handler: src/aws-lambda.handler

    Usage

    Using it as a module, a Lambda or via CLI, it just expects an input and a output path from AWS S3.

    {
        "input": "https://s3.amazonaws.com/bucket-name/path/to/input.csv.gz",
        "output": "s3://bucket-name/path/to/output.jsonl.gz"
    }

    As shown above, S3 protocol is accepted as well for both cases (e.g. s3://bucket-name/path/to/input.csv).

    Gzip is supported for both input and output. The .gz extension is all that is needed to determine when the input must be decompressed and if the output should be compressed. If there is no .gz extension it will be handled as a text file (UTF-8).

    Node.js Module

    const s3CsvToJson = require('s3-csv-to-json.js');
     
    // Assuming that the following is within an async function
    const response = await s3CsvToJson({
        input: "https://s3.amazonaws.com/bucket-name/path/to/input.csv.gz",
        output: "s3://bucket-name/path/to/output.jsonl.gz"
    });

    Lambda

    Just post a JSON as already shown above:

    {
        "input": "https://s3.amazonaws.com/bucket-name/path/to/input.csv.gz",
        "output": "s3://bucket-name/path/to/output.jsonl.gz"
    }

    CLI

    Clone the repo and execute it using the module MagiCLI via npx:

    npx magicli ./path-to-s3-csv-to-json-module --input="s3://bucket-name/path/to/input.csv.gz" --output="s3://bucket-name/path/to/output.jsonl.gz"

    Example

    CSV Input:

    ID,NAME
    1,John Doe
    2,Jane Doe
    

    JSON lines output:

    {"ID":"1","NAME":"John Doe"}
    {"ID":"2","NAME":"Jane Doe"}
    

    Install

    npm i s3-csv-to-json

    DownloadsWeekly Downloads

    3

    Version

    0.0.8

    License

    MIT

    Unpacked Size

    5.24 kB

    Total Files

    4

    Last publish

    Collaborators

    • avatar