S4QS-RS reads S3 object creation events from SQS queues and copies data from S3 to Redshift using COPY with S3 manifests (see Redshift's COPY documentation for more information.)
See the S3 documentation for more information about setting up the object creation events. Note that if you publish the events to SNS and then subscribe your SQS queue to the SNS topic, you must set the "Raw Message Delivery" subscription attribute to "True".
Data can be either copied into monolithic tables, or time series tables with configurable rotation periods and retention, and multiple rolling views with different periods. View creation is done so that adding or removing columns from time series tables doesn't break the view: missing columns are filled in with
NULL as [missing_col_name] in the view queries.
More details about how S4QS-RS works in the configuration section.
npm install -g s4qs-rs
S4QS-RS is configurable with JSON files located in either the
./config subdirectory of the current working directory, or at
./config/default.json is always loaded and used for default settings, and
$NODE_ENV is used to determine environment-specific configuration files: For example
NODE_ENV=production would cause
./config/production.json to be loaded and merged with
NODE_ENV-specific configuration files will always override the default configuration. If
./config/default.json is, for example
SQS.params.QueueUrl would be overridden.
S4QS-RS defaults to
NODE_ENV is omitted.
AWS credentials are loaded by the Node AWS SDK. See the SDK's documentation for more information. S4QS-RS needs IAM permissions to perform
Configuration structure as a JSON file
S4QS-RS uses the debug NPM package for outputting debug messages to stdout and error messages to stderr. To see any error messages, you must run S4QS-RS with the env
DEBUG=s4qs-rs:*:error. To see debug messages, run with
DEBUG=s4qs-rs:* See debug's documentation for more information.
You can map environment variables to configuration parameters by creating a file with the name
./config/custom-environment-variables.json. For example
S3Copier.Redshift.connStr. Values set with environment variables always override configuration file values.
After setting up your configuration, you can run S4QS-RS with
DEBUG=*:error s4qs-rs or
DEBUG=sq4s-rs:* s4qs-rs to see debugging information.
SIGINT, and aborts in-flight transactions when terminated.
require('s4qs-rs'), you can use the different components of S4QS-RS separately from the command line tool. See the source code for the documentation, plus have a look at app.js.