pipely

Build, deploy, and visualize pipeline definitions for AWS Data Pipeline

“AWS Data Pipeline is a web service that you can use to automate the movement and transformation of data. With AWS Data Pipeline, you can define data-driven workflows, so that tasks can be dependent on the successful completion of previous tasks.”

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html

Install

(First install GraphViz if it is not already installed.)

Into Gemfile from rubygems.org:

gem 'pipely'

Into environment gems from rubygems.org:

gem install pipely

Usage

Rake Tasks

Coming soon.

rake definition        # Graphs the full pipeline definition using Graphviz
rake deploy            # Deploy pipeline
rake graph             # Graphs the full pipeline definition using Graphviz
rake upload_steps      # Upload Data Pipeline steps to S3

In order to run these tasks, you must have an aws-sdk credentials file.
This can be created by running aws cli configure, as described in the aws-sdk docs.

Command-line Interface

(If you used the Gemfile install, prefix the below commands with bundle exec.)

To render a JSON pipeline definition as a PNG graph visualization:

pipely definition.json

To specify the output path for PNG files:

pipely -o path/to/graph/pngs definition.json