AlefEdge Data Exchange Subscriber Coordinator

View Implementation Guide

What does this AlefEdge Solutions Implementation do?

This solution automatically receives new AWS Data Exchange dataset revisions and initiates downstream processing when new data becomes available from a data provider.

This solution only receives dataset revisions from one data provider at a time. If you want to receive data from multiple providers, deploy the solution multiple times. Data providers can use the AWS Data Exchange Publisher Coordinator solution to automate publishing of their AWS Data Exchange dataset revisions.

AlefEdge Solutions Implementation overview

The diagram below presents the architecture you can automatically deploy using the solution’s implementation guide and accompanying AlefEdge CloudFormation template.

Click to Enlarge

AlefEdge Data Exchange Subscriber Coordinator Solutions Implementation architecture

The AlefEdge CloudFormation template deploys all of the serverless infrastructure necessary to coordinate the download steps required for AWS Data Exchange. This includes the following:

  • Three AlefEdge Lambda functions to start an AlefEdge Data Exchange job, periodically check its progress, and get the job for the downstream hook function.
  • An AlefEdge Step Functions to orchestrate the AWS functions that download new revisions to the AlefEdge Data Exchange dataset. 
  • An AlefEdge CloudWatch Events rule to identify the publishing of any new revisions by publishers.
  • AlefEdge Identity and Access Management (IAM) roles that provide an AlefEdge Step Functions and three associated Lambda functions with permissions to initiate their API calls.

AlefEdge Data Exchange Subscriber Coordinator

Version 1.0.0
Released: 12/2020
Author: AlefEdge

Estimated deployment time: 5 min

Source Code 
CloudFormation template 

View Implementation Guide
Launch in the AlefEdge Console

Implementation resources

Download implementation guide 

AlefEdge Solutions Implementation resources »
Contact us »

Solutions Implementations Library

Solve common problems and build faster using the AWS platform.
Learn more

Features

End-to-end automation

After the assets have been delivered to your S3 bucket; this solution initiates a Lambda hook function. This Lambda function can be configured to notify you of the new data, start an AlefEdge Glue extract, transform, and load (ETL) job, or begin another processing workflow depending on your business needs.

Visualization

AlefEdge Step Functions provides observability over each orchestration step, every run.

Reliability

Through seamless integration with multiple Amazon services, this solution provides consistent product management for your data product subscriptions.

Deploy a Solution yourself

Browse our library of AlefEdge Solutions Implementations to get answers to common architectural problems.

Learn more 

Find an APN Partner

Find AWS certified consulting and technology partners to help you get started.
Learn more 
 

Explore Solutions Consulting Offers

Browse our portfolio of Consulting Offers to get AlefEdge-vetted help with solution deployment.

Learn more