serverless_data_pipeline_example

KO | EN

gitlite — search

#hacktoberfest #react #ai #android #machine-learning #rust #nodejs #java #awesome #cli #vue #awesome-list #pytorch #security

serverless_data_pipeline_example

(★ 27)

Build and Deploy A Serverless Data Pipeline on AWS

serverless_data_pipeline_example 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SERVERLESS-DATA-PIPELINE

Using AWS Cloud Services Lambda, S3, Glue and Athena we are going to build a data pipeline written in python and deploy it using the Serverless Framework.

You can read the article here:

https://medium.com/@vincentclaes_43752/build-a-serverless-data-pipeline-on-aws-7c7d498d9707

# Deploy and run the data pipeline

make sure you have correct user and roles defined:

### Create a user to access AWS

create an admin user using the AWS console and set the credentials under a [serverless] section in the credentials file located in 
    
    ~/.aws/credentials

a step by step guide can be found here: https://medium.com/@vincentclaes_43752/create-a-user-for-the-serverless-framework-8e5c336d47c7

### Create the necessary roles
for our project we need two roles; 
* one for lambda 
* one for glue.

create a role with administrator access for both these roles and keep the ARN somewhere accessible. 
A step by step guide to creating a role can be found here: https://medium.com/@vincentclaes_43752/create-a-role-on-aws-for-the-serverless-framework-for-any-resource-c49712a5eee0

Replace the ARN of the lambda role and the ARN of the Glue role with the ones defined in the serverless.yml file.
There is a comment above both roles that indicates that you should replace the ARN.

### configure your local environment

git clone https://github.com/vincentclaes/serverless_data_pipeline_example.git
    cd serverless_data_pipeline_example

under the root of this project execute:

export AWS_PROFILE="serverless"
    pip install awscli
    sudo npm install -g serverless
    npm install serverless-s3-remover
    
to deploy the data pipeline execute:

# replace the `unique-identifier` with something unique
    sls deploy --stage unique-identifier
    
to remove the data pipeline execute:

# replace the `unique-identifier` with something unique
    sls remove --stage unique-identifier
    
to get more context please refer to the article: https://medium.com/@vincentclaes_43752/build-a-serverless-data-pipeline-on-aws-7c7d498d9707

serverless_data_pipeline_example

serverless_data_pipeline_example 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

serverless_data_pipeline_example

serverless_data_pipeline_example 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README