Build Lambda Functions using Python, Lambda Triggers, Deploy using layers and Docker, Validate using Glue and Athena
What you will learn
Setup required tools on Windows to develop the code for ETL Data Pipelines using Python and AWS Services
Setup Project or Development Environment to develop applications using Python and AWS Services
Getting Started with AWS by creating account in AWS and also configure AWS CLI as well as Review Data Sets used for the project
Develop Core Logic to Ingest Data from source to AWS s3 using Python boto3
Getting Started with AWS Lambda Functions using Python 3 Run-time Environment
Refactor the application, build zip file to deploy as AWS Lambda Function
Create AWS Lambda Function using Zip file and Validate
Troubleshoot issues related to AWS Lambda Functions using AWS Cloudwatch
Build custom docker image for the application and push to AWS ECR
Create AWS Lambda Function using the custom docker image in AWS ECR
Develop Applications using AWS Lambda Functions by adding Python Modules as Layers
Description
Do you want to learn AWS Lambda Functions by building an end-to-end data pipeline using Python as Programming Language and other key AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc? Here is one course using which you will learn AWS Lambda Functions by implementing an end-to-end pipeline by using all the services mentioned.
As part of this course, you will learn how to develop and deploy lambda functions using the zip files, custom docker images as well as layers. Also, you will understand how to trigger lambda functions from Eventsbridge as well as Step Functions.
- Set up required tools on Windows to develop the code for ETL Data Pipelines using Python and AWS Services. You will take care of setting up Ubuntu using wsl, Docker Desktop, and Visual Studio Code along with Remote Development Extension Kit so that you can develop Python-based applications using AWS Services.
- Setup Project or Development Environment to develop applications using Python and AWS Services on Windows and Mac.
- Getting Started with AWS by creating an account in AWS and also configuring AWS CLI as well as Review Data Sets used for the project
- Develop Core Logic to Ingest Data from source to AWS s3 using Python boto3. The application will be built using Boto3 to interact with AWS Services, Pandas for date arithmetic, and requests to get the files from the source via REST API.
- Getting Started with AWS Lambda Functions using Python 3.9 Run-time Environment
- Refactor the application, and build a zip file to deploy as AWS Lambda Function. The application logic includes capturing bookmarks as well as Job Run details in Dynamodb. You will also get an overview of Dynamodb and how to interact with Dynamodb to manage Bookmark as well as Job Run details.
- Create AWS Lambda Function using a Zip file, deploy using AWS Console and Validate.
- Troubleshoot issues related to AWS Lambda Functions using AWS Cloudwatch
- Build a custom docker image for the application and push it to AWS ECR
- Create AWS Lambda Function using the custom docker image in AWS ECR and then validate.
- Get an understanding of AWS s3 Event Notifications or s3-based triggers on Lambda Function.
- Develop another Python application to transform the data and also write the data in the form of Parquet to s3. The application will be built using Pandas by converting 10,000 records at a time to Parquet.
- Build orchestrated pipeline using AWS s3 Event Notifications between the two Lambda Functions.
- Schedule the first lambda function using AWS EventsBridge and then validate.
- Finally, create an AWS Glue Catalog table on the s3 location which has parquet files, and validate by running SQL Queries using AWS Athena.
- After going through the complete life cycle of Deploying and Scheduling Lambda Function and also validating the data by using Glue Catalog and AWS Athena, you will also understand how to use Layers for Lambda Function.
Here are the key takeaways from this training:
- Develop Python Applications and Deploy as Lambda Functions by using a Zip-based bundle as well as a custom docker image.
- Monitor and troubleshoot the issues by going through Cloudwatch logs.
- The entire application code used for the demo along with the notebook used to come up with core logic.
- Ability to build solutions using multiple AWS Services such as Boto3, S3, Dynamodb, ECR, Cloudwatch, Glue Catalog, Athena, etc
English
language
Content
Introduction to Mastering AWS Lambda Functions for Data Engineers
Introduction to Mastering AWS Lambda Functions for Data Engineers
Resources used for Mastering AWS Lambda Functions for Data Engineers
Getting Started on Windows with Required Tools
Introduction to Getting Started on Windows with Required Tools
Overview of Powershell on Windows 10 or Windows 11
Setup Ubuntu VM on Windows 10 or 11 using wsl
Setup Ubuntu VM on Windows 10 or 11 using wsl
Setup Docker Desktop on Windows
Validate Docker on Windows using Command Line leveraging Power Shell
Review Docker Desktop Resource Configurations
Install Visual Studio Code on Windows
Install Remote Development Extension Kit for Visual Studio Code
Install Python 3.9 and Distutils on Windows using wsl Ubuntu
Review Tools Installed for Application Development using Python and AWS Service
Install Required Tools on Windows
Overview of Powershell on Windows 10 or Windows 11
Setup Ubuntu VM on Windows 10 or 11 using wsl
Setup Ubuntu VM on Windows 10 or 11 using wsl
Setup Docker Desktop on Windows
Validate Docker on Windows using Command Line leveraging Power Shell
Review Docker Desktop Resource Configurations
Install Visual Studio Code on Windows
Install Remote Development Extension Kit for Visual Studio Code
Setup Development Environment to build Data Pipelines using AWS
Setup Project Folder using Visual Studio Code
Ensure Python 3.9 for the Project
Create Python Virtual Environment using Python 3
Install Required Dependencies for the Project using AWS Services
Ensure AWS CLI to interact with AWS Services using AWS CLI Commands
Recommendation to use Personal AWS Account for the course
Getting Started with AWS and Review Data Sets
Setup and Login into AWS Account
Setup AWS IAM User with Administrator Permissions
Configure and Validate AWS CLI
Configure AWS CLI with custom profile as default
Recap of Date Arithmetic using Python
Validate Python boto3 to interact with AWS Services
Setup and Validate Jupyter based Interactive Environment
Review GHActivity Data Details
Download GHActivity Data using requests
Review GHActivity Data using Pandas
Core Logic to Ingest Data from Web Service to AWS s3
Managing s3 using Python boto3
Overview of AWS Dynamodb
Create DynamoDB Table for Job Details
Create DynamoDB Table for Job Run Details
Recap of Date Arithmetic using Python
Get First Run Details to Copy GHActivity Data to AWS s3
Get Incremental Load Logic for next file
Understand AWS s3 concepts such as buckets and objects
Copying or Uploading Files to AWS s3 as objects using Python boto3
Writing Python Objects or Data as AWS s3 Objects using boto3
Save GHActivity Data to AWS s3
Convert Date Time to Integer Unix Epoch using Python
Save Job Run Details to DynamoDB Table
Validate Data Copied to AWS s3 and job run details
Run and Validate End to End Process
Getting Started with AWS Lambda Functions
Introduction to Getting Started with AWS Lambda Functions
Overview of AWS Lambda and Getting Started using Python 3
Passing Arguments to AWS Lambda and Processing using Python
Using Custom Handlers for AWS Lambda Functions using Python 3
Using AWS Services such as s3 in AWS Lambda Functions
Recap of handling permissions using AWS IAM Roles and User Groups
Develop AWS Lambda Function to list objects from AWS S3 Bucket
Passing Environment Variables to AWS Lambda Functions
Customizing Resources such as memory used for AWS Lambda Function
Setup Local Development Environment for AWS Lambda Functions
Develop logic for AWS Lambda Function using external packages
Build Zip file to deploy as AWS Lambda Function
Deploy Application with External Dependencies as AWS Lambda Function
Understand Problem Statement for Python Application for AWS
Setup Python Project for AWS Lambda using Visual Studio Code
Core Logic to upload files to AWS S3 using Python boto3
Develop Python Application to upload files to AWS s3 using Python boto3
Build Zip File for Python Application to deploy as AWS Lambda Function
Deploy Python Application as AWS Lambda Function using Zip File
Conclusion and request for rating and feedback
Build and Deploy AWS Lambda Function using Zip File
Introduction to Build and Deploy AWS Lambda Function using Zip File
Update Application Code with Core logic for Ingestion
Overview of Validating User Defined Functions using Python CLI
Validate Application using Core Logic to ingest data
Add Lambda Handler to ingest data to AWS s3
Build Zip File for Python Application to deploy as AWS Lambda Function
Upload Python Application Zip File to s3 and deploy as AWS Lambda Function
Set Custom Handler and required Environment Variables for AWS Lambda Function
Granting Permissions on AWS s3 and Dynamodb to AWS Lambda Function via Role
Change Memory and Timeout for AWS Lambda Function and Test
Recap and Overview of Monitoring Lambda Functions using Cloudwatch
Limitations of Deploying AWS Lambda Function using Zip file
Automate Build of AWS Lambda Function using Shell Scripts
Build and Deploy AWS Lambda Function using Custom Docker Image
Introduction to Build and Deploy AWS Lambda Function using Custom Docker Imag
Create Dockerfile for Custom Docker Image for AWS Lambda Function
Create Custom Docker Image for AWS Lambda Function using Python 3 Run-time
Validate Custom Docker Image by creating Docker Container
Run the application using Python CLI in the Docker Container
Run the Docker Container with the Credentials and Environment Variables
Validate AWS Lambda Function Locally using Docker and Curl
Create AWS ECR Repository for Custom Docker Image
Push Custom Docker Image for AWS Lambda Function to AWS ECR
Create AWS Lambda Function using Custom Docker Image in AWS ECR
Run and Validate AWS Lambda Function created using Custom Docker Image
Create Shell Script to Build and Push Docker Image to AWS ECR
Add Command to build script to reconfigure AWS Lambda Function to latest docker
Overview of AWS s3 Event Notifications with Lambda or s3 Triggers on Lambda
Introduction to AWS s3 Event Notifications with Lambda or s3 Triggers on Lamb
Switching to Different Profile for the demos on AWS s3 Event Notifications
Setup Project to explore AWS Lambda Triggers or s3 Event Notifications
Setup Required Datasets for AWS Lambda Triggers or s3 Event Triggers
Listing AWS s3 Buckets and Objects using Python Boto3
Listing AWS s3 Objects based on Prefix using Python boto3
Overview of AWS s3 Events and Event Notification
Create Simple AWS Lambda Function for s3 Event Notifications or Triggers
Create Trigger for AWS Lambda Function for s3 Put Event
Get AWS s3 Object Details from AWS Lambda Arguments
Develop Read Logic to read from s3
Convert CSV to JSON Strings using AWS Lambda Function
Add Logic to Upload JSON File to s3 and Validate AWS Lambda Trigger
Configure AWS Lambda as AWS s3 Event Notification
Develop and Deploy AWS Lambda Function to transform the data
Introduction to Develop and Deploy AWS Lambda to transform the data
Basic Approach to convert JSON to Parquet using Python Pandas
Convert JSON Files to Parquet using Pandas in Chunks
Setup Job Metadata for JSON to Parquet
Add Core Logic to transform data to the Application Code
Develop Wrapper to transform the data from JSON to Parquet
Validate Function to Convert JSON to Parquet
Develop Lambda Handler to transform JSON Files to Parquet
Validate Lambda Handler to transform the data locally using curl
Push Docker Image to ECR and Update Existing Lambda Function
Deploy and Run Transformer as AWS Lambda Function
Perform Validations of AWS Lambda Function to Transform the data
Build and Schedule the Pipeline using AWS Lambda Trigger and EventsBridge
Introduction to Build and Schedule Data Pipeline using AWS Lambda Trigger and
Review AWS Lambda Trigger and understand arguments
Add Custom Lambda Handler for s3 Trigger
Add Trigger to AWS Lambda Function deployed to transform the data
Validate s3 Lambda Trigger by invoking ingestor
Understand Internals of AWS Lambda Functions
Overview of AWS EventBridge Rules
Overview of AWS EventBridge Rule for Schedule
Schedule Data Pipeline using AWS EventBridge Rule
Validate Data Pipeline Run by AWS EventBridge Schedule
Validate the pipeline after few Scheduled Runs
Reset Data Pipeline built using AWS Services
Determine File Not Found in the source using requests
Troubleshoot the bug in the Data Pipeline
Perform Adhoc Analysis of Data using Glue Catalog Tables and Athena Queries
Introduction to Adhoc Analysis of Data using Glue Catalog Tables and Athena Quer
Overview of AWS Glue to create Catalog Tables
Creating AWS Glue Crawler for GHActivity Data
Create AWS Glue Catalog Table using Glue Crawler
Getting Started with Amazon Athena to run SQL Queries on s3 and Glue Catalog Tab
Preview first few records and get count using AWS Athena Queries
Get Count By Date using AWS Athena SQL Queries
Setup AWS Profile for the VSCode Project and Install Pyathena
Getting Started with Pyathena and Pandas
Additional Athena Queries for adhoc Analysis using Glue Catalog Tables on s3Dat
Deploy AWS Lambda Functions using Python Runtime with Layers
Introduction to Deploying AWS Lambda Functions using Python Runtime with Layers
Create Lambda Function to explore layers
Get list of Python Libraries installed in AWS Lambda Runtime
Add Existing AWS Layer to Lambda Function using Python runtime
Steps to Add and Configure Custom Layers to AWS Lambda Functions
Setup Local Environment using AWS Cloud Shell to Create Custom Layer
Install Required Dependencies for Lambda Layer for Python Runtime
Create Zip File and Upload to s3 with Python dependencies for AWS Lambda Layer
Create Lambda Layer using AWS Lambda Console using zip file in AWS s3
Configure Lambda Function with Custom Layer for Pandas and Requests
Troubleshoot and Fix the issues related to Lambda Layers for AWS Lambda Functio
Upload Zip File with Python boto3 to s3 for AWS Lambda Layer
Create Lambda Layer with latest version of Python boto3 for AWS Lambda Function
Deploy AWS Lambda Function Sample Application with Layers