Case Study: Creating an ETL Data Pipeline using AWS Services - Real-World Problem

Architecture

Overview

This ETL pipeline leverages several AWS services to fetch, process, and store YouTube videos with translated subtitles. The core components include:

• AWS Lambda for processing video and audio
• Step Functions for Orchestrating Workflows
• S3 Buckets for storing raw and processed data
• AWS Transcribe for converting audio to text
• AWS Translate for translating the text into desired languages

---

Pipeline Steps

Fetching Video from YouTube

• Video is fetched from YouTube and sent to an initial Lambda Function.
• This function splits the video into separate audio and video files.

Step Functions Orchestration

• AWS Step Functions orchestrate all the following processes.
• The workflow controls each step, passing data between services and managing the pipeline flow.

Storing Raw Video and Audio

• Audio and video files are stored separately here.
• The bucket sends an email notification to confirm the files were successfully stored.

Merging Video and Audio Files

• This Lambda function combines the audio and video files into a single file.
• Error Handling: If merging fails, the pipeline stores the failed files in an Error Bucket.

Transcribing Audio to Text

• The merged video’s audio is sent to AWS Transcribe to convert audio into text.
• The resulting text is passed to the next Lambda function.

Proofreading Transcribed Text

• This Lambda function checks the transcription for accuracy and readability.
• If the text is poor quality, it can be flagged for manual review or re-transcription.

Translating Text into Target Language

• The proofread text is sent to AWS Translate to convert it into a chosen language (e.g., Arabic, Italian, or Spanish).
• The translated text is then passed to the next Lambda function.

Generating Subtitles and Merging Paragraphs

• This function formats the translated text as subtitles and merges paragraphs if needed.
• The final file is prepared for storage.

Storing Processed Video with Subtitles

• The fully processed video with subtitles in the desired language is stored here.
• This is the final output location where the processed video is accessible.

---

Error Handling

1. Error Bucket for Failed Merges: If audio and video files fail to merge, the Lambda Merge Function sends them to a designated Error Bucket.
2. Transcription and Proofreading Quality Check: Poor transcription quality detected by the Proofread Function can trigger a flag for manual review or re-transcription.

---

Notifications

**Email Notifications **for File Arrival in S3:After the raw video and audio files are stored in the S3 Raw Video and Audio Bucket, an email notification is sent to confirm successful storage.

---

Benefits of This Pipeline

1. Automation: Streamlines the entire video processing workflow without manual intervention.
2. Scalability: AWS services like Lambda and Step Functions allow the pipeline to handle many videos.
3. Language Support: AWS Translate enables easy translation to multiple languages, broadening the video’s reach.
4. Error Management: Dedicated error-handling buckets and flags ensure issues are logged and handled efficiently.