DEV Community: Warner Bell

I Found an LLM Weakness. Fixing It Cut My Token Usage 50%.

Warner Bell — Sat, 03 Jan 2026 13:47:53 +0000

I Found an LLM Weakness. Fixing It Cut My Token Usage 50%.

I hit this wall while building a personalized AI assistant I named Nathaniel. The prompt grew to over 1,000 lines: behavioral frameworks, crisis protocols, specialized knowledge, and operational procedures. It looked like it was working beautifully, until I began to notice particular instructions or directives were not being followed, despite consuming ~3,000 tokens before the conversation even started. After some digging and testing, I discovered the issue: for very long prompts, the model wasn't reliably processing content later in the document. Whether due to attention limitations, chunked file reading, or premature response generation, the effect was the same: instructions buried deep in the prompt were being missed.

The fix: put a Table of Contents at the top. When the model hits the TOC first, it has a reference map before it starts processing. It knows what sections exist and where to find them. Instead of hoping the model reaches the relevant instructions before responding, the TOC tells it exactly where to look. Loading only the relevant sections means those instructions are no longer buried deep, they're front and center. This also provided the ability to point the model directly at a particular section by name: "Use your writing protocol to enhance the following draft."

Other practitioners handle large prompts by splitting into multiple documents with routing logic, or building multi-prompt orchestration layers. Those work, but add deployment complexity. I wanted one document that the model could actually navigate, a way to prevent it from responding prematurely after the first chunk, or deprioritizing instructions buried later in the context.

This approach works in consumer interfaces like ChatGPT and Claude, but shows the most promise in agents and IDE-based assistants, environments where you control the system prompt and can apply context engineering practices with attached data files.

The traditional solutions all felt like compromises:

Split into multiple documents: Dependency management and deployment complexity
Load everything every time: Inefficient and slow
Create a simplified version: Lose the comprehensive capability that made it valuable

The Mechanics

LLMs have a finite window when reading and parsing files. For very long one-shot prompts, this means chunking: the model reads the first chunk, processes it, moves to the next. The problem: the model often hits the first chunk and attempts to respond immediately, missing critical directives laid out later.

The TOC changes this dynamic. When the model hits the TOC first, it has a reference map before it starts processing. Instead of deciding what to use from three massive chunks (and probably getting it wrong), it can load the entire context set that actually matters for the request.

The benefits cascade:

Smarter search: The model knows where to look before it starts looking
Token savings: Load only relevant sections instead of everything
Full context: Room for the complete context that matters, not truncated chunks
Speed: Faster processing by eliminating unnecessary content

I called it TOC-Based Dynamic Loading.

The initial implementation used line numbers to identify section boundaries. That broke immediately. Any edit to the document shifted all downstream line numbers. For a living prompt document that evolves constantly, line numbers were a maintenance nightmare.

The solution: HTML comment markers as anchors. Insert  at each section boundary. These markers are invisible to the LLM's output but serve as stable reference points. When you edit content within a section, the anchors don't move. The routing table maps keywords to anchor names, not line numbers. The pattern became maintainable.

💡 Key insight: Anchors survive content edits. Line numbers don't. For living documents, this is everything.

After weeks of refinement, production validation confirmed consistent results:

Metric	Result
Token reduction	44-63% per interaction
Processing speed	30-40% faster
Targeted loading rate	82% of requests
Fallback rate	18% (under 20% target)

Note: Fallback rate depends on keyword mapping granularity. Tighter mappings yield lower fallback rates.

How It Works for Personalizing Agents

The pattern has four components:

1. Core Context (Always Loaded)

The foundation every request needs: identity, behavioral constraints, quick reference. Think of it as the "personality kernel" that maintains assistant consistency.

Sizing guideline: Keep core context under 25% of your total document (if including it in a one-shot doc). If it's bigger, re-evaluate what's truly "always needed."

Another pattern: input core context (personality kernel) directly into agent instructions, with a pointer to an attached reference doc containing full behavioral instructions laid out in sections.

2. Keyword Mapping Table

A table that maps user request keywords to document sections:

| Keywords | Target Section | Priority |
|----------|---------------|----------|
| crisis, emergency, urgent | Crisis Management | HIGH |
| planning, strategy, roadmap | Strategic Frameworks | HIGH |
| code, build, deploy | Development Protocols | MEDIUM |

3. Targeted Loading

Based on keyword analysis, load 1-2 relevant sections instead of the entire document. Most requests need only a fraction of a comprehensive prompt.

4. Fallback Protocol

When requests are ambiguous or span multiple areas, load the full document. This ensures no capability loss. The pattern optimizes common cases without sacrificing edge cases.

The Critical Decision: Section Boundaries

This is where most implementations fail. How do you reliably identify where sections start and end?

I tested three approaches:

Approach	Mechanism	Maintainability	Drift Risk	Verdict
Line Numbers	Map keywords to line ranges	Low	High (breaks on any edit)	❌ Avoid
Split Files	Separate docs + index	Medium	None	⚠️ Situational
Anchor-Based	Map keywords to section anchors	High	Low (survives edits)	✅ Winner

Why Anchors Win

Line Numbers broke immediately. Any edit shifted all downstream line numbers. Maintenance nightmare.

Split Files worked, but added deployment complexity and made the system harder to reason about.

Anchor-Based Routing is the winner. Use HTML comment markers as anchors at section boundaries:

<!-- #crisis-management -->
## Crisis Management Protocol
[... 200 lines of crisis content ...]

<!-- #development -->
## Development Protocols  
[... 350 lines of dev content ...]

These HTML comment markers () are invisible to the LLM's output but serve as stable reference points. When you edit content within a section, the anchors don't move. For living prompt documents that evolve constantly, this is critical.

Your routing header maps keywords to anchor names, not line numbers. This is the Simple Version:

## TOC - Dynamic Section Loading

**Load only the section(s) matching task keywords. Fallback to full load if ambiguous.**

**Table of Contents:**

| Keywords | Anchor | Description |
|----------|--------|-------------|
| crisis, emergency | #crisis-management | Production issues |
| code, build, test | #development | Development work |

Anchors survive content edits within sections. For living prompt documents that you're constantly refining, this is the difference between a maintainable system and a fragile one.

A Real Example

Here's what this looks like in practice. My AI assistant prompt transformation:

Before:

1,089 lines loaded every interaction
~3,000+ tokens consumed before the conversation started
Noticeable latency on complex requests

After:

Core context: ~200 lines (always loaded) or loaded first.
Targeted sections: 200-400 lines (based on request)
Total per interaction: 400-600 lines (~1,200-1,800 tokens)
Fallback rate: <20% of interactions

Sample Routing Header (Robust Version)

For complex one-shot prompts with core context requirements:

## TOC - Dynamic Section Loading

### STEP 1: LOAD CORE CONTEXT (ALWAYS)
Identity & Behavioral Framework (~200 lines)

### STEP 2: ANALYZE USER REQUEST
Match keywords from the Table of Contents below.

### STEP 3: LOAD TARGETED SECTIONS

**Table of Contents:**

| Keywords | Anchor | Lines |
|----------|--------|-------|
| customer, account, support | #customer-protocols | ~250 |
| code, build, architecture | #development | ~300 |
| crisis, outage, emergency | #crisis-mgmt | ~150 |
| planning, strategy | #strategic | ~200 |
| team, mentor, feedback | #team | ~180 |
| write, document, blog | #writing | ~120 |

### STEP 4: FALLBACK
If no keywords match OR request spans 3+ areas → load full document

**Multi-section triggers**:
- "deploy to production" → #development + #crisis-mgmt
- "customer escalation" → #customer-protocols + #crisis-mgmt

When to Use This Pattern

✅ Apply when:

Document > 500 lines (~4,000 tokens): Smaller docs don't need optimization
3+ distinct, independently-useful sections: Content must be modular
Typical task uses < 50% of content: If most content is needed anyway, skip this
Sections have clear boundaries: Can be anchored reliably

❌ Skip when:

Sections are highly interdependent (need most content anyway)
Content is consumed sequentially (not random access)
Document changes frequently (keyword mapping overhead)

Use Cases Beyond AI Assistants

The efficiency gains matter, but what excites me most is what this pattern enables:

Portable Personal Context: Carry your AI assistant's personality, preferences, and learned patterns across different models and platforms. A well-structured prompt with dynamic loading can move from Claude to GPT to Gemini while maintaining consistent behavior.

Product Requirements Documents (PRDs): Feature specs with distinct sections (overview, requirements, constraints, acceptance criteria).

Technical Specifications: Architecture docs, API specs, system design documents with modular sections.

Behavioral Systems: AI agent instructions, persona definitions, role-specific operational knowledge.

Reference Documentation: Large reference materials, training content, compliance procedures accessed by topic.

Try It Yourself

To get started with large prompts:

Audit your prompt: Identify distinct, independently-useful sections
Define core context: What's needed for ANY request? Keep it under 25%
Add anchors: HTML comments at section boundaries
Build the routing table: Map user keywords to anchors
Add fallback logic: Ensure no capability loss

Start simple. Refine keyword mappings as you observe how users interact with your assistant.

Let's Discuss

I'm curious what patterns others use for prompt optimization:

Are you splitting documents?
Using different routing mechanisms?
Running into similar challenges?

Drop a comment. I'd love to hear what's working (or not working) for you.

Why I'm Sharing This

I spent weeks refining this pattern through trial and error. After my validation, I wanted to give back to the prompt engineering community. Context engineering is becoming a critical skill as the use of agents develops and prompts grow more sophisticated. Patterns like this will matter more as we push models to handle complex, persistent behaviors.

This article covers the pattern itself. The full case study on building Nathaniel, my AI assistant with 1,000+ lines of behavioral protocols, learning systems, and adaptive personality, is coming soon. If you're interested in the complete architecture and lessons learned, follow along.

This full pattern documentation is available as a GitHub Gist if you want to fork and adapt it.

🎧 AWS Audio Transcription Automation with CloudFormation

Warner Bell — Sat, 09 Nov 2024 16:51:45 +0000

Welcome to the AWS Audio Transcription Automation project! This CloudFormation stack automates transcription of audio files (MP4, MP3, and WAV) using Amazon Transcribe. Easily upload your audio files to S3, trigger transcription jobs, and store results in an output S3 bucket — all automated! 🎉

Architecture Diagram

GitHub repo: https://github.com/Warner-Bell/audio-transcription-build

🚀 Features

Automatic Transcription: Supports MP4, MP3, and WAV audio files, using Amazon Transcribe.
Secure Storage: AES-256 encryption for S3 buckets.
Lifecycle Management: Automatically expire input files after 1 day and log files after 2 days.
Logging: Logs Lambda function execution and S3 access.
Event-Driven Transcription: Automatically triggers transcription jobs upon file upload.
Final HTML Output: Transcription results are processed into an HTML file, viewable on any browser for easy reading.

🛠️ Technology Stack

Amazon S3: Storage for audio files, transcription results, and access logs.
AWS Lambda: Event-driven function to trigger transcription jobs.
Amazon Transcribe: Speech-to-text transcription service.
Amazon CloudWatch: Logs Lambda function activity.
IAM Roles: Manages permissions for Lambda and S3.

🎯 How It Works

Upload audio files (MP4, MP3, or WAV) to the designated S3 input bucket.
Lambda Triggered: Upon upload, a Lambda function triggers an Amazon Transcribe job.
Transcription Results: Transcription results are stored in the specified S3 output bucket, and formatted as an HTML file for easy viewing.

🧩 CloudFormation Resources

This CloudFormation stack provisions the following resources:

Input S3 Bucket: For audio files awaiting transcription.
Output S3 Bucket: Stores completed transcription results.
Logging S3 Bucket: Logs access events for both input and output buckets.
AWS Lambda Function: Automatically triggers transcription jobs.
IAM Role: Provides necessary permissions for the Lambda function.
CloudWatch Log Group: Logs Lambda function execution.

📦 Installation & Setup

Prerequisites

AWS CLI: Installed and configured on your local machine.
AWS Account: Ensure permissions to create CloudFormation stacks, Lambda functions, and S3 buckets.
Edit Deploy File: Edit the STACK_NAME constant to a unique name in the deploy-transcription.sh file. Update the region in the s3-trigger.sh file if your region is not us-east-1.

Deploy the Stack

Clone the repository:

   git clone https://github.com/Warner-Bell/audio-transcription-build.git
   cd audio-transcription-automation

Update the s3-trigger.sh file with you stack name

Run the deployment script:

   ./bin/deploy-transcription.sh

This script deploys the CloudFormation stack using a template file, setting up the required S3 buckets, Lambda function, and permissions.

Deploy S3 Trigger for Lambda: After deploying the CloudFormation stack, set up the S3 bucket trigger using s3-trigger.sh:

   ./bin/s3-trigger.sh

Note: The S3 trigger configuration script (s3-trigger.sh) is created as a separate script to avoid circular dependencies between the S3 bucket, Lambda function, and logging configuration.

Verify Deployment: Monitor the deployment progress in the AWS CloudFormation Console to ensure all resources are created successfully.
Start Transcribing: Once deployed, simply upload audio files to the designated input S3 bucket to start transcription. After running the S3 bucket trigger, upload your files to S3 using upload-to-s3.sh (Edit the file variables with your info.)

   ./bin/upload-to-s3.sh

Note on Output: Transcription output files are stored in both JSON and HTML format in the output S3 bucket. The HTML format can be viewed directly in any web browser for easy reading, eliminating the need to manually copy text from JSON files.

📂 Project Structure

.
├── audio-samples        # 3 sample audio files in mp3, mp4, and wav format
├── bin/
│   ├── deploy-transcription.sh # Deployment script
│   ├── empty-buckets.sh        # Empty all Project Buckets
│   ├── s3-trigger.sh           # Lambda role configuration and S3 bucket notifications
│   ├── upload-to-s3.sh         # Upload selected audio file to s3
├── cfn/audio-transcription.yaml         # CloudFormation template
├── README.md                   # Project documentation

📊 Monitoring & Logging

CloudWatch Logs: View Lambda execution logs in the AWS CloudWatch console.
S3 Access Logs: Access logs for input and output S3 buckets are stored in the logging bucket.

⚙️ Recommended Practices

Adjust Bucket Retention: Modify lifecycle policies to suit your data retention requirements.
Configure Notifications: Install and configure notify-send on Linux for deployment notifications (optional).
Review IAM Policies: Ensure permissions are as restrictive as possible for production use.

🤝 Contributing

We welcome contributions! To contribute, follow these steps:

How to Contribute

Fork the repo 🍴
Create a new branch:

   git checkout -b feature/awesome-feature

Commit your changes 💻
Push your branch and submit a PR 🛠️

🌍 License

This project is licensed under the MIT License. See the LICENSE file for details.

🏆 Acknowledgements

Thanks to AWS for their robust services. 💪
Special thanks to OpenAI for inspiring innovation with AI-based tools. 🙌

📬 Contact

Warner Bell - Tap In!

✨ Happy Transcribing! ✨

Building 'TheContentCaddie': My Participation in the PartyRock Generative AI Hackathon

Warner Bell — Sun, 27 Oct 2024 16:50:42 +0000

🚀 Introduction

Hello, Dev.to community! I’m excited to share my journey in the PartyRock Generative AI Hackathon by AWS. I recently worked on an exciting project: creating "TheContentCaddie," an AI Video Content Generation & Promotion Assistant. This app is designed to aid content creators in the daunting task of producing and promoting digital content, particularly for platforms like YouTube.

🤔 What Inspired Me?

My inspiration for the app came from my understanding of creating content for social media, especially video content for platforms like YouTube. I know how conceptualizing, organizing, gathering data, creating content, recording, editing, designing art and graphics, uploading, and promoting content is a job fit for a team. However, many creators, myself included, are trying to navigate all aspects of getting their content out to help the masses, grow a personal brand, or build an online following, often as a one-person band.
when I first got started I wanted to create some content documenting my experience and processes participating in the hackathon but juggling the content creation and meeting the hackathon's requirements seemed too daunting a task, especially with everything else going on in my work and life. That's when it hit me: my hackathon entry should be an app that helps creators with all the legwork involved in creating their content. Even better, I can use it to create the content documenting my hackathon joining experience, closing the loop, and achieving a state of Zen, with a paradoxical connotation where the need for a tool to help with content creation, inspired the creation of a tool, that would help with the creation of the content documenting the creation of the tool, that helped create the content… 🤯🤯🤯

🛠️ How I Built the Project

Building "TheContentCaddie" on AWS's PartyRock platform was an exhilarating challenge. I embraced the world of generative AI, leveraging the platform's capabilities to bring my vision to life. My process was a blend of creativity, technical skill, and a deep dive into the realms of LLM prompting.

The Development Process

Conceptualization: I started by defining the core functionalities: script generation, SEO optimization, concept image creation, and social media promotion.
Integration of Generative AI: Leveraging PartyRock's AI capabilities, I incorporated features for generating video scripts, titles, descriptions, and promotional content.
LLM Prompting: Learning about LLM prompting was a game-changer. Understanding how to craft prompts effectively was crucial in getting the desired outputs.
Building the Chatbot: The AI chatbot was integrated to offer expert social media marketing advice, enhancing the app's utility.

Tools and Technologies

AWS PartyRock: For the AI-driven backend.

🚧 Challenges Faced

One of the main challenges was harnessing the AI to produce the desired outcomes. It required a lot of ideation, iteration, and testing to understand the intricacies of the AI models and find the best methods to achieve my goals.

AI Responsiveness: Fine-tuning the AI to generate relevant and engaging content was challenging. It involved numerous iterations and tweaking of LLM prompts.
Integration Complexity: Ensuring seamless integration of different AI functionalities while maintaining a user-friendly interface was a steep learning curve.
Time Management: Balancing the project with my professional commitments was demanding, requiring effective time management skills.

📚 What I Learned

Throughout this journey, I've learned about the ease and speed of developing applications on PartyRock. I've gained insights into LLM prompting and how different inputs can significantly alter the outputs of language models. This experience has opened my eyes to the broader world of hackathons, and I'm excited to participate in more in the future.

LLM Prompt Engineering: The art of crafting effective prompts to guide AI output was a key takeaway.
Rapid Prototyping on AWS: The agility and power of AWS tools in bringing a concept to life were enlightening.
Hackathon Culture: The collaborative, innovative spirit of hackathons has sparked a new interest in me to participate in more such events.

🏆 Accomplishments that I'm proud of

This hackathon has been an incredible learning journey. I'm proud of diving into this challenge, exploring new territories in cloud technology, and creating an app that could potentially revolutionize content creation for many creators out there.

🎉 Closing Thoughts

Building "TheContentCaddie" was not just about creating an app; it was about solving a real-world problem that resonates with many in the digital content space. This project has been a journey of learning, growth, and immense satisfaction. I'm excited to see how it helps content creators streamline their workflows and achieve their creative visions more efficiently.

I hope my journey inspires fellow developers and creators. Let's continue to innovate and solve problems, one hackathon at a time!

Thanks for reading! Feel free to drop your thoughts or questions in the comments below. Happy coding!

Would you like to know more about "TheContentCaddie"? Check out the project on PartyRock or reach out to me on LinkedIn. Let's connect! 🌟

🔐 How to Generate a GitHub Personal Access Token (PAT)

Warner Bell — Sun, 27 Oct 2024 16:32:07 +0000

Follow these steps to create a GitHub Personal Access Token (PAT) to use for authentication with Git or other tools:

1. 🔑 Log in to GitHub:

Go to GitHub and log in to your account.

2. ⚙️ Access Developer Settings:

In the upper-right corner of any GitHub page, click your profile picture.
Select Settings from the dropdown.
Scroll down and, on the left sidebar, click Developer settings.

3. 🛠 Generate a Personal Access Token:

In Developer settings, select Personal access tokens from the sidebar.
Click on Tokens (classic) and then click Generate new token.
Generate New Token (Classic)
Give the token a descriptive name, such as "Git CLI Token", so you can remember why you created it.

4. 🔐 Select Scopes and Permissions:

Choose the scopes or permissions to grant this token. For full access to repositories (push, pull, etc.), select the repo scope.
You may also want to choose workflow for GitHub Actions or admin:repo_hook for repository webhooks.
If you only need specific permissions, select the minimum necessary scopes for better security.

5. ⚡ Generate the Token:

After selecting your desired scopes, click Generate token at the bottom of the page.

6. 📋 Copy the Token:

Important: Once the token is generated, copy it immediately. You won't be able to view the token again after you leave the page.

7. 🛡 Store the Token Safely:

Save the token in a secure location, like a password manager.
Use this token instead of your password when performing operations like pushing or pulling from a GitHub repository.

🧑‍💻 Using the Token with Git:

When prompted for a username and password during Git operations, use:

Username: Your GitHub username.
Password: Paste your newly generated PAT.

Example during git push or git pull:


shell
Username: your-github-username
Password: your-access-token

Setting Up a Basic Dev Environment on a Windows/Linux Machine

Warner Bell — Sun, 27 Oct 2024 16:30:04 +0000

🚀 Set Up Your Dev Environment

Welcome to your step-by-step guide for setting up a powerful development environment using Windows Subsystem for Linux (WSL), Visual Studio Code (VSCode), and essential tools like Git and Node Version Manager (NVM).

📑 Table of Contents

💻 IDE - Install Visual Studio Code (VSCode)
🔧 Configure VSCode for Development
🐧 WSL - Setting Up Ubuntu and Updating Linux Kernel
🛠️ Essential Dev Tools
- Node Version Manager (NVM)
- AWS CLI v2
🐍 Python and pip
⚡ Git - Installation and Configuration
🔑 Git Credential Manager (GCM)
🔒 Configure Security Settings
🎉 Final Congratulations

💻 IDE - Install Visual Studio Code (VSCode)

VSCode is a crucial tool for development. Let's get it installed and ready!

🔽 Steps to Install VSCode:

Download VSCode: Visit Visual Studio Code and download the Windows installer.
Run the Installer: Double-click the downloaded file and follow the instructions. Check the options to:
- Add VSCode to the PATH.
- Create a desktop icon for quick access.

🚀 Install VSCode Using PowerShell CLI:

$InstallerUrl = "https://code.visualstudio.com/sha/download?build=stable&os=win32-x64-user"
$InstallerPath = "$env:TEMP\VSCodeInstaller.exe"
Invoke-WebRequest -Uri $InstallerUrl -OutFile $InstallerPath
Start-Process -FilePath $InstallerPath -ArgumentList "/verysilent" -NoNewWindow -Wait
Remove-Item $InstallerPath

🚀 Install VSCode Using Linux CLI:

sudo apt install curl gpg
curl -fsSL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor -o /usr/share/keyrings/ms.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/ms.gpg] https://packages.microsoft.com/repos/code stable main" | sudo tee /etc/apt/sources.list.d/vscode.list
sudo apt update
sudo apt install code

After installing, launch VSCode by typing code in your terminal.

🔧 Configure VSCode for Development

🛠️ Steps to Configure VSCode:

Open VSCode: Launch VSCode from your Start Menu or Desktop Icon.
Install Extensions: Go to the Extensions Marketplace by pressing Ctrl+Shift+X. Recommended extensions:
- 🔗 Remote Development – for remote development within WSL.
- 🐍 Python – for Python development.
- 🧩 GitLens – enhances Git capabilities.
- 🐳 Docker – for containerization.
- ✨ ESLint – for JavaScript development.
- 📜 YAML – for YAML language support.
- 💻 Live Share – for collaborative coding.
- 🔐 Remote - SSH – to connect via SSH to remote servers.

🐧 WSL - Setting Up Ubuntu and Updating Linux Kernel

Build a robust foundation by setting up WSL and Ubuntu on your Windows machine.

🛠️ Steps to Install WSL & Ubuntu:

Install WSL Ubuntu:
- In VSCode Open a PowerShell terminal and enter:
```
 wsl --install -d Ubuntu
```

Click Yes if prompted to allow program to make changes, view install progress in terminal.
Wait for the installation to complete,
Reboot your machine. (a terminal window will open upon reboot)
In the terminal window type a username when prompted (all lowercase), enter a password wait for the process to complete then close the window.

Update Linux Packages:
- Now that WSL is up and running, we need to ensure our Linux packages are up-to-date.
- Open VSCode open your Ubuntu(WSL) terminal and run:
```
 sudo apt update && sudo apt upgrade -y
```

🛠️ Essential Dev Tools

To build high-quality software, install these essential tools:

sudo apt-get install wget ca-certificates curl

🚀 Node Version Manager (NVM):

Manage your Node.js versions efficiently with NVM:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/master/install.sh | bash

🚀 AWS CLI v2:

The AWS Command Line Interface (CLI) v2 is essential for interacting with AWS services directly from the command line. Install it with the following steps:

Installation:

For Linux:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

For Windows (via PowerShell):

msiexec.exe /i https://awscli.amazonaws.com/AWSCLIV2.msi

Verify the installation:

aws --version

This should return the installed version of AWS CLI v2.

🐍 Python and pip

Python is an essential tool for cloud development. Install Python and pip with:

sudo apt install python3-pip -y

✅ CFN-Lint (CloudFormation Linter):

Ensure quality in your CloudFormation templates:

pip3 install cfn-lint

Don't forget to install the CloudFormation Linter extension in VSCode!

🔧 jq (Command-line JSON processor):

jq is a lightweight command-line JSON processor, essential for parsing and manipulating JSON data:

sudo apt-get install jq

💽 Install Database Tools (if needed)

If your development requires a database, install the appropriate database. For example, for MySQL:

Download MySQL: Go to the MySQL Downloads page and select the Windows version.
Install MySQL: Follow the setup wizard, set up your root password, and configure the server.

⚡ Git - Installation and Configuration

Git is essential for version control. Install and configure Git with the following:

🛠️ Install Git:

sudo apt-get install git

🧑‍💻 Configure Git Identity:

git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"

🔍 View Git Configuration:

git config --list

Install Git CLI:

sudo apt install gh

🔑 Git Credential Manager (GCM)

Git Credential Manager (GCM) simplifies authentication for Git services like GitHub and Azure DevOps.

💡 Setting Up GCM in WSL:

Set up GCM within WSL:

git config --global credential.helper "/mnt/c/Program\ Files/Git/mingw64/bin/git-credential-manager.exe"

🔒 Configure Security Settings

🔧 Steps to Secure Your System: ### (Optional)

Windows Security: Ensure Windows Defender or your preferred antivirus is active and updated.
Firewall Settings: Configure your Windows Firewall to allow development tools while blocking unauthorized access.
Secure Browsers: Install security extensions like HTTPS Everywhere and ensure your browsers are up to date.
Regular Updates: Keep Windows OS, VSCode, and all installed packages/extensions updated.

🎉 Final Congratulations

You now have a fully configured, secure development environment on your Windows machine using WSL, VSCode, Git, and other standard development tools. You're ready for cloud development and ensuring your CloudFormation templates are linted before merging into the main branch.

Happy Coding! 💻🌐

Automating CloudFormation Deployments: The DevOps Way with AWS Git sync & GitHub Actions

Warner Bell — Sat, 27 Jan 2024 17:12:36 +0000

Hey Devs, Let's Git This Party Started!

CloudFormation Git sync provides remote management of stacks, enabling customers to synchronize their stacks from a CloudFormation template stored in a remote Git repository. You can enable CloudFormation Git sync through the AWS Console, CLI, and SDKs. With Git sync, you can manage your CloudFormation stacks using source control it works with GitHub, GitHub enterprise, GitLab, and BitBucket.

Step-by-Step Walk-Through Video

Kickoff: Setting Up Your GitHub Repo

Hop into GitHub: Just slide into your GitHub account. No account yet? No sweat, signing up is a breeze.
Dashboard Time: Once you're in, you're gonna land on your dashboard. It's your command center.
Repo Magic:

Hit the '+' sign up top. Feels like unlocking a treasure, right?
Choose 'New Repository' from the dropdown.

Repo Details:

Name your brainchild.
Public or Private? Your call.
Add a catchy description.
Init with a README? Always a smart move.
Pick a license if you're feeling it.

Create It: Smash that 'Create repository' button.
Bam! Repository Ready: Your new digital playground is set on GitHub.

Securing the Fort with a Token

Settings, Please: Click your profile pic and hit 'Settings'.
Developer Mode On: Find 'Developer settings' on the left.
Token Time: Under 'Access Tokens', select 'Personal access tokens'.
Token Generation:

Click 'Generate token'. Feels like launching a rocket, huh?
Name it something cool.
Pick the powers (permissions) wisely.
Expiry date? Your choice.

Copy That Token: Guard it like a secret treasure.

Local Repo Clone: Bringing it Home

Git Ready: No Git? Download it from the Git website.
Terminal or Command Prompt: Open it. It's like opening the door to Narnia.
Directory Navigation:

Use 'cd' to move to your desired folder. Desktop? cd ~/Desktop.

Clone Command:

git clone
Replace with yours.

Token Authentication: Private repo? Use your token as the password.
Cloned and Ready: Your repo's now on your local machine. Time to make magic!

Crafting Your CloudFormation Template.yaml

New File in IDE: Click 'File' > 'New File'.
Save with .yaml Extension: Name it like 'cfn-template.yaml'.
Template Time: Paste your CloudFormation template here.

AWSTemplateFormatVersion: 2010-09-09

Description: This is my gitsync demo template

Resources:

  GitsyncVpc:

    Type: AWS::EC2::VPC

    Properties:

      CidrBlock: 10.0.0.1/16

      EnableDnsSupport: true

      Tags:

        - Key: Name

          Value: gitsyncvpc

  GitsyncSubnet:

    Type: AWS::EC2::Subnet

    Properties:

      VpcId: !Ref GitsyncVpc

      CidrBlock: 10.0.0.1/24

      Tags:

        - Key: keyname

          Value: value

Template Validation: No Errors Allowed

Save and Validate: Run cfn-lint -t .yaml in your terminal. To check your template for any syntax errors and provide feedback in the Output pane - keep it error-free.

If you don't have cfn-lint installed, you can install it with the following command:

pip3 install cfn-lint

Deployment File: Setting Up for Success

New File Again: In your IDE, 'File' > 'New File'.
Save as 'deployment-file.yaml'.
Path and Tags: Include the file path of your CloudFormation template and any tags.

template-file-path: ./cfn-temp.yaml

tags:

    Name: 'gitsync'

    Project: 'Demo'

GitHub Actions: Keeping Your Code in Check

Setup and establish the following folder structure: .github/workflows/pull-request.yaml.
Define workflow to lint your pull requests.

name: Pull Request workflow

on:

  - pull_request

jobs:

  cloudformation-linter:

    runs-on: ubuntu-latest

    steps:

      - name: Checkout

        uses: actions/checkout@v3

      - name: Linter install

        uses: scottbrenner/cfn-lint-action@v2

        with:

          command: cfn-lint -t ./cfn-temp.yaml

Branching and Committing: Own Your Updates

Create a new branch
Add and commit your configs.

git switch -c gitSync-updateBranch

git add -A

git commit -m "add gitsync configs"

git push origin gitSync-updateBranch

Git Sync Prereqs: Keep Your Stack in Sync

AWS Console Login: Head over to the AWS Management Console.
Developer Tools Connection:

Search ' Code Pipeline ' in AWS services.
Navigate to Connections
Create a new connection to GitHub.

IAM Role Creation: Create the role that will deploy our CloudFormation template. Be sure to note the name you select for this as you'll be using it to manage your stack later. This example uses gitsync-cloudformation-deployment-role.

Navigate ' IAM ' in AWS services.
Navigate to Connections
Create a new role for Cloudformaton. (gitsync-demo-role)
Create an inline policy for the role. (gitsync-demorole-policy)

JSON:

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": [

                "ec2:CreateVpc",

                "ec2:CreateSubnet",

                "ec2:DescribeVpcs",

                "ec2:DescribeSubnets",

                "ec2:DeleteVpc",

                "ec2:DeleteSubnet",

                "ec2:ModifySubnetAttribute",

                "ec2:CreateTags",

                "ec2:ModifyVpcAttribute"

            ],

            "Resource": "*",

            "Condition": {

                "ForAnyValue:StringEquals": {

                    "aws:CalledVia": [

                        "cloudformation.amazonaws.com"

                    ]

                }

            }

        }

    ]

}

Link up with CloudFormation!

Once the role has been created, you'll create a new Stack:

Here, you can see the new option to select Sync from Git template source, which you can configure on the next screen. Since you already created your stack deployment file, you can select I am providing my own file in my repository.

Next, you can configure your Git integration to choose your repository. You'll need to use the Connection you created beforehand and select your repository. Select GitHub, your connection, the repository, and branch, the deployment file location.

Now, you will select New IAM Role to create a service managed role. This role will enable Git sync to connect to your repository. The role will be re-usable.

On the next page, you'll select the IAM Role you created earlier to manage this stack. This role controls the resources that CloudFormation will deploy..

Finally, you can see the status of your sync in the new "Git sync" tab, including the configuration you provided earlier as well as the status of your sync, your previous deployments, and the option to retry or disconnect the sync if needed.

Give It a Whirl

Get over to GitHub: log into your Github account.
Create a Pull Request: Create a new pull request and wait for lint checks to pass.
Merge Pull Request: Include the file path of your CloudFormation template and any tags. Return to the CloudFormation Console and see that the stack is being provisioned. You can also look at the stack details to see the events, outputs, etc. When the sync is complete Git sync will show the provisioning status as succeeded and the stack with all its resources will be deployed.

Conclusion: Wrapping Up Like a Pro

You've just set up a slick, automated environment for CloudFormation templates. Validation on pull requests and auto-deployment to your stack? Now when your repo/template is updated and merged gitsync will automatically make the changes to your cloudformation stack and resources. We got this! This is that next-level CI/CD for your infrastructure code, a brand new workflow for me, and as always Builder Fun!

Taking on the Cloud-Resume-Challenge: My Experience

Warner Bell — Tue, 16 May 2023 21:53:56 +0000

Inspiration
I recently stumbled across the Cloud-Resume-Challenge and thought to myself, “That looks like a fun and useful little project I can do to learn and play in the cloud!” After doing some research, I found all the information I needed on the challenge’s website and began planning my approach.

I had previously created a “Secure Static Website on AWS” using only the AWS console as a tutorial for my YouTube channel, @cloudoptimized. However, for this challenge, I wanted to do as much of it as possible in “Infrastructure as Code” (IaC).

The Cloud-Resume-Challenge has several components, including certifications, HTML, CSS, a static website, HTTPS, DNS, JavaScript, a database, an API, testing, IaC, source control, and CI/CD. I will describe how I approached each component of the challenge.

Certifications

Prior to finding this challenge, I had obtained the following certifications: AWS Certified Cloud Practitioner, AWS Certified SysOps Administrator — Associate, AWS Certified Solutions Architect — Associate, and Microsoft Certified Azure Fundamentals. These certifications provide a solid foundation for cloud computing and will be useful in my future career.

HTML and CSS

For my HTML and CSS, I used a free sample site template from HTML5UP. They have some amazing offerings 100% free under their creative commons. I modified the template to create a basic configuration resume.

Static Website

I created a fully responsive static portfolio website using AWS S3. I used AWS CLI, CloudFormation, and AWS CDK to automate the deployment of the website. The FQDN was registered using AWS ACM, and the site was hosted on AWS S3. I followed the structure laid out by the Cloud-Resume-Challenge as much as possible.

HTTPS

To ensure the website’s security, I obtained an SSL/TLS certificate from AWS Certificate Manager (ACM). ACM is a service that lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with AWS services and your internal connected resources.

DNS

I used Route53 hosted zones to create DNS records that alias to a CloudFront distribution and CNAMEs for the certificate. Route 53 is a highly available and scalable Domain Name System (DNS) web service. It connects user requests to internet applications running on AWS or on-premises.

JavaScript

I used JavaScript to code the visits counter for the cloud portfolio frontend. JavaScript is a scripting or programming language that allows you to implement complex features on web pages. Every time a web page does more than just sit there and display static information, JavaScript is involved.

Database

I used AWS DynamoDB as the database for the site’s visits counter. DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. It lets you offload the administrative burdens of operating and scaling a distributed database, so you don’t have to worry about hardware provisioning, setup and configuration, replication, software patching, or cluster scaling.

API

For the API requirement of the challenge, I used Lambda URL. A function URL is a dedicated HTTP(S) endpoint for your Lambda function. You can create and configure a function URL through the Lambda console or the Lambda API. When you create a function URL, Lambda automatically generates a unique URL endpoint for you. Once you create a function URL, its URL endpoint never changes.

Python

The code for the Lambda function, which retrieves and updates the views in the DynamoDB database, was written in Python. Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built-in data structures, combined with dynamic typing and dynamic binding, make it very attractive.

Tests

During and after completing the build process, I deployed my website and began testing its functionality. I tested each component of the website to ensure that it was working as intended and that there were no errors or bugs. I also ran load tests to ensure that my website was able to handle high levels of traffic.

Infrastructure as Code

To manage and provision infrastructure more efficiently, I’m building CloudFormation templates for key components of the site and writing bash scripts to deploy them. Additionally, for small deployments and testing, I plan to use either AWS CDK or AWS SAM. This approach is known as Infrastructure as Code (IaC), which involves managing infrastructure through code rather than through manual processes. By using IaC, I can more easily manage and maintain my infrastructure and ensure that any changes are properly versioned and documented.

Source Control

For source control, I chose GitHub, an Internet hosting service for software development and version control using Git. GitHub provides a distributed version control of Git as well as access control, bug tracking, software feature requests, task management, continuous integration, and wikis for every project. By using GitHub, I can easily collaborate with others and keep track of changes to my codebase over time. Additionally, GitHub’s built-in tools for issue tracking and task management make it easy to keep track of bugs and feature requests.

CI/CD (Back end)

To streamline the development process, I decided to combine the continuous integration and continuous deployment (CI/CD) of both the front end and back end into one process using GitHub Actions. GitHub Actions allows me to automate, customize, and execute my software development workflows right in my repository. By using GitHub Actions, I can more easily build, test, and deploy my code and ensure that everything works as expected before pushing it to production.

CI/CD (Front end)

For the continuous integration and continuous deployment of my portfolio website’s front end, I’ll also be using GitHub Actions. By leveraging GitHub Actions, I can automate the process of building and deploying my front-end code, which will save me a lot of time and effort. I’ve written a GitHub .yaml file using Workflow syntax for GitHub Actions with a customization for security that will update my portfolio website upon push from the local repo. This approach will help me maintain a more efficient development workflow and ensure that my site stays up to date with the latest changes.

Challenges

One of the things that I found challenging during this project was getting the automation set up and configured properly. There were a lot of different components to configure, and it took some time to get everything working together. However, once I had everything set up, it was very easy to deploy and manage my website.

Overall, I found the Cloud Resume Challenge to be a fun and challenging project that helped me learn a lot about cloud infrastructure and web development. The challenge forced me to learn about a lot of different tools that I hadn’t used before, and it was rewarding to see everything come together into a fully functional website.

End of The Day

The Cloud Resume Challenge is a great way to learn about cloud infrastructure and web development. The challenge forces you to learn about a lot of different services and tools, and it provides a structured framework for building a fully functional website. While the project can be challenging at times, the end result is a rewarding and valuable learning experience. I highly recommend the Cloud Resume Challenge to anyone who is interested in learning more about cloud infrastructure and web development.