DEV Community: gu1lh3rm3_x

Cost-Effective Log Management Strategy

gu1lh3rm3_x — Sat, 20 Sep 2025 03:10:35 +0000

At some point in the lifecycle of any application, log management becomes essential. Ideally, teams should start monitoring logs as early as possible, but in practice, logs often get deprioritized until a real problem arises. While cloud providers like AWS CloudWatch or Google Cloud Logging (Cloud Run) offer built-in solutions, they are not always the most convenient or cost-efficient tools for deeper log analysis.

Another challenge is sharing access. If you want another team—such as support, security, or compliance—to review logs, are you really going to grant them AWS or GCP access? That introduces unnecessary risk, since cloud environments contain sensitive resources beyond just logs.

So, the question becomes: how do we handle logs efficiently, securely, and cost-effectively?

The Cost Challenge of Logs

Most log management platforms (e.g., Datadog, Splunk, New Relic, Elastic Cloud) charge based on data volume ingested and retention period. This means costs scale directly with:

Log volume per month (e.g., 500GB vs 2TB).
Retention requirements (e.g., 7 days vs 90 days).

For example, retaining 1TB of searchable logs for 90 days can quickly rack up thousands of dollars in costs. If you don’t plan ahead, you might be paying for hot storage on data you’ll rarely query.

Step 1: Understand Your Log Requirements

Before designing your logging pipeline, answer these key questions:

1 What is our log volume?

Example: 2TB/month.
Collect actual metrics from CloudWatch, Cloud Logging, or your app servers

2 How long do we need searchable logs?

Example: 60 days retention.
Sometimes compliance requirements mandate 90–180 days (or more).

3 Where are our logs generated and stored today?

Example: logs stored in AWS S3.
S3 is excellent as a cold storage layer: inexpensive, durable, and compression-friendly.

Once you know volume + retention + source, you can make better architectural decisions.

Step 2: Separate Hot and Cold Storage

Not all logs need to be instantly searchable. To save costs, split logs into:

Hot storage:
- Short retention (e.g., 7–14 days).
- Indexed and searchable in tools like OpenSearch, Elasticsearch, or Datadog.
- Used for active debugging, monitoring, and incident response.
Cold storage:
- Long retention (e.g., 60–180 days).
- Stored cheaply in S3, Glacier, or equivalent.
- Only re-indexed when needed (via scripts or batch jobs).

This layered approach drastically reduces costs while still keeping historical data available.

Step 3: Indexing and Visualization

S3 alone is not searchable—you need a system that can index logs and provide visualization. Options include:

Open-source:
Commercial SaaS (faster to set up, but pricier):
- Datadog Logs
- New Relic
- Splunk

Each comes with trade-offs in scalability, query performance, and price. OpenSearch and Loki are great for cost-conscious teams, while Datadog and Splunk offer convenience at a premium.

Step 4: Log Shipping (Data Ingestion Layer)

To make logs available in your chosen tool, you need to ship logs from the source to the destination. Options include:

AWS-native:
- Amazon Kinesis Data Firehose – reliable, scalable, but sometimes overkill.
Lightweight log shippers:
- Fluent Bit – very fast, low resource usage, widely adopted.
- Vector (by Datadog) – simple config, good performance.
- Filebeat – part of the Elastic stack.

These agents can pull logs from CloudWatch, Cloud Logging, or directly from application containers, and push them into OpenSearch, Elasticsearch, or a SaaS platform.

Step 5: Rehydrating Logs from Cold Storage

When older logs are needed (for audits, investigations, or post-mortems), you don’t want them indexed 24/7. Instead:

Pull compressed logs from S3 (or Glacier).
Run a reindexing script that ingests them back into OpenSearch or Elasticsearch temporarily.
Once the investigation is done, drop them from hot storage again.

This “rehydration on demand” model ensures you balance cost efficiency with data availability.

Example Architecture

Here’s how the pieces fit together:

Applications & Services generate logs.
Fluent Bit / Vector agents collect and forward logs.
Logs flow into OpenSearch (hot storage) with a retention of ~7 days.
In parallel, logs are stored in S3 (cold storage) with a retention of 60–180 days.
If old logs are needed, a rehydration script reindexes data from S3 into OpenSearch.
Teams access logs securely via OpenSearch Dashboards / Kibana, without needing AWS or GCP console access.

Key Takeaways

Don’t pay for hot storage on all logs—split between hot and cold.
Use lightweight shippers like Fluent Bit or Vector to control ingestion.
Leverage S3 for retention—cheap, reliable, and compression-friendly.
Provide access through log platforms, not cloud consoles—safer and easier for collaboration.

By structuring your logging pipeline this way, you’ll achieve a balance between cost, performance, and security, while keeping your team efficient when troubleshooting issues.

Using Docker + Traefik + WordPress on Hostinger VPS

gu1lh3rm3_x — Mon, 08 Sep 2025 18:49:15 +0000

Recently, a friend of mine came to me with an idea: he wanted a WordPress site where he could “upload” old console games (SNES, Game Boy, etc.) so people could play them directly from their browser—even on mobile.

He already knew what to do on the WordPress side, but first he needed the right infrastructure to host everything.

I told him:

“Hey, I actually have a VPS at Hostinger. If you want, we can split the cost and I’ll set up WordPress there for you.”

He agreed—and that’s where the fun began.

Step 1: Running Multiple Projects on One VPS

I knew I wanted the flexibility to run multiple projects on this VPS, not just WordPress. That meant I needed a way to isolate apps and keep them easy to manage.

The answer: Docker.

The idea was straightforward:

Run WordPress in a container.
Use MariaDB as the database.
Store data in persistent Docker volumes.

Here’s my first docker-compose.yml:

version: '3.9'

services:
  db:
    image: mariadb:10.11
    container_name: wordpress_db
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: somepass
      MYSQL_DATABASE: somedatabase
      MYSQL_USER: someuser
      MYSQL_PASSWORD: somepassword
    volumes:
      - db_data:/var/lib/mysql

  wordpress:
    image: wordpress:latest
    container_name: wordpress_app
    depends_on:
      - db
    ports:
      - "8080:80"
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: someuser
      WORDPRESS_DB_PASSWORD: somepass
      WORDPRESS_DB_NAME: somedatabase
    volumes:
      - wordpress_data:/var/www/html

volumes:
  db_data:
  wordpress_data:

It worked fine, but I didn’t want credentials hardcoded in YAML. So, I moved them into a .env file and updated the configuration.

Step 2: Making It Production-Ready with Traefik

At this point, we had a functional WordPress setup—but it was only accessible via:
http://VPS_IP:8080

That’s not practical. We needed:

HTTPS support with certificates.
A reverse proxy to route requests to different containers (since we planned multiple projects).
Proper DNS mapping.

Enter Traefik—an HTTP reverse proxy and ingress controller.

The setup looked like this:

SSL certificates via Let’s Encrypt
Traefik as reverse proxy → routes traffic to WordPress
MariaDB as the database
Everything connected on the same Docker network

Here’s the updated docker-compose.yml:

version: '3.9'

services:
  traefik:
    image: traefik:v3.0
    container_name: traefik
    restart: always
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--api.dashboard=true"
      - "--api.insecure=false"
      - "--certificatesresolvers.le.acme.email=youremail@example.com"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.tlschallenge=true"

    ports:
      - "80:80"
      - "443:443"
      - "8080:8080" # Dashboard (optional)
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./letsencrypt:/letsencrypt"
      - "./traefik_logs:/var/log/traefik"
    networks:
      - wpnet

  db:
    image: mariadb:10.11
    container_name: wordpress_db
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
      MYSQL_DATABASE: ${MYSQL_DATABASE}
      MYSQL_USER: ${MYSQL_USER}
      MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    volumes:
      - db_data:/var/lib/mysql
    networks:
      - wpnet

  wordpress:
    image: wordpress:latest
    container_name: wordpress_app
    depends_on:
      - db
    restart: always
    environment:
      WORDPRESS_DB_HOST: db:3306
      WORDPRESS_DB_USER: ${WORDPRESS_DB_USER}
      WORDPRESS_DB_PASSWORD: ${WORDPRESS_DB_PASSWORD}
      WORDPRESS_DB_NAME: ${WORDPRESS_DB_NAME}
      WORDPRESS_CONFIG_EXTRA: |
        define('WP_HOME','https://HOST_DNS.com');
        define('WP_SITEURL','https://HOST_DNS.com');
    volumes:
      - wordpress_data:/var/www/html
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.wordpress.rule=Host(`HOST_DNS.com`) || Host(`www.HOST_DNS.com`)"
      - "traefik.http.routers.wordpress.entrypoints=websecure"
      - "traefik.http.routers.wordpress.tls.certresolver=le"
    networks:
      - wpnet

volumes:
  db_data:
  wordpress_data:

networks:
  wpnet:
    driver: bridge

Now the site could be accessed via:
https://HOST_DNS.com

Step 3: Backups & Recovery

A working site is great—but what about backups?

I wrote a simple backup script with a cronjob that:

Dumps the database
Archives WordPress files
Copies Traefik certificates
Deletes backups older than 7 days

I also created a restore script, which let me choose which parts to restore (DB, WordPress files, or certificates).

This way, if anything breaks, I can recover quickly.

Step 4: Managing Secrets

Right now, secrets live in a .env file, which isn’t ideal for production. The next step is to move them into a proper secrets manager (e.g., Docker secrets, HashiCorp Vault, or a cloud provider’s secret manager).

That will make the setup more secure.

Final Thoughts

This little project started as “let’s host WordPress on a VPS” and turned into a practical exercise in:

Containerization
Reverse proxying with Traefik
SSL automation
Backups & recovery
Secrets management

It was a great hands-on way to think about infrastructure from scratch.

If you found this helpful, drop a like, leave a comment, or share it with a friend who might enjoy it. 🚀

Automating YouTube Shorts with Python and AI

gu1lh3rm3_x — Sat, 26 Jul 2025 03:30:21 +0000

Once again, I found myself a bit bored — and when that happens, I usually end up building something random. After chatting with an AI for a while, I decided what my next mini project would be: automating the creation of short videos.

The initial idea was simple:

Use AI to generate a short, curiosity-driven text

Generate an image related to the topic

Convert the text to speech using tools like gTTS or ElevenLabs

Combine everything into a short video

🛠️ First Attempt: Static Image + Audio

Here’s the basic code that generates a short video from an image and an audio file:

from moviepy import ImageClip, AudioFileClip, CompositeVideoClip

def create_video(image_path, audio_path):
    audio = AudioFileClip(audio_path)
    image = ImageClip(image_path).with_duration(audio.duration).resized(height=1280)
    image = image.with_position("center").with_audio(audio)

    video = CompositeVideoClip([image])
    video_path = "content/short.mp4"
    video.write_videofile(video_path, fps=24)
    return video_path

The result? It worked — but it was just a static image with background narration.

AI-generated text:

Cleopatra lived closer in time to the invention of the iPhone than to the building of the Great Pyramid.

Image suggestion:

iPhone

Not bad for a first try — a basic but functional automated Shorts generator!

📝 Adding Text to the Video

Next, I wanted to overlay the generated text on top of the video. I ran into a small font issue, which I fixed by explicitly setting a font path:

font_path = '/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf'
if not os.path.exists(font_path):
    font_path = None  # fallback if not found

With that, the video creation function evolved:

from moviepy import ImageClip, AudioFileClip, CompositeVideoClip, TextClip

def create_video(image_path, audio_path, text):
    audio = AudioFileClip(audio_path)
    image = ImageClip(image_path).with_duration(audio.duration).resized(height=1280)
    image = image.with_position("center").with_audio(audio)

    font_path = '/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf'
    if not os.path.exists(font_path):
        font_path = None

    txt_clip = TextClip(
        text=text,
        font=font_path,
        font_size=48,
        color='white'
    ).with_position('top').with_duration(audio.duration)

    video = CompositeVideoClip([image, txt_clip])
    video_path = "content/short.mp4"
    video.write_videofile(video_path, fps=24)
    return video_path

Now, we had videos with text overlays!

Still not perfect — the text stayed static throughout the video — but progress nonetheless.

🎬 Making Text Dynamic (Like Subtitles)

I wanted the text to appear gradually, in sync with the narration. I decided to break the text into sentences and display each one sequentially. Here’s how I handled that:

import re

# Split text into sentences
sentences = re.split(r'(?<=[.!?]) +', text)
n = len(sentences)
duration_per_sentence = audio.duration / n if n > 0 else audio.duration

subtitle_clips = []
for i, sentence in enumerate(sentences):
    start = i * duration_per_sentence
    end = start + duration_per_sentence
    subtitle = TextClip(
        text=sentence,
        font=font_path,
        font_size=20,
        color='black'
    ).with_position('center').with_start(start).with_duration(duration_per_sentence)
    subtitle_clips.append(subtitle)

video = CompositeVideoClip([image] + subtitle_clips)

The result? A much more engaging video with properly timed subtitles.

🧠 What I’ve Learned So Far

This mini project isn’t finished — but here’s what I’ve picked up along the way:

🎥 How to create videos in Python using moviepy

🗣️ How to convert text to speech with gTTS and ElevenLabs

🕒 How to sync subtitles with narration

🤖 How to integrate simple AI-generated content

🖼️ How to add multiple images in a slideshow format (WIP)

There’s still plenty of room to improve — syncing voice and subtitles more precisely, adding transitions, animations, or even background music — but this foundation already opens up a lot of possibilities.

If you’re curious, the project is on GitHub:
👉 shortomated GitHub repo

💭 Final Thoughts
Automation is becoming increasingly accessible — especially with the help of AI. While this project isn't fully AI-powered, it demonstrates how combining tools like gTTS, Unsplash API, and moviepy can produce impressive results with relatively little effort.

Hope you found this article useful or at least a little inspiring. Stay curious — and keep building!

How TDD Can Save You from Shipping Broken Code

gu1lh3rm3_x — Thu, 10 Jul 2025 22:30:00 +0000

Whenever we start thinking about testing our applications, the same questions always come up:

How many tests are enough?
Should I only write end-to-end tests?
Are unit tests enough?
What about the test pyramid — am I even doing it right?

Well, here’s the easy (and annoying) answer: it depends. But one thing is always true — when we have good tests, we have more robust systems.

That’s a fact.

You can’t rely solely on QA to catch everything. You can’t even rely on yourself. As developers, we naturally follow the “happy path” and often overlook edge cases. That’s why having automated tests in place is so crucial — they protect your system and give you immediate feedback when something breaks.

🧱 Start Small, Start Smart
You might be thinking:
"I have a huge project, and we don’t even have the initial test setup correctly!"

Don’t worry — I’ve been there. And the best way to get started is to start simple.

Begin by defining which types of tests you want to introduce first. My suggestion? Start with small unit tests.

Focus on functions that are heavily used across your codebase — maybe a validation function or a core utility. These are the areas where a single change can ripple through your entire application.

Let’s say someone decides to refactor one of those key functions.
Without tests, how will you know the behavior hasn’t changed? How long will it take before a user notices something’s broken — or worse, before production crashes?

🚨 Tests Catch Problems Immediately
Once you have a basic test setup and start covering the most critical parts of your system — and assuming you’ve integrated those tests into your CI/CD pipeline — something magical happens:

You get immediate feedback.

If something breaks, you’ll know right away. And when that happens, you can fix the issue before it reaches production, not after a flood of user complaints.

🧠 TDD: Think Before You Build
Now let’s talk about TDD — Test-Driven Development.

TDD means writing your tests before you write the actual implementation. Yeah, I know — it sounds backwards. But there’s a reason why so many seasoned devs advocate for it.

When you write tests first, you force your brain to:

Think about edge cases before coding.
Fully understand the task before jumping in.
Consider how users might misuse or break your feature.
Anticipate security and data issues upfront.

In short, you’re not just writing code — you’re designing behavior.

🤔 Should You Do TDD?
That’s a tough one. I won’t give a blanket “yes,” because it depends on your project, your team, and your deadlines.

But you should absolutely consider trying it.

At first, it might feel unnecessary or slow. But as your system grows, you’ll start realizing something important:
Every single test you wrote is holding your system together like glue.

Even if you don’t start with full-on TDD, writing tests right after completing a task is still a huge win. You’ll build confidence and avoid regressions later.

💬 Testing is Culture — But Also Survival
In some companies, testing is seen as a cultural thing — something dev teams “believe in.” But it’s more than that.

It’s a survival strategy for long-term projects. It’s what keeps your app stable as new features get added and refactors happen.

Testing (and TDD in particular) isn’t just about code quality — it’s about peace of mind.

🚀 Final Thoughts
If you’ve never tried TDD before, give it a shot. Pick a small feature this week, and try writing the tests first. You might be surprised how much clarity it gives you — and how much time it saves you later.

Thanks for reading — and happy testing!

How I automated recon with LLMs + nmap?

gu1lh3rm3_x — Wed, 09 Jul 2025 13:36:41 +0000

Since diving into CTFs more seriously, I found myself stuck in the same loop:
🔍 Run Nmap
📄 Read the results
🤖 Ask GPT for insights

One day I thought:
"Why not automate this?"
What if I could create a tool that runs Nmap in the background and feeds the output directly into an AI agent?

That was the spark. I started chatting with GPT to figure out how to approach it. My goal wasn’t to reinvent Nmap — I wanted to build on top of it, keeping all its power intact.

I chose Python to build the CLI, keeping things simple. The flow looked like this:

Run Nmap with some default parameters
Parse the results and organize them into a clean table
Send a prompt with context to an LLM (I used Gemini because it's free 😄)
Make an API call to Vulners to look up known vulnerabilities
Add a final touch: some ASCII art for fun 🎨

And that was it — I ended up with a tool that pulls recon data, enriches it with external sources, and asks an AI to help me interpret it.

It was a super fun project, and I learned a lot by building it from scratch.

🚀 Curious? Check it out at the link below!

https://github.com/Guischweizer/net-AINALYZER

Why does observability matter?

gu1lh3rm3_x — Fri, 04 Jul 2025 11:30:00 +0000

When we think about software development, we often focus on features, code quality, and scalability — but we sometimes forget one of the most critical aspects: observability.

Observability is the ability to measure the internal state of a system based on the data it produces — like logs, metrics, and traces. It’s what enables us to answer vital questions about our systems without shipping new code or SSH-ing into servers.

For example:

How much CPU and memory are our services using?
Which endpoints are consuming the most resources?
Are we handling traffic efficiently, or is our system under strain?
Could we scale down and save money, or do we need to prepare for 3x the load?

But observability isn't just about cool dashboards and fancy graphs. It's about proactive control. It's about being alerted before something goes wrong — not after users are already complaining.

By setting up proper alerts (e.g., high CPU, memory usage, request latency), you can catch problems early and react before incidents escalate.

Tools like Prometheus and Grafana make it easier to collect and visualize metrics from all your services, giving you a centralized view of your infrastructure's health. Once you have that in place, it opens up a whole new layer of responsibility: incident response and recovery plans. What happens if one service fails? Are we prepared to handle it?

Observability is not optional for systems with real users and business impact — it's a necessity. It helps you move from a reactive to a proactive mindset.

Thanks for reading!

How would I scale a project

gu1lh3rm3_x — Thu, 03 Jul 2025 10:30:00 +0000

Scalability is one of the most important aspects of software development — but what does it really mean?

The answer often depends on the project you're working on. For example, let’s say you're building a small e-commerce site for a local business. How much does it really need to scale? Is it necessary to handle millions of requests per minute? Probably not.

But now imagine you're working on a platform like Facebook or Google — you're dealing with millions of simultaneous requests, users, services, and data. At this level, your system must be prepared to scale reliably and efficiently.

Start Simple
Below is a very simple example of a common architecture pattern:

This represents a basic flow: a client sends a request to an API, which processes business logic, validates data, and interacts with the database.

This kind of architecture is fine for small or personal projects — it's easy to understand, quick to deploy, and sufficient for limited traffic.

Scaling Up: What Changes?
When we move toward a robust and scalable architecture, several new concerns come into play — especially around availability, performance, and security.

Some critical components to consider:

Load Balancer – Distributes traffic across multiple servers to prevent overload.
Cache – Reduces database load by storing frequent responses (e.g., Redis, CDN).
Observability – Monitoring, logging, and tracing to understand system behavior.
Security – Protecting data and services at all levels of the stack.
Latency – Physical server location matters. Regional distribution reduces delay.
Availability – Ensuring the system is always accessible through redundancy and backups.

A More Scalable Architecture
Here's what a more advanced setup might look like:

With more components in place, the complexity increases significantly. You’ll need to:

Analyze request times and performance bottlenecks.
Identify content that can be cached to reduce database hits.
Ensure databases have replication and automated backups.
Set up monitoring systems to catch failures before users notice them.
Plan for failure: a good architecture expects components to break and handles it gracefully.

But Here's the Catch: Trade-Offs
Scaling a project isn't just a technical challenge — it's also about managing compromises. Every decision adds complexity, cost, or both.

Here are some common trade-offs:

🧠 Complexity vs Simplicity:
A scalable system is harder to build, test, and understand. More moving parts = more potential points of failure.

💸 Performance vs Cost:
High availability, redundancy, and distributed regions all add cloud costs. You need to ask: Is it worth it at my current scale?

🧪 Speed vs Reliability:
Introducing caching or async processing improves speed but can introduce eventual consistency issues.

🔐 Access vs Security:
Scaling often means exposing more APIs, services, and endpoints — all of which need proper access control and protection.

These trade-offs mean that scaling should be intentional, not automatic. Just because you can build a system like Netflix doesn’t mean you should — especially if your needs are much simpler.

Final Thoughts

This is just a glimpse of how quickly software systems can become complex as they grow. Starting simple is fine — but as your application gains users and responsibilities, so must your architecture evolve.

Next time you’re starting a project, ask yourself:

“What would it take for this to scale 10x? 100x?”

And more importantly:

“What am I willing to trade to make that happen?”

Should You Create Your Own Tools?

gu1lh3rm3_x — Wed, 02 Jul 2025 21:30:00 +0000

As developers, engineers, or hackers, we often find ourselves surrounded by a vast array of existing tools. Most of the time, it makes perfect sense to use what’s already available — it saves time, and it's battle-tested.

But what if you're working on a personal project?
Why should you rely only on what’s already built?
Why not create something of your own — tailored to your needs?

Let’s be clear: if you're working on a company project where time is critical, building a tool from scratch is usually not the best option. The cost of reinventing the wheel might be too high.
But when you're on your own time, learning and experimenting, building your own tool can be one of the most valuable things you can do.

We’ve become so focused on efficiency and productivity that we often forget the fun part of being a developer: the act of creating.

When you build something from scratch, you’re forced to understand the problem it solves deeply. You’re also free to pick the language, framework, and approach — and that’s where true learning happens. You’ll build your curiosity, grow your creativity, and maybe even come up with something useful for others.

So next time you want to learn a new language or skill, don’t just build a "hello world" or basic CRUD.
Build something you think is cool.
And don't be afraid to ask for help — from a friend, a mentor, or even AI like ChatGPT.

Make things. Break things. Learn things. That’s what being a developer is all about.