DEV Community: capcom6

tnfy.link - What's about ID?

capcom6 — Tue, 14 Jan 2025 00:00:17 +0000

Hello everyone!

Welcome to the second part of my series about tnfy.link — yet another URL shortener! In this post, we’ll dive into the fascinating process of generating short links. While this might sound straightforward, choosing the right method for link generation comes with unique challenges.

At its core, generating a short link involves creating a short, unique ID for each long URL. The ID should meet several criteria:

Be unique to avoid conflicts.
Be short enough for practical use.
Be easy to type without introducing errors.
Be unpredictable to prevent guessing.

After researching various approaches, I identified four main methods for generating short links. Let’s explore them in detail.

1. Random Bytes

The simplest method involves generating random bytes and encoding them. However, it’s important to understand the difference between pseudo-random and cryptographically secure random numbers.

Pseudo-Random Numbers

The math/rand package in Go provides a pseudo-random number generator (PRNG). With the same seed (an initial value), it produces the same sequence of numbers. While this is sufficient for many applications, it’s not suitable for secure or unpredictable link generation.

Cryptographically Secure Random Numbers

For more secure random numbers, the crypto/rand package is ideal. It generates truly random and unpredictable values by leveraging system noise. For example, electromagnetic noise captured at the physical level can be used. This ensures high entropy, but keep in mind that virtual machines may rely on their host for random data, which could slow down generation in high-load environments.

Encoding Random Bytes

Random bytes alone aren’t suitable for URLs, so they must be encoded. Here are the most common encoding methods:

Integer: Converts bytes to an integer. Easy to type but may result in longer IDs.
HEX: Encodes bytes in hexadecimal (0-9, A-F). Case-insensitive and typo-resistant.
Base64: Encodes bytes with characters A-Z, a-z, 0-9, +, /, and =. However, it’s case-sensitive and prone to typos.
Base58: Similar to Base64 but excludes confusing characters (e.g., I, l, O, 0). This makes it more user-friendly. Examples include the implementations by Bitcoin, Ripple, and Flickr.

For user-friendly short links, Base58 is often the best choice due to its balance of compactness and error resistance.

Key Takeaways:

Random bytes are unique and unpredictable.
Encoding methods like Base58 enhance usability.
Cryptographically secure randomness ensures reliability.

2. Hashing

Hashing involves generating a fixed-length value based on the input (e.g., the long URL). While it guarantees consistency—hashing the same input always produces the same output—it lacks randomness. This means multiple requests to shorten the same URL will yield identical IDs, which doesn’t meet the unpredictability requirement.

Adding a random salt to the input before hashing can introduce variability, but at that point, using raw random bytes becomes simpler and more efficient.

3. UUID

UUIDs (Universally Unique Identifiers) are widely used for generating unique values. While they are effective, their default format is too long for short links. However, re-encoding UUIDs (e.g., in Base58) can reduce their size.

An alternative to UUID is NanoID, which generates shorter strings (21 characters by default) by using a customizable alphabet. This allows you to optimize IDs for readability and error resistance.

Why Not Use UUID?
UUIDs are ultimately based on random bytes, so there’s no significant advantage over generating raw random values directly.

4. Sequence

Random values can occasionally result in duplicates, especially under high load or with shorter IDs. While tnfy.link isn’t designed for high-load scenarios, it’s still worth considering potential issues.

Using a sequential counter ensures uniqueness by design. Tools like Redis can implement a distributed counter with the INCR command. However, sequential IDs are predictable. Combining a sequence with random bytes addresses this issue, ensuring both uniqueness and unpredictability.

For example:

Random Value + Incrementing Sequence: If two instances generate the same random value, the sequence ensures uniqueness.

Note: Including a sequential component in your IDs might reveal the total number of links generated, which could be undesirable in some contexts.

Conclusion

In this post, we explored various methods to generate short links:

Random bytes: Simple and effective, especially with secure encoding like Base58.
Hashing: Reliable but lacks randomness for this use case.
UUID/NanoID: Great alternatives but add unnecessary complexity compared to raw random bytes.
Sequence: Solves collisions but increases ID length.

For most use cases, random bytes with Base58 encoding are sufficient. To handle collisions in high-load scenarios, combining random bytes with a sequential component is a robust option. While this isn’t yet implemented in the current version of tnfy.link’s backend, I plan to add it as an optional feature in the future.

Thanks for reading! I’d love to hear your thoughts and experiences with link generation. Share your feedback in the comments below!

Related Post

If you’re interested in learning more about my projects, check out my article on SMS Gateway for Android.

tnfy.link - One more shortener?

capcom6 — Tue, 24 Dec 2024 03:00:00 +0000

Because English is not my native language I've used some AI-tools to improve the text, you can find original version here. I appreciate your feedback which version you like more.

Hello, everyone!

The Beginning

This story begins with a simple task: I needed to create posts containing a link to a service I didn’t control and track the number of clicks on that link.

The first thing I did was turn to Google and search for URL shortener services. I found plenty of options. However, some were overly complicated, others included ads or required registration, and some were frustratingly slow. None of them fit my requirements.

I also came across several Open Source projects on GitHub. But then I thought: I’m a developer too. Why not create my own URL shortener—simple, functional, and tailored to my needs?

What Should I Use?

Having prior experience with Go and Fiber, choosing my web stack was easy. Initially, I experimented with Huma, but I encountered some limitations, such as restricted support for sub-routers. Since my project only required a few REST endpoints (where Huma excels), I decided to stick with raw Fiber.

Next came the question of storage—for mapping short and long links and storing statistics. Since I wasn’t planning a high-traffic service, I could use any solution: RDBMS, NoSQL, or key-value storage.

I prefer self-hosted solutions, so SaaS like Firebase was off the table. If this series interests you, I’ll write a post about the deployment process.

I value simplicity. Storing a short-long link pair sounded like a “key-value” use case, so I chose Redis.

Some may think I picked Redis for its speed. You’re partly right, but not entirely. Yes, Redis is fast because it stores everything in RAM, and redirects in my service took less than 1 ms. However, users’ requests must still travel to and from the server, which often adds 50–100 ms or more to the response time. In most cases, Redis’ performance advantage is negligible to end-users.

Note: While response times impact user experience, they also affect the service itself. Faster response times reduce resource consumption, enabling higher requests per second (RPS).

The Storage

I’ll describe the steps I took during development. Some initial ideas weren’t optimal, and I’ll explain why.

Since I chose Redis, my options were limited to its data types. RAM is finite, so I decided to add a TTL (Time to Live) of 7 days for links, with flexibility to adjust later.

Initially, Redis’ String type seemed ideal. Keys like links:<id> could store target URLs as values. Commands like SET with NX (to prevent overwriting) and EX (to set TTL) provided a straightforward solution.

However, what if I wanted to store additional metadata in the future, such as a creation date or user ID? I could create separate keys (links:<id>:createdDate, links:<id>:userId, etc.), but managing TTL across multiple keys would become cumbersome and error-prone.

Thankfully, Redis’ Hash data type solved this issue. With hashes, I could store all link-related data under one key (links:<id>:meta) and set fields with HSET. TTL could be applied to the entire key using EXPIRE, ensuring consistency.

But there was a catch: Redis hashes lack an NX option to prevent collisions. Checking for key existence with EXISTS before HSET isn’t reliable in concurrent environments. A distributed lock could help, but it would reduce write throughput.

Even without high throughput requirements, I needed to plan for potential load increases.

Redis 7.4 introduced a better solution: the HEXPIRE command. It allows TTL to be set on individual hash fields, which was perfect for this use case.

Here’s my final approach:

Generate an ID and attempt to save the ID-URL pair in an index hash (links:index) using HSETNX links:index <id> <target>. If the result is 0, there’s a collision.
If successful, set metadata using HSET links:<id>:meta key1 value1 key2 value2 ....
Set TTL for the <id> field in links:index using HEXPIRE links:index <id> <ttl>, and for the metadata with EXPIRE links:<id>:meta <ttl>.

Steps 2 and 3 can be pipelined to minimize round-trips to Redis.

Data Model

links
  | index - HASH
      | <id> -> target URL
  | <id>
      | meta - HASH
          | targetUrl -> target URL
          | createdAt -> created at date
          | validUntil -> valid until date

Conclusion

Today, we began engineering a new URL shortener and discussed one critical component: link storage.

If this project interests you, the next part will explore ID generation and possibly start writing code.

I’d love to hear your thoughts and feedback on this project! The service is currently in alpha, but feel free to try it out.

The source code is available on GitHub: tnfy-link. Documentation will be updated as the service approaches beta.

Thanks for reading!

SMS Gateway for Android™: A Simple and Private SMS Gateway for Your Projects

capcom6 — Fri, 20 Dec 2024 04:22:48 +0000

In the world of digital communications, SMS remains a vital channel for authentication, notifications, and instant alerts. SMS Gateway is a solution that allows you to quickly and easily integrate SMS functionality into your projects by using an Android smartphone as a gateway.

It was created by a developer, for developers, with a focus on ease of use, data privacy, and the principles of open-source software. There’s no UI—just an API. What else could a developer need?

From Idea to Implementation

SMS Gateway started as a personal project but quickly found demand among other users. Today, its primary goals are simplicity and privacy. These goals are achieved while keeping costs minimal and ensuring the product remains completely free and open source.

Three Modes of Operation

The app operates in three modes, all sharing the same API:

Local

REST API runs directly on the Android device, enabling operation without the need for an external server. This mode allows entirely offline operation within a local network, ensuring enhanced privacy without requiring internet access.
Cloud

If direct access to the device is not possible, requests are routed through a shared server. This mode is free to use unless message volumes impact other users. For enhanced privacy, end-to-end encryption is available.
Private

For maximum security, you can deploy your own server instance. In this mode, your messages never leave your infrastructure. While push notifications still rely on the Cloud server, no message content is ever shared.

And because the app is open source, you can even build your own version of the service to suit your specific needs.

Webhooks for Incoming Messages

SMS Gateway can be used not only to send messages but also to receive incoming SMS messages directly to your backend. Webhooks make it easy to process incoming messages in real time without storing them on third-party servers.

How It Works

When an SMS is received, the app sends a POST request to the configured webhook URL. The payload includes details like:

Sender’s phone number
Message content
Timestamp

Developers can use this to process responses, trigger automated workflows, or store the data securely.

Learn more in the webhook documentation.

Why Choose SMS Gateway?

Here are several key advantages that make SMS Gateway stand out among competitors:

No Registration: You don’t need to provide an email or phone number to start using the application.
Privacy:
- Supports end-to-end encryption, private servers, and webhooks sent directly from the device for maximum privacy.
- All message data on the Cloud server is hashed within 15 minutes after the app acknowledges receipt of the queued message.
Simple Setup:
- Start with the Cloud server—all you need is an Android device running the app.
- Deploy a Private server with a single command using Docker if you need more privacy.
Webhooks Support: Real-time delivery of incoming SMS directly to your system with privacy-first features.
Multiple SIMs Support: Use devices with multiple SIM cards to manage different numbers easily. You can specify SIM usage explicitly or enable the SIM rotation feature.
Developer Tools:
- A CLI utility for sending SMS directly from the command line.
- Libraries for popular programming languages: Go, Python, JavaScript/TypeScript, PHP.

And all of this is completely free.

Technical Stack

SMS Gateway is built using:

Go for the backend
Kotlin for the Android app
MariaDB for data storage
Docker Swarm for orchestration
Terraform for infrastructure automation

Usage Examples

REST API Example

Send an SMS using the local REST API:

curl -X POST -u <username>:<password> \
  -H "Content-Type: application/json" \
  -d '{ "message": "Hello, doctors!", "phoneNumbers": ["+19162255887", "+19162255888"] }' \
  https://api.sms-gate.app/3rdparty/v1/message

Replace <username> and <password> with the credentials available on the Home tab of the Android app.

CLI Utility Example

Send SMS using the CLI utility:

# Send a message
smsgate send --phone '+19162255887' 'Hello, Dr. Turk!'

# Send a message to multiple numbers
smsgate send --phone '+19162255887' --phone '+19162255888' 'Hello, doctors!'
# or
smsgate send --phones '+19162255887,+19162255888' 'Hello, doctors!'

Who Is This For?

SMS Gateway is ideal for:

Small and medium businesses needing quick and affordable SMS integration.
Developers seeking a flexible, secure, and open-source solution.
Privacy enthusiasts who want to avoid relying on third-party cloud services.

Limitations

The app is not for you if your goal is to send large batches of messages, such as for advertisements or spam campaigns. Although it includes features to mitigate mobile operator restrictions, SMS Gateway is designed for legitimate, low-volume use cases. For bulk messaging, consider specialized services.

Conclusion

SMS Gateway prioritizes user privacy and convenience. With no registration required, flexible configuration options, and completely free access, it’s a reliable solution for any SMS-related tasks.

With its seamless setup, focus on privacy, robust webhook support, and open-source principles, SMS Gateway is the ultimate tool for integrating SMS functionality into your applications.

Ready to try it out? Start now: SMS Gateway or contact me via comments.