URL Expansion with Dart

A few months ago, I was proud to announce during a team meeting that I had successfully deployed the deeplinking service for a Flutter app we were developing. However, just a few days later, I was jolted awake in the middle of the night by a flurry of Slack alerts. The deeplinking service was down, and customers couldn't pay using our payment links! Although I had thoroughly tested the service, I hadn't accounted for one thing: our recent switch to using shortened URLs through an Iterable SMS campaign.

In this article, I will share the solution that finally fixed the deeplinking service's shortcomings. We'll dive into how we developed a URL expander app using Dart's HttpClient class, making our service more robust and reliable. First, we'll outline the app's structure, explaining its purpose and how it works. Then, we'll provide some practical usage examples to give you a clearer understanding of its capabilities. We'll also identify some initial limitations that we needed to address to make the tool more effective.

Next, we'll go through the steps of unit testing the app, ensuring that it meets our requirements and behaves as expected. Unit tests are crucial for confirming that the code functions properly, especially when changes are made in the future. Through this journey, we aim to provide you with a comprehensive understanding of how to tackle issues like the one we faced with deeplinking, thereby improving your own development practices.

What is a URL expander?

We are likely very familiar with URL shortening services like bit.ly which allow for users to generate shorter (sometimes custom, meaningful) URLs from long and potentially unwieldy URLs. However the task of resolving the original links is trivialised by modern browsers, which often resolve to original urls automatically, thus little attention is often paid to URL expansion.

However, in non-browser environments, like a Flutter mobile app, automatic URL resolution isn't a given. In these contexts, URL expanders become essential. You can think of a URL expander as a tool or service that resolves a shortened URL back to its original, longer form. In this article, we'll walk you through how to implement such a URL expander.

The URL Expander

Our URL Expander is implemented as a Dart CLI app exposing a utility via which shortened URLs may be expanded to their original form.

Overview of URL Expander App

The structure of the App patterns in agreement with the template for dart CLI apps as highlighted below.

.
├── CHANGELOG.md
├── README.md
├── analysis_options.yaml
├── bin
│   └── url_expander.dart
├── build
│   ├── test_cache
│   │   └── build
│   │       └── c075001b96339384a97db4862b8ab8db.cache.dill.track.dill
│   └── unit_test_assets
│       ├── AssetManifest.bin
│       ├── AssetManifest.json
│       ├── FontManifest.json
│       ├── NOTICES.Z
│       └── shaders
│           └── ink_sparkle.frag
├── coverage
│   ├── coverage.json
│   └── lcov.info
├── lib
│   └── url_expander.dart
├── pubspec.lock
├── pubspec.yaml
├── test
│   └── url_expander_test.dart
└── tree.txt

9 directories, 17 files

Steps to Url Expansion

1. Get input URL (shortened).
2. Get input base URL for expected expanded URL.
3. Validate input URL.
4. IF input URL is invalid throw an exception; else
5. Open a HTTP HEAD request to fetch headers associated with the URL
6. Close request to receive resolved headers.
7. Extract location header from the response
8. IF location header is not found then throw an exception; else
9. Validate the location against the expected base URL for the expansion of input URL
10. IF validation fails for location but there is a redirect in the response; recursively goto 1; decrease times we can goto 1; else
11. IF validation fails and there is no redirect throw an exception; else
12. IF validation is successful for location return it as expanded URL

The UrlExpander encapsulates these steps, the UrlExpander class is described in greater detail in the following section.

The UrlExpander class

The UrlExpander class is the central functional unit of our app. The UrlExpander class exposes a public method getFullUrl, a wrapper to the encapsulated _extractFullUrl method. The UrlExpander class also encapsulates the _client variable (it's HttpClient instance). The _isValidUrl method and the _isValidFullUrl method, which serve as checks and guards for licit expansions.

import 'dart:developer';
import 'dart:io';

class UrlExpander {
  final HttpClient _client = HttpClient();

  Future<String> getFullUrl(
    String url, {
    required String expandedBaseUrl,
  }) async {
    try {
      return await _extractFullUrl(url, expandedBaseUrl);
    } catch (e, t) {
      log('$e', error: e, stackTrace: t);
      throw Exception(e);
    }
  }

  _extractFullUrl(
    String url,
    String expandedBaseUrl, [
    int iterationsAllowed = 5,
  ]) async {
    try {
      if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
      if (!_isValidUrl(url)) throw "Invalid URL";

      final uri = Uri.parse(url);

      final request = await _client.headUrl(uri);
      request.followRedirects = false;

      final response = await request.close();

      stdout.write(
          "\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");

      final fullUrl = response.headers.value(HttpHeaders.locationHeader);

      if ((fullUrl ?? '').isEmpty) throw "URL not found";

      final isValidUrl = _isValidFullUrl(
        fullUrl,
        expandedBaseUrl,
      );

      if (!isValidUrl && response.isRedirect) {
        return await _extractFullUrl(
          fullUrl!,
          expandedBaseUrl,
          --iterationsAllowed,
        );
      }

      if (!isValidUrl) throw 'Cannot fetch expanded URL';

      return fullUrl!;
    } catch (_) {
      rethrow;
    }
  }

  bool _isValidUrl(String? url) {
    if (url != null && Uri.tryParse(url) != null) {
      final RegExp urlRegex = RegExp(
        r"\b[(http(s)?):\/\/(www\.)?a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)\b",
        caseSensitive: false,
      );

      return urlRegex.hasMatch(url);
    }

    return false;
  }

  bool _isValidFullUrl(String? url, String baseUrl) {
    if (url == null) return false;
    return url.startsWith(baseUrl);
  }
}

The _extractFullUrl method

  _extractFullUrl(
    String url,
    String expandedBaseUrl, [
    int iterationsAllowed = 5,
  ]) async {
    try {
      if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
      if (!_isValidUrl(url)) throw "Invalid URL";

      final uri = Uri.parse(url);

      final request = await _client.headUrl(uri);
      request.followRedirects = false;

      final response = await request.close();

      stdout.write(
          "\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");

      final fullUrl = response.headers.value(HttpHeaders.locationHeader);

      if ((fullUrl ?? '').isEmpty) throw "URL not found";

      final isValidUrl = _isValidFullUrl(
        fullUrl,
        expandedBaseUrl,
      );

      if (!isValidUrl && response.isRedirect) {
        return await _extractFullUrl(
          fullUrl!,
          expandedBaseUrl,
          --iterationsAllowed,
        );
      }

      if (!isValidUrl) throw 'Cannot fetch expanded URL';

      return fullUrl!;
    } catch (_) {
      rethrow;
    }
  }

The _extractFullUrl method is the central functional unit of the UrlExpander class. The method receives three arguments url, expandedBaseUrl and iterationsAllowed.

The url argument serves as the shortened URL input. The expandedBaseUrl serves as input to the associated _isValidFullUrl method (along with the url input), ensuring that only full urls that match the host expected for the expanded shortened URL are marked as valid. The iterationsAllowed is primarily given in recursive calls to prevent runaway/infinite recursions.

The stepwise operations of the _extractUrl method are as follows;

1. WHEN recursively called it check that the recursive iteration is permitted in a guard; if not an exception is thrown.
2. It checks that the input URL is valid in a guard else it throws an exception; then.
3. It generates Uri object instance from the input URL
4. It creates a HttpClientRequest instance request from an HTTP HEAD request on the generated Uri.
5. We disable automatic redirects on the request instance.
6. We create a HttpClientResponse instance from the result of closing the request.
7. We print the response header in green.
8. We extract the fullUrl string from the location header
9. IF fullUrl is null or empty we throw an exception; else
10. We validate fullUrl against its expected base URL denoted by the expandedBaseUrl argument; then
11. IF fullUrl is invalid but response has a redirect, we recursively call _extractFullUrl, reducing the value of the iterationsAllowed argument; else
12. IF fullUrl is invalid but there is no redirect in the response we throw an exception; else;
13. We return fullUrl as the output.

With these operations stated, we find it necessary to highlight the use of the CLI, this is shown below.

Current gaps of the _extractFullUrl method

The current iteration of the _extractFullUrl method while apparently well forms, has a shortcoming. It fails to parse full URL inputs correctly, as full URLs may not return the location header from a HEAD request. This is highlighted below:

We can introduce a guard to force an early exit from the method when a valid full URL is input as shown below.

  _extractFullUrl(
    String url,
    String expandedBaseUrl, [
    int iterationsAllowed = 5,
  ]) async {
    try {
      if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
      if (!_isValidUrl(url)) throw "Invalid URL";
      if (_isValidFullUrl(url, expandedBaseUrl)) return url;

      final uri = Uri.parse(url);

      final request = await _client.headUrl(uri);
      request.followRedirects = false;

      final response = await request.close();

      stdout.write(
          "\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");

      final fullUrl = response.headers.value(HttpHeaders.locationHeader);

      if ((fullUrl ?? '').isEmpty) throw "URL not found";

      final isValidUrl = _isValidFullUrl(
        fullUrl,
        expandedBaseUrl,
      );

      if (!isValidUrl && response.isRedirect) {
        return await _extractFullUrl(
          fullUrl!,
          expandedBaseUrl,
          --iterationsAllowed,
        );
      }

      if (!isValidUrl) throw 'Cannot fetch expanded URL';

      return fullUrl!;
    } catch (_) {
      rethrow;
    }
  }

Consequently the behaviour of the method changes, to return the valid full URL immediately as shown below.

Testing the url expander

Unit tests were written to ensure that the url expander functions as expected in a plethora of scenarios, a presentation of these tests in action is presented below.

The source code for these tests may be found here.

This article made a presentation on the implementation of a URL expander CLI app in Dart. I would like to thank Kenneth Ngedo and Samuel Abada for their reviews on earlier drafts of this article.

dirisujesse / url_expander

URL EXPANDER

A dart CLI application for fetching full urls from short URLs

Usage Example

View on GitHub