DEV Community: Samuel Mugane

Syncing AWS S3 Storage Bucket Objects to LocalStack S3 Storage Bucket

Samuel Mugane — Thu, 27 Mar 2025 12:06:46 +0000

Banner image courtesy: Tudip Digital

Getting Started

LocalStack is a cloud service emulator that runs in a single container on your laptop or in your CI environment. With LocalStack, you can run your AWS applications or Lambdas entirely on your local machine without connecting to a remote cloud provider.

Installation

In this tutorial, we will install LocalStack CLI on our x86-64 Linux machine, for ARM64 kindly check the official documentation. The CLI starts and manages the LocalStack Docker container.

# Download the pre-built binary
curl --output localstack-cli-4.2.0-linux-amd64-onefile.tar.gz \
    --location https://github.com/localstack/localstack-cli/releases/download/v4.2.0/localstack-cli-4.2.0-linux-amd64-onefile.tar.gz

# Extract the LocalStack CLI from the terminal
sudo tar xvzf localstack-cli-4.2.0-linux-*-onefile.tar.gz -C /usr/local/bin

# Verify installed version
localstack --version
4.2.0

Configuring and Starting LocalStack

We need an API key from LocalStack to configure LocalStack on our local machine. Head over to LocalStack and signup. Within your LocalStack dashboard, go to WorkSpace → Auth Tokens and copy the personal auth token provided. This personal auth token will be used to activate the LocalStack pro hobby subscription which grants use access to all of LocalStack’s features such as the Resource Browser which will use in this tutorial. Now add the auth token to your local stack CLI and start LocalStack:

localstack auth set-token <YOUR_AUTH_TOKEN>
# By default, LocalStack runs on port 4566. Make sure no service is running on this port otherwise LocalStack won't start
localstack start

     __                     _______ __             __
    / /   ____  _________ _/ / ___// /_____ ______/ /__
   / /   / __ \/ ___/ __ `/ /\__ \/ __/ __ `/ ___/ //_/
  / /___/ /_/ / /__/ /_/ / /___/ / /_/ /_/ / /__/ ,<
 /_____/\____/\___/\__,_/_//____/\__/\__,_/\___/_/|_|

- LocalStack CLI: 4.2.0
- Profile: default
- App: https://app.localstack.cloud

[12:31:41] starting LocalStack in Docker mode 🐳                                                       localstack.py:512
2025-03-27T12:31:41.282  INFO --- [  MainThread] l.p.c.b.licensingv2        : Successfully activated cached license <license-id>:hobby from /home/user/.cache/localstack-cli/license.json 🔑✅
──────────────────────────────────── LocalStack Runtime Log (press CTRL-C to quit) ─────────────────────────────────────

LocalStack version: 4.2.1.dev87
LocalStack build date: 2025-03-26
LocalStack build git hash: c6fc9b010

2025-03-27T09:31:48.092  INFO --- [  MainThread] l.p.c.b.licensingv2        : Successfully activated cached license <license-id>:hobby from /etc/localstack/conf.d/license.json 🔑✅
2025-03-27T09:31:52.329  INFO --- [  MainThread] l.p.c.extensions.platform  : loaded 0 extensions
Ready.

Installing AWS CLI and LocalStack’s AWS CLI

We now need to have both the AWS official CLI and LocalStack’s AWS CLI which is a thin wrapper of the AWS official CLI for LocalStack.

AWS CLI installation

# Verify installation of the official AWS CLI
aws --version
aws-cli/2.25.4 Python/3.12.9 Linux/5.15.167.4-microsoft-standard-WSL2 exe/x86_64.ubuntu.24

# Install LocalStack's AWS CLI
pip install awscli-local
awslocal --version
aws-cli/2.25.4 Python/3.12.9 Linux/5.15.167.4-microsoft-standard-WSL2 exe/x86_64.ubuntu.24

Creating S3 Storage Bucket in LocalStack

Before syncing the buckets, we need to create an S3 storage bucket within LocalStack in our local machine:

# Create bucket
awslocal s3api create-bucket \
  --acl public-read \
  --bucket test-s3-bucket \
  --object-ownership BucketOwnerPreferred \
  --region us-east-2 \
  --create-bucket-configuration '{"LocationConstraint": "us-east-2"}' 

# Sample output with bucket name set to test-s3-bucket
{
    "Location": "http://test-s3-bucket.s3.localhost.localstack.cloud:4566/"
}

Syncing S3 Storage Buckets

In this tutorial, we will focus on two keys methods for syncing our AWS S3 storage bucket with our LocalStack S3 storage bucket within our local machine.

Method 1: Sequential sync

The sequential sync approach is whereby we sync the bucket to a local temp folder in our local machine then sync our LocalStack S3 storage bucket with data in the local temp folder.

# Syncing bucket data to local temp folder
aws s3 sync s3://<source-bucket i.e test-s3-bucket> ./local-folder --region <region i.e us-east-2>

# Syncing local folder to LocalStack S3 bucket
aws s3 sync ./local-folder s3://<destination-bucket i.e test-s3-bucket> --endpoint-url http://localhost:4566 

# Verify upload
awslocal s3 ls s3://test-s3-bucket
2025-03-27 13:04:21     165892 test-s3-file.jpeg

Method 2: Direct Sync (Without Local Storage)

The direct sync approach avoids syncing the bucket data to local storage first but rather syncs the bucket data directly to the LocalStack S3 storage bucket. In this tutorial, we will use a bash script to sync the buckets. You can create a python script and use boto3 and LocalStack’s python client to sync the buckets.

aws s3 ls s3://<aws-s3-bucket> --recursive | awk '{print $4}' | \
xargs -P 10 -I {} sh -c 'aws s3 cp --debug "s3://<aws-test-s3-bucket>/{}" - | awslocal s3 cp - "s3://<localstack-test-s3-bucket>/{}" --debug'

# Sample output in terminal running localstack
2025-03-27T10:12:24.654  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.PutObject => 200
2025-03-27T10:12:27.443  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.PutObject => 200
2025-03-27T10:12:37.219  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.PutObject => 200
2025-03-27T10:12:41.503  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.PutObject => 200
2025-03-27T10:12:45.346  INFO --- [et.reactor-0] localstack.request.aws     : AWS s3.PutObject => 200

After the upload is complete, you can head to your LocalStack dashboard navigate to local instances, go to resource browser, search for S3, choose bucket and browse your files or folders in your bucket.

Conclusion

The local storage first approach is more performant as compared to the direct approach since the direct approach relies on xargs to achieve parallel transfers using sub-processes. One can spawn multiple separate sub processes using xargs -P N when N is the number of sub processes required. The sub process parallelism is great for S3 sync as this is an I/O bound task. In terms of speed, using xargs even with tuned parallelism can be slow. If you are looking to speed up syncs, you can use tools such as s5cmd which supports true multithreading. I will sure do a tutorial on s5cmd and do a benchmark to check how it compares to using xargs in parallel mode. The approach in this tutorial was not created with the context of large S3 storage buckets. If you have a tutorial on this please share. Thanks again for reaching to the end of this tutorial. Cheers!

Web Authentication on Stellar: A Deep Dive

Samuel Mugane — Fri, 25 Aug 2023 03:04:37 +0000

Banner copyright: Gemini

Introduction

The Stellar Web Authentication also referred to as SEP-0010 is based on JSON Web Token (JWT) RFC7519 which is the most used self-contained way for securely transmitting information between parties as a JSON object. One of the use cases for JWT is authentication which is the focus of this article. Stellar Web Authentication uses a wallet's keypair which comprises the public key: G... and the secret key: S... as base credentials for authentication.

Stellar Web Authentication

This walk-through comprises different components and keywords that will be highlighted earlier for ease of understanding.

Note: This implementation of SEP-0010 does not feature client attribution which entails the verification of the client_domain and verify_challenge_transaction_signed_by_client_master_key operation.

Assuming you have already set up your Django project, let's go ahead and define an app that will handle our auth:

python manage.py startapp sep10

Add the created app to the installed apps list in the settings.py file in the root folder of your project:

INSTALLED_APPS = [
    "sep10",
]

Define the variables below in the settings.py file in the root folder of your project:

from stellar_sdk import Server

STELLAR_NETWORK = "Test SDF Network ; September 2015"
HORIZON_URI = "https://horizon-testnet.stellar.org"
SERVER_JWT_KEY = "key_used_to_hash_the_jwt_str"
HORIZON_SERVER = Server(horizon_url=HORIZON_URI)
SIGNING_KEY = "server_account_public_key"
SIGNING_SEED = "server_account_secret_seed"
HOST_URL = "http://localhost:8000"
SEP10_HOME_DOMAINS="foo.com, bar.io"

Define a custom JWT token class

In the sep10 app folder add the JWT token class below in the token.py file:

from jwt import decode
from datetime import datetime, timezone
from jwt.exceptions import InvalidTokenError
from typing import Optional, Union, Dict

from stellar_sdk import Keypair, MuxedAccount
from stellar_sdk.strkey import StrKey
from stellar_sdk.exceptions import (
    Ed25519PublicKeyInvalidError,
    MuxedEd25519AccountInvalidError,
)
from project_name import settings # replace project_name with actual Django project name


class SEP10Token:
    """An object that represents the authenticated sessions of the client."""

    _CLAIMS = {"iss", "sub", "exp", "iat"}

    def __init__(self, jwt: Union[str, Dict]):
        if isinstance(jwt, str):
            try:
                jwt = decode(jwt, settings.SERVER_JWT_KEY, algorithms=["HS256"])
            except InvalidTokenError:
                raise ValueError("jwt token has expired or is invalid")

        elif not isinstance(jwt, Dict):
            raise ValueError(
                "invalid type for 'jwt' parameter: must be a string or dictionary"
            )

        if not self._CLAIMS.issubset(set(jwt.keys())):
            raise ValueError(
                f"jwt is missing one of the required claims: {', '.join(self._CLAIMS)}"
            )

        self.memo = None
        self.stellar_account = None

        if jwt["sub"].startswith("M"):
            try:
                StrKey.decode_muxed_account(jwt["sub"])
            except (ValueError, MuxedEd25519AccountInvalidError):
                raise ValueError(f"invalid muxed account: {jwt['sub']}")
        elif ":" in jwt["sub"]:
            try:
                self.stellar_account, self.memo = jwt["sub"].split(":")
            except ValueError:
                raise ValueError(f"improperly formatted 'sub' value: {jwt['sub']}")
        else:
            self.stellar_account = jwt["sub"]

        if self.stellar_account:
            try:
                Keypair.from_public_key(self.stellar_account)
            except Ed25519PublicKeyInvalidError:
                raise ValueError(f"invalid Stellar public key: {jwt['sub']}")

        if self.memo:
            try:
                int(self.memo)
            except ValueError:
                raise ValueError(
                    f"invalid memo in 'sub' value, expected 64-bit integer: {self.memo}"
                )

        try:
            iat = datetime.fromtimestamp(jwt["iat"], tz=timezone.utc)
        except (OSError, ValueError, OverflowError):
            raise ValueError("invalid iat value")
        try:
            exp = datetime.fromtimestamp(jwt["exp"], tz=timezone.utc)
        except (OSError, ValueError, OverflowError):
            raise ValueError("invalid exp value")

        now = datetime.now(tz=timezone.utc)
        if now < iat or now > exp:
            raise ValueError("jwt is no longer valid")

        self._payload = jwt

        @property
        def account(self) -> str:
            """
            The Stellar account (`G...`) authenticated. Note that a muxed account
            could have been authenticated, in which case `Token.muxed_account` should
            be used.
            """
            if self._payload["sub"].startswith("M"):
                return MuxedAccount.from_account(self._payload["sub"]).account_id
            elif ":" in self._payload["sub"]:
                return self._payload["sub"].split(":")[0]
            else:
                return self._payload["sub"]

        @property
        def muxed_account(self) -> Optional[str]:
            """
            The M-address specified in the payload's ``sub`` value, if present
            """
            if self._payload["sub"].startswith("M"):
                return self._payload["sub"]
            else:
                return None

        @property
        def memo(self) -> Optional[int]:
            """
            The memo included with the payload's ``sub`` value, if present
            """
            return (
                int(self._payload["sub"].split(":")[1])
                if ":" in self._payload["sub"]
                else None
            )

        @property
        def issuer(self) -> str:
            """
            The principal that issued a token, RFC7519, Section 4.1.1 — a Uniform
            Resource Identifier (URI) for the issuer
            (https://example.com or https://example.com/G...)
            """
            return self._payload["iss"]

        @property
        def issued_at(self) -> datetime:
            """
            The time at which the JWT was issued RFC7519, Section 4.1.6 -
            represented as a UTC datetime object
            """
            return datetime.fromtimestamp(self._payload["iat"], tz=timezone.utc)

        @property
        def expires_at(self) -> datetime:
            """
            The expiration time on or after which the JWT will not accepted for
            processing, RFC7519, Section 4.1.4 — represented as a UTC datetime object
            """
            return datetime.fromtimestamp(self._payload["exp"], tz=timezone.utc)

        @property
        def payload(self) -> dict:
            """
            The decoded contents of the JWT string
            """
            return self._payload

Define a custom authentication scheme

The authentication scheme inherits from the BaseAuthentication class and allows us to override the authenticate method that performs checks on the request object such as:

Verifying authorization header format in request i.e. Bearer <token>.
Verifying and decoding the JWT token provided in the authorization header.
Verifying that the credentials provided are tied to an existing User object assuming the User model contains the stellar account as a field.

In the sep10 app folder add the authentication scheme class below in the authentication.py file:

from django.contrib.auth import get_user_model
from rest_framework.request import Request
from rest_framework.authentication import BaseAuthentication
from rest_framework.exceptions import AuthenticationFailed as SEP10AuthenticationFailed
from sep10.token import SEP10Token

User = get_user_model()


class SEP10Authentication(BaseAuthentication):
    def authenticate(self, request: Request) -> tuple[User, SEP10Token]:
        """
        This function authenticates a user by verifying and decoding
        a JWT token from the authorization header of a request.

        :param request: The HTTP request object that contains information
                        about the incoming request, such as headers, query
                        parameters, and request body
        :return: a tuple containing the authenticated User object and
                 the SEP10 token object.
        """
        auth_header = request.headers.get("Authorization")

        if not auth_header:
            return None

        if not auth_header.startswith("Bearer "):
            raise SEP10AuthenticationFailed("Invalid auth header format.")

        jwt = auth_header.split(" ", 1)[1]

        token = SEP10Token(jwt)

        try:
            user = User.objects.get(stellar_account=token.stellar_account)
        except User.DoesNotExist:
            raise SEP10AuthenticationFailed(
                "No active account found with the given credentials."
            )

        return (user, token)

The custom authentication scheme can be set globally on the default authentication classes setting:

REST_FRAMEWORK = {
    "DEFAULT_AUTHENTICATION_CLASSES": ["sep10.authentication.SEP10Authentication",]
}

Or on a per-view or per-viewset basis, using the APIView class-based views:

from sep10.authentication import SEP10Authentication
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView

class ExampleView(APIView):
    authentication_classes = [SEP10Authentication]
    permission_classes = [IsAuthenticated]

    def get(self, request, format=None):
        content = {}
        return Response(content)

Or, if you're using the @api_view decorator with function-based views:

@api_view(['GET'])
@authentication_classes([SEP10Authentication])
@permission_classes([IsAuthenticated])
def example_view(request, format=None):
    content = {}
    return Response(content)

Define helper functions

These helper functions wrap the functionality of the SEP10 Authentication into tangible views or viewsets using the following operations: build_challenge_transaction, read_challenge_transaction and verify_challenge_transaction_threshold which can be found in detail in the stellar-sdk.

In the sep10 app folder add the helper functions below in the helpers.py file:

import os
import jwt
import logging
from urllib.parse import urlparse
from stellar_sdk.sep.stellar_web_authentication import (
    build_challenge_transaction,
    read_challenge_transaction,
    InvalidSep10ChallengeError,
    verify_challenge_transaction_threshold,
)
from stellar_sdk import MuxedAccount, exceptions
from project_name import settings


logger = logging.getLogger(__name__)


def _challenge_transaction(
    client_account_id,
    home_domain,
    client_domain=None,
    client_signing_key=None,
    memo=None,
):
    """
    This function generates a challenge transaction for a client account in Python, as per SEP 10.

    :param client_account_id: The Stellar account ID of the client requesting authentication
    :param home_domain: The domain of the Stellar account issuing the challenge transaction.
    This is typically the domain of the server that is providing the authentication service
    :param client_domain: The domain of the client requesting the challenge transaction. It is an
    optional parameter
    :param client_signing_key: The public key of the client account's signing key. This is used to
    verify the signature of the challenge transaction
    :param memo: A memo to include in the challenge transaction. This is an optional parameter and can
    be used to include additional information in the transaction
    :return: The XDR encoding of the challenge transaction for a client account.
    """
    return build_challenge_transaction(
        server_secret=settings.SIGNING_SEED,
        client_account_id=client_account_id,
        home_domain=home_domain,
        web_auth_domain=urlparse(settings.HOST_URL).netloc,
        network_passphrase=settings.STELLAR_NETWORK,
        client_domain=client_domain,
        client_signing_key=client_signing_key,
        memo=memo,
    )


def _generate_jwt(envelope_xdr: str) -> str:
    """
    Generates the JSON web token from the challenge transaction XDR.

    See: https://github.com/stellar/stellar-protocol/blob/master/ecosystem/sep-0010.md#token
    """
    try:
        challenge = read_challenge_transaction(
            challenge_transaction=envelope_xdr,
            server_account_id=settings.SIGNING_KEY,
            home_domains=settings.SEP10_HOME_DOMAINS,
            web_auth_domain=urlparse(settings.HOST_URL).netloc,
            network_passphrase=settings.STELLAR_NETWORK,
        )

    except (InvalidSep10ChallengeError, TypeError) as e:
        logger.error(f"Invalid challenge transaction: {e}")
        raise ValueError(f"Invalid challenge transaction: {e}")

    # extract the Stellar account from the muxed account to check for its existence
    stellar_account = challenge.client_account_id
    if challenge.client_account_id.startswith("M"):
        stellar_account = MuxedAccount.from_account(
            challenge.client_account_id
        ).account_id

    with settings.HORIZON_SERVER as server:
        try:
            account = server.load_account(stellar_account)
        except exceptions.NotFoundError:
            raise ValueError(f"Account {stellar_account} not found")

    signers = account.load_ed25519_public_key_signers()
    threshold = account.thresholds.med_threshold
    try:
        signers_found = verify_challenge_transaction_threshold(
            challenge_transaction=envelope_xdr,
            server_account_id=settings.SIGNING_KEY,
            home_domains=settings.SEP10_HOME_DOMAINS,
            web_auth_domain=urlparse(settings.HOST_URL).netloc,
            network_passphrase=settings.STELLAR_NETWORK,
            threshold=threshold,
            signers=signers,
        )
        logger.info(
            f"Generating SEP-10 token for account {challenge.client_account_id}"
        )
    except InvalidSep10ChallengeError as e:
        logger.error(f"Invalid challenge transaction: {e}")
        raise ValueError(f"Invalid challenge transaction: {e}")

    logger.info(
        "Challenge verified using account signers: "
        f"{[s.account_id for s in signers_found]}"
    )

    # set iat value to minimum timebound of the challenge so that the JWT returned
    # for a given challenge is always the same.
    # https://github.com/stellar/stellar-protocol/pull/982
    issued_at = challenge.transaction.transaction.preconditions.time_bounds.min_time

    # format sub value based on muxed account or memo
    if challenge.client_account_id.startswith("M") or not challenge.memo:
        sub = challenge.client_account_id
    else:
        sub = f"{challenge.client_account_id}:{challenge.memo}"

    jwt_dict = {
        "iss": os.path.join(settings.HOST_URL, "auth"),
        "sub": sub,
        "iat": issued_at,
        "exp": issued_at + 24 * 60 * 60,
        "jti": challenge.transaction.hash().hex(),
    }
    return jwt.encode(jwt_dict, settings.SERVER_JWT_KEY, algorithm="HS256")

Define views to handle the authentication flow

The views will be defined based on the auth flow below:

Going with the single responsibility principle, we'll separate the auth flow into three separate class-based views; GetChallengeTransaction, SignChallengeTransaction and GetJWTToken

In the sep10 app folder add the view-based classes below in the views.py file:

import logging

from rest_framework.request import Request
from rest_framework.response import Response
from rest_framework.views import APIView

from rest_framework import status
from stellar_sdk import Keypair, exceptions, helpers
from sep10.helpers import _challenge_transaction, _generate_jwt
from project_name import settings

logger = logging.getLogger(__name__)


class GetChallengeTransaction(APIView):
    def get(self, request: Request) -> Response:
        with settings.HORIZON_SERVER as server:
            client_account = request.query_params.get("client_account")
            home_domain = request.query_params.get("home_domain")

            try:
                server.load_account(client_account)
            except exceptions.NotFoundError:
                return Response(
                    {"error": "Client account not found."},
                    status=status.HTTP_404_NOT_FOUND,
                )

            if home_domain not in settings.SEP10_HOME_DOMAINS:
                return Response(
                    {"error": "invalid 'home_domain' value."},
                    status=status.HTTP_400_BAD_REQUEST,
                )
            try:
                transaction = _challenge_transaction(client_account, home_domain)
            except ValueError as e:
                return Response({"error": str(e)}, status=status.HTTP_400_BAD_REQUEST)

            logger.info(f"Returning SEP-10 challenge for account {client_account}")

            return Response(
                {
                    "transaction": transaction,
                    "network_passphrase": settings.STELLAR_NETWORK,
                },
                status=status.HTTP_200_OK,
            )


class SignChallengeTransaction(APIView):
    def post(self, request: Request) -> Response:
        with settings.HORIZON_SERVER as server:
            transaction = request.data.get("challenge_transaction_xdr")
            client_account_seed = request.data.get("client_account_seed")
            signer_keypair = Keypair.from_secret(client_account_seed)
            source = signer_keypair.public_key

            # load source account. fail-safe if source account is not found
            try:
                server.load_account(source)
            except exceptions.NotFoundError:
                return Response(
                    {"error": "Client account not found."},
                    status=status.HTTP_404_NOT_FOUND,
                )

            # sign challenge transaction
            try:
                # Parse xdr obj to transaction envelope
                envelope = helpers.parse_transaction_envelope_from_xdr(
                    transaction, settings.STELLAR_NETWORK
                )
                envelope.sign(signer_keypair)
                logger.info(f"Signed SEP-10 challenge for client account {source}")
                return Response(
                    {
                        "transaction": envelope.to_xdr(),
                        "network_passphrase": settings.STELLAR_NETWORK,
                    },
                    status=status.HTTP_200_OK,
                )
            except (exceptions.SignatureExistError, ValueError) as e:
                logger.error("%s", str(e))
                return Response(
                    {"error": str(e)}, status=status.HTTP_400_BAD_REQUEST
                )

class GetJWTToken(APIView):
    def post(self, request: Request) -> Response:
        transaction = request.data.get("signed_challenge_transaction_xdr")
        # Parse xdr obj to transaction envelope to validate
        try:
            helpers.parse_transaction_envelope_from_xdr(
                transaction, settings.STELLAR_NETWORK
            )

        except ValueError as e:
            logger.error("%s", str(e))
            return Response(
                {"error": str(e)}, status=status.HTTP_400_BAD_REQUEST
            )

        try:
            jwt_token = _generate_jwt(transaction)
            return Response(
                {
                    "token": jwt_token,
                },
                status=status.HTTP_200_OK,
            )
        except Exception as e:
            logger.error("%s", str(e))
            return Response({"error": str(e)}, status=status.HTTP_400_BAD_REQUEST)

Finally, in the urls.py file in the sep10 app folder add the following paths:

from sep10 import views
from django.urls import path


app_name = "sep10"

urlpatterns = [
    path(
        "challenge",
        views.GetChallengeTransaction.as_view(),
        name="sep10-challenge",
    ),
    path(
        "sign",
        views.SignChallengeTransaction.as_view(),
        name="sign-challenge",
    ),
    path(
        "token",
        views.GetJWTToken.as_view(),
        name="jwt-token",
    ),
]

add sep10 app URLs in project_name/urls.py file:

from django.urls import include, re_path

urlpatterns = [
    re_path(r"^v1/auth/", include("sep10.urls", namespace="v1:sep10")),
]

Response snippets:

GET auth/challenge

 data: {
    "transaction": "AAAAAgAAAAAOzp8SYe...",
    "network_passphrase": "Test SDF Network ; September 2015"
  },

POST auth/sign

 data: {
    "transaction": "AAAAAgAAAAAOzp8SYe...",
    "network_passphrase": "Test SDF Network ; September 2015"
  },

POST auth/token

 data: {
    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...."
  },

Conclusion

The Stellar network boasts ecosystem proposals that depict engineering marvels and Stellar Web Authentication (SEP-0010) is one of them, you can find other ecosystem proposals in detail here. Thank you for reading to the end of this article. You can find a live implementation of the SEP-0010 protocol here

Resources

Get started with JWT

Django Polaris

What happens when you type a URL on the browser and press enter

Samuel Mugane — Fri, 12 May 2023 11:24:12 +0000

Overview

Understanding how the web works is a great start to building web stacks that align with the principles associated with the web. Hence, in this article, the focus will be to debunk what happens when you request a specific URL through the browser. The article goes into detail in explaining the various components of the URL, communication protocols used on the web, and server components that are involved in processing the request.

The Flow of the request

1. Typing the URL and pressing enter

When you type the URL, it is usually characterized by various components separated by special characters such as semi-colons, backward slashes, or periods. The URL is composed of the scheme i.e. https://, the domain i.e. google.com, the path i.e. /maps, and the resource i.e. {long, lat}.

The Scheme

The scheme https:// stands for HyperText Transfer Protocol Secure. Its an extension of HTTP, an application protocol (OSI 7) built on TCP as the transport layer (OSI 3). This scheme tells the browser to make a connection to the server using TLS (Transport Layer Security). TLS is a secure successor to SSL (Secure Sockets Layer). SSL is a technology that clients and servers rely upon to communicate in a secure manner using encrypted communication channels over any network. To set up the encrypted communication channel, the client and the server perform a handshake facilitated through digital certificates known as `TLS/SSL certificates.
Domain

The component google.com is the domain name. A domain name is a human-readable address that points to a specific IP (Internet protocol) address. Domain names make it easier for clients to access resources across the web. It is hard to memorize an IP address in comparison to a domain name. IP addresses are versioned where we have IPV4 the fourth version of the Internet Protocol made up of 32 bits, divided into four 8-bit segments i.e. 8.8.8.8/8.8.4.4 for google.com. IPV4 is only limited to 4.3B unique addresses. There is also IPV6, the sixth version of the Internet Protocol made up of 128-bit addresses that allow for trillions of unique addresses i.e. 2001:4860:4860::8888 for google.com.
Path
A path i.e. /maps points a resource that the user wants to access on the site visited. The resource might be static content i.e. HTML, CSS or JavaScript or dynamically generated content i.e. reviews.
Resource
The resource {long, lat} represents a place within the maps resource path provided by Google in the maps service. The resource might be in the form of static or dynamic content.

2. Browser resolves the domain to its unique IP

When the URL is accessed, the browser first makes a DNS request to the DNS server where the domain in the URL is resolved and an IP address is returned as a response.

DNS (Domain Name System)

A database system that maps domain names to IP addresses, allowing browsers to request specific websites and servers to communicate with each other. It is a high-level abstraction that abstracts the complexity of IP addresses.
The lookup process is shown below:

3. Browser initiates a `TCP` connection with the server

Transmission Control Protocol (TCP) is a communications protocol that enables reliable and ordered delivery of data packets between two hosts i.e. the client and the server. The browser uses the IP address from the DNS server to make the TCP connection using a 3-way handshake shown below:

The communication channel establish between the client and the server relies on TLS to encrypt all data packets shared between the two hosts.

4. Browser sends the HTTP request to the server.

After the encrypted communication channel is established, the browser can now send a HTTP request to the server and receive a response safely. A HTTP request is a message sent by a client to a server to request access to some resource. A HTTP request is made up of various components including a request line, request headers, and message body (optional).

Request line

The request line contains the method i.e. GET, POST, PUT, or DELETE. It also contains the path i.e. / depicting the root path and the version of HTTP being used to make the request i.e. HTTP/1.1.
Request headers

The request provides additional information such as the browser making the request i.e User-Agent: Chrome/68.0.3440.106, the type of content the server accepts i.e Content-Type: application/json to show the server accepts content in JSON format, or the host the request is being made to i.e Host: google.com.
Message body

The message body represents data being sent to the server and it is used with request methods such as POST and PUT.

Example of a HTTP request:
GET / HTTP/1.1 Host: google.com User-Agent: Chrome/68.0.3440.106 Accept: text/html Accept-Language: en-US Accept-Encoding: gzip Connection: keep-alive Content-Type: multipart/form-data Content-Length: 256

5. Server receives request, processes the request and returns a response

The HTTP request goes through the Web Application Firewall (WAF) which is a component of the web stack that protects the web application from malicious HTTP traffic by analyzing and filtering the traffic.

The HTTP request then goes through a load balancer which is another component within the web stack that ensures traffic is distributed appropriately to the web server(s). The load balancer is also used to terminate TLS/SSL. HAProxy is one of the most used load balancers.

The web server, a software that serves static or dynamic content to the web receives the request and processes it. It ensures that all the required headers are in the request and that the resource path exists. If the request is for static content i.e. GET /index.html, the web server will look up the .html file within the application code. If the content is dynamic and requires any business logic to generate, the web server communicates with the application server. Examples of web servers include NGINX and Apache.

An application server is a software framework that handles business logic, database transactions, and communication between applications and users. The application server resolves the request for dynamic content and returns a response to the web server.

The web server returns the content as a HTTP response back to the client (browser). The HTTP response is made up of various components including the status line, response headers, and message body.

Status line

The status line contains the version of HTTP used i.e HTTP/1.1 and the status code of the request i.e 200 OK
Response headers

The response headers can include: Server: Apache/2.4.46, Content-Length: 1024, or Content-Type: text/html.
Message body

The message body contains the content returned by the web server i.e. a text or a JSON response.

Example of a HTTP response

`
HTTP/1.1 200 OK
Date: Fri, 12 May 2023 01:11:12 GMT
Server: Nginx/1.24
Last-Modified: Thur, 11 May 2023
ETag: "0-23-4024c3a5"
Accept-Ranges: bytes
Content-Length: 42
Connection: close
Content-Type: text/html

Holberton School

6. Browser renders the content

The browser receives the response from the server and checks the response headers to know how to render the content. for example, if the Content-Type header is text/html, the browser knows that it will render a html file to the user. If the Content-Type is application/json the browser will parse the JSON data into an object and wrap the object in a HTML element and render it. The rendering of the content is done through the DOM, a high-level Web API provided by the browser to efficiently render a webpage.

Closing

Understanding the concepts outlaid above helps assess avenues for improving the reliability, performance, and security of the web.

Caching Django Application using Redis

Samuel Mugane — Tue, 31 May 2022 18:36:44 +0000

Banner Source

Introduction

When building web applications, optimization is key for user experience. In most cases, requests made in the client-side prompt the back-end to communicate with the database.The back-end retrieves the data from the database and relays it to the client. In a normal use case, the client will prompt the server multiple times. Hence, multiple hits are made by the back-end to the database to retrieve data and process it to the client.

In Django, this can be a performance trade-off. Caching can help eliminate the performance trade-off by storing copies of data where it can be accessed more quickly in the client-side when the same request is made multiple times. A browser cache comes into play when handling client-side requests. There are more optimization tricks that one can implement as discussed in the official documentation

A visual of Browser Cache functionality

Image Source: Pressidium

Learn more about the browser cache functionality here

Getting Started

To implement caching in Django application. First, set up your Django project and application. Then install django-redis, a project by Jazzband:

pip install django-redis

In settings.py file, add the following configuration:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {  
          "CLIENT_CLASS":"django_redis.client.DefaultClient",         
        }
    }
}

You can add more configurations as provided in the package documentation.

Implementing caching on views

In this case, the caching functionality will be implemented to a view in an e-commerce API I developed, you can check it here

from rest_framework.views import APIView
from django.core.cache import cache

class ProductDetailView(APIView):

   def get(self, request, *args, **kwargs):
       productID = kwargs['pk']
       if cache.get(productID):
          product = cache.get(productID)
          print("Cache hit")
       else:
           try:
             product = Product.objects.get(pk=productID)
                cache.set(productID, product)
                print("Cache miss")
           except Product.DoesNotExist:
                return Response(status=404)
       serializer = ProductSerializer(product, many=False)
       return Response(serializer.data)

In the code above, when a request is made to the ProductDetailView route, the database will be prompted to retrieve a product based on the productID. When the same request is made, the console will print out Cache hit and use the data stored prior to the request to serve to the client.

When the request is new in the subsequent queue, the console will print out Cache miss since no data regarding the request was cached prior to the request being made. Hence, the database will be prompted again and the back-end will process the new request. Otherwise, a code 404 will be raised to signal that the Product Model doesn't exist; hence, no data is retrievable.

Final Note

In production, the cache configuration in the settings.py file will require changes depending on the cache service used.
The example below is deployment on heroku:

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": os.getenv('REDIS_URL'),
        "OPTIONS": {
          "CLIENT_CLASS":"django_redis.client.DefaultClient",
        }
    }
}

The location is changed to match the url of the Cache service for example ElastiCache from AWS.

Setting up a PostgreSQL database in AWS RDS and connecting it to pgAdmin.

Samuel Mugane — Mon, 28 Mar 2022 11:14:34 +0000

Introduction

PgAdmin is a client-side platform for adminstration and development of PostgreSQL database. It simplifies the visualization of schemas, logs and other relevant information based on SQL standards. Amazon has a great relational database hosting service known as Amazon Relational Database Service (RDS). The article focuses on creating a PostgreSQL database in RDS and connecting it pgAdmin.

Creating a PostgreSQL database in AWS

Setup and login to your AWS account as ROOT user. Go to services, choose database and select RDS.
You will be redirected to a dashboard as shown below. Click on create database.

Select PostgreSQL under engine options and choose templates as shown below. Use Free Tier if setting up the database for personal projects.

Scroll down to settings and set master username and password. Copy these credentials to the clipboard as they will be used to make the pgAdmin connection.

Scroll down to connectivity and select "Yes" for public access. This will allow the database to accept the connection from local setup.

Scroll down and click on "create database". The process will take a few minutes and you will be directed to the page below. Copy the endpoint to the clipboard for later reference.

Under security, click on the VPC security groups and you will be redirected to an EC2 console as shown below.

Click on security group ID and you will be redirected to the page below.

Under inbound rules, check if there are rules for PostgreSQL that accepts TCP connections. If there isn't click on "edit inbound rules". Click on "Add rule". Under type, select PostgreSQL. Under source, select custom. Under the search input, select 0.0.0.0/0 and click on "Save rules".

The PostgreSQL database has been successfully been created and ready to accept connections from a local host.

Installing pgAdmin and making RDS connection

First we need to install pgAdmin to make the connection to our PostgreSQL database hosted in AWS RDS.
Here is an installation guide for installing pgAdmin on Ubuntu, Mac and Windows.

After installing pgAdmin, launch it and register a new server as shown below.

Under general, enter server name and move to connection tab. Under host name/address, paste the endpoint copied from AWS-RDS console. Enter the port set in RDS. Enter the master username set on RDS under username and corresponding password. Under advanced, set connection timeout (seconds) to 1000 secs to avoid timeout during connection. Leave every other configuration in its default.

After the connection is accepted, the pgAdmin dashboard will look as follows;

Now you can view tables created under schemas/tables.

Conclusion

PgAdmin offers a versatile interface to view database information while AWS RDS offers a great service to host databases to the cloud for better security and ease of access.

Using Django Template Context

Samuel Mugane — Sun, 13 Mar 2022 12:03:04 +0000

Introduction

Django has proven its resilience as a framework for web development. Django provides developers with a discrete playing field with powerful libraries that help cut down on development time. Django enables the developer to engage with both the client-side and the server-side in a well versed environment. In this article, we explore the Django template context that helps with rendering content to a template on the client-side.

Prerequisites

Python 3.x
Django
Pip

Getting Started

As always, having a virtual environment set up is crucial as it helps with controlling code dependencies. So create a virtual environment using python venv a standard library for creating and managing virtual environments. Using terminal, implement the following steps.

Make new directory mkdir directoryname and cd into the new directory cd directoryname
Create virtual environment python3 -m venv myvenv
CD into the virtual environment cd myvenv
Activate virtual environment source bin/activate

Now install Django in new virtual environment

First, install pip sudo apt-get install python3-pip
Using pip, install Django framework pip3 install django
Verify installation django-admin --version

Set up Django project

With Django framework now installed, create Django project using django-admin startproject projectname
CD into the project created cd projectname
Make migrations for the database python3 manage.py migrate
Create super user for admin access using python3 manage.py createsuperuser and enter your username, email and password.
Now create your first Django app using python3 manage.py startapp appname
Start server python3 manage.py runserver and on your terminal you will get a link i.e http://127.0.0.1:8000/
When you click the link, you will be directed to a webpage on your browser and it should look as shown below:

Rendering your first template

Django has a dynamic templating system that helps in rendering HTML files to the web.

Configuring settings.py file

Go to settings.py under your project folder and configure the INSTALLED_APPS list variable. Add 'my_app.apps.WebAppConfig' to the list variable.

Creating templates folder

In your app folder, create a new folder and name it templates. Inside the templates folder create another folder with the same name as your app name i.e if you app name is my_app the folder inside the templates folder should be my_app too.

Create an index.html file in the newly created folder.

Creating your first view

Go to views.py file in your app folder.

from django.shortcuts import render

def home(request):
    return render(request, 'my_app/index.html')

In your app folder, create a urls.py file and initiate your home page route

from django.urls import path
from my_app.views home

urlpatterns = [
    path('', home, name='home'),
]

In your project folder, edit the urls.py file to include the urls from your app.

from django import urls
from django.contrib import admin
from django.urls import path, include


urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('my_app.urls'))
]

Using template context

The template context is a dictionary with variable names represented as key value pairs. It is initiated within a view function to pass content to be render to a webpage. For example, we want to render data from the database to a webpage. Let the data be personal information for users.

Let's create a model for users in models.py

from django.db import models

class Users(models.Model):
    fullname = models.CharField(max_length=100)
    email = models.CharField(max_length=50)
    mobile = models.CharField(max_length=15)

Make migrations to database:

python3 manage.py makemigrations
python3 manage.py migrate

To render the data, we can add the context to the home view function as follows:

from multiprocessing import context
from django.shortcuts import render
from .models import Users

def home(request):
    context = {'users_list': Users.objects.all()}
    return render(request, 'my_app/index.html', context)

In the index.html file, we can use template tags to render the user information as follows:

<!--Users Table-->

<div class="row">
    <div class="table-responsive bg-white my-3" style="border-radius: 5px;">
    <div class="title fw-bold fs-3">
        Users List
    </div>
    <table class="table table-striped">
        <thead class="border-bottom font-weight-bold">
        <tr
            <td>Full Name</td>
            <td>Mobile</td>
            <td>Email</td>
        </tr>
        </thead>
        <tbody>
        {% for user in users_list %}
        <tr>
            <td>{{user.fullname}}</td>
            <td>{{user.mobile}}</td>
            <td>{{user.email}}</td>
        </tr>
        {% endfor %}
        </tbody>
    </table>
    </div>
</div>

All the user information will be render on the webpage inform of a table. The {{user.fullname}} will be replaced by the fullname of the user and the same is true for the other variable names.

Conclusion

The template context's functionality helps in rendering content to a webpage in a quick and efficient manner. You can learn more about Django template context here.

Using Twilio API in Node.js

Samuel Mugane — Thu, 03 Mar 2022 14:12:42 +0000

Banner courtesy of Spencer Gabor

Introduction

APIs (Application programmable interfaces) have revolutionized how data is shared in different sectors across the globe. Developers have a more easier way of accessing data, and providing uniquely, well crafted services to their customers through APIs. Twilio has been a key player in revolutionizing how communication can be leveraged by developers to enhance customer service. Twilio offers programmable voice calls, short-message service (SMS), email service and whatsapp integration as some of the products available to developers. The Twilio API is a pay as you go service, and provides a reliable API for communication. Find more about pricing here.

The article focuses on making programmable voice calls, and sending SMS. Twilio API calls for both voice calls, and SMS are easy to make especially when one is familiar with the node environment. The following steps will have you setup, and running:

Prerequisites

Node.js
NPM
Dotenv

STEP 1

Firstly, install twilio library through node package manager (npm). As prerequisites, install node.js, and npm. If you are on Ubuntu 20.04, open terminal, and run the following commands.

sudo apt install nodejs
sudo apt install npm

You can also use different methods to install Node, and npm as shown here.

Make a new directory mkdir, and cd into that directory. Initialize the directory with:

npm init

Then install the node.js twilio library through:

npm install twilio

STEP 2

Sign up to twilio, and set up a phone number. Under your account info, you'll see your account SID, Auth token, and phone number. Copy them to the clipboard.

STEP 3

In your node development environment, set up a dotenv (.env) file to create environment variables to secure your account SID, and Auth token. As a prerequisite, install dotenv using npm:

npm install dotenv --save

In your dotenv file, paste your account SID, and Auth Token in the following format:

TWILIO_ACCOUNT_SID={your account SID}
TWILIO_AUTH_TOKEN={your Auth token}

Ensure no spacing between variable name, equal sign, and value. Also, remove curly braces when keying the values.

In your development environment, create script.js file, and paste the following code:

const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;
require('dotenv').config();
const client = require('twilio')(accountSid, authToken);

client.calls
      .create({
         url: 'http://demo.twilio.com/docs/voice.xml',
         to: '',
         from: ''
       })
      .then(call => console.log(call));

Paste your phone number as a string to the from: key, and the phone number your want to call on to: key i.e (area code) – (phone number) or (XXX)- XXX - XXXX . Ensure to follow the E.164 format while entering the phone numbers. For message API calls. Using the following code:

const accountSid = process.env.TWILIO_ACCOUNT_SID;
const authToken = process.env.TWILIO_AUTH_TOKEN;
require('dotenv').config();
const client = require('twilio')(accountSid, authToken);

client.messages
      .create({
         body: ' ',
         to: '',
         from: ''
       })
      .then(call => console.log(call));

Under body: key, enter your message.

N/B To avoid error during call due to Geo restrictions, go back to your twilio account dashboard, on your left tab click on #phone, and go to settings. Click on Geo permissions, search for your country and enable as shown below: Select country, and save.

Run and Test
Back in you development environment, save your code, and run it on terminal:

node script.js

A JSON response will be logged to your console:

{
  "account_sid": "ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "annotation": null,
  "answered_by": null,
  "api_version": "2010-04-01",
  "caller_name": null,
  "date_created": "Tue, 31 Aug 2010 20:36:28 +0000",
  "date_updated": "Tue, 31 Aug 2010 20:36:44 +0000",
  "direction": "inbound",
  "duration": "15",
  "end_time": "Tue, 31 Aug 2010 20:36:44 +0000",
  "forwarded_from": "+141586753093",
  "from": "+987654321",
  "from_formatted": "+987654321",
  "group_sid": null,
  "parent_call_sid": null,
  "phone_number_sid": "PNXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "price": "-0.03000",
  "price_unit": "USD",
  "sid": "CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
  "start_time": "Tue, 31 Aug 2010 20:36:29 +0000",
  "status": "completed",
  "subresource_uris": {
    "notifications": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Notifications.json",
    "recordings": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Recordings.json",
    "feedback": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Feedback.json",
    "feedback_summaries": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/FeedbackSummary.json",
    "payments": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Payments.json",
    "events": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Events.json",
    "siprec": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Siprec.json",
    "streams": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Streams.json"
  },
  "to": "+123456789",
  "to_formatted": "+123456789",
  "trunk_sid": null,
  "uri": "/2010-04-01/Accounts/ACXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/Calls/CAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.json",
  "queue_time": "1000"
}

if status is complete, you'll receive a demo call to the number you entered to the to: key.

Conclusion

Twilio has a great use case in automating sending confirmation code through SMS or email to users. The programmable voice allows one to make, and receive calls with no hassle.

DEV Community: Samuel Mugane

Syncing AWS S3 Storage Bucket Objects to LocalStack S3 Storage Bucket

Getting Started

Installation

Configuring and Starting LocalStack

Installing AWS CLI and LocalStack’s AWS CLI

Creating S3 Storage Bucket in LocalStack

Syncing S3 Storage Buckets

Method 1: Sequential sync

Method 2: Direct Sync (Without Local Storage)

Conclusion

Web Authentication on Stellar: A Deep Dive

Introduction

Stellar Web Authentication

Define a custom JWT token class

Define a custom authentication scheme

Define helper functions

Define views to handle the authentication flow

Conclusion

Resources

What happens when you type a URL on the browser and press enter

Overview

The Flow of the request

1. Typing the URL and pressing enter

2. Browser resolves the domain to its unique IP

3. Browser initiates a TCP connection with the server

4. Browser sends the HTTP request to the server.

5. Server receives request, processes the request and returns a response

Holberton School

6. Browser renders the content

Closing

Caching Django Application using Redis

Introduction

A visual of Browser Cache functionality

Getting Started

Implementing caching on views

Final Note

Setting up a PostgreSQL database in AWS RDS and connecting it to pgAdmin.

Introduction

Creating a PostgreSQL database in AWS

Installing pgAdmin and making RDS connection

Conclusion

Using Django Template Context

Introduction

Prerequisites

Getting Started

Now install Django in new virtual environment

Set up Django project

Rendering your first template

Configuring settings.py file

Creating templates folder

Creating your first view

Using template context

Conclusion

Using Twilio API in Node.js

3. Browser initiates a `TCP` connection with the server