DEV Community: Amel Spahić

How to create and publish a TypeScript library with ease

Amel Spahić — Tue, 07 Mar 2023 22:57:23 +0000

amelspahic / typescript-library-template

Initial boilerplate for TypeScript libraries including tests, coverage, documentation.

typescript-library-template

A starter template for TypeScript libraries. Use this repository as a starting point for your own TypeScript library. This template includes the following features:

Compiles TypeScript code using both the tsconfig.json and tsconfig.module.json files.
Formats TypeScript code using Prettier.
Lints TypeScript code using ESLint.
Runs unit tests using AVA.
Generates code coverage reports using NYC.
Generates HTML documentation using TypeDoc.
Uses Husky Git hooks and Lint-staged pre-commit hooks.

Installation

Clone the repository:

git clone https://github.com/amelspahic/typescript-library-template.git

Install the dependencies:

npm install

There are several scripts available to help you get started:

Compile the TypeScript code using both the tsconfig.json and tsconfig.module.json files.

npm run build

Formats the TypeScript code using Prettier and lints the code using ESLint, fixing any issues found.

npm run fix

Lints the TypeScript code using ESLint, checks the code formatting using Prettier, and runs the unit tests using AVA.

npm run test

…

View on GitHub

TypeScript is a popular and influential language that extends JavaScript with static typing and other features. It can help you write more reliable, maintainable, scalable code for your projects. However, creating and publishing a TypeScript library can be challenging, especially if you are new to the TypeScript ecosystem. Many tools and configurations are involved, and finding a good template or guide that covers everything you need is challenging.

For the reasons mentioned above, I created a TypeScript library template you can use as a starting point for your library. It is based on my experience and best practices from the TypeScript community (this is usually subjective), and it includes everything you need for developing, testing, documenting, and publishing your library. This article will explain what this template offers, how to use it, and how to customize it for your needs.

What is typescript-library-template?

The typescript-library-template is a GitHub repository that you can use as a template for creating your own TypeScript library. It has the following features:

TypeScript: as the primary language for writing your library code. It also includes some recommended settings for the tsconfig.json and tslint.json files.
AVA: as the testing framework for writing and running unit tests for your library. It also supports code coverage reports with Istanbul (NYC).
Typedoc: as the documentation generator for creating API documentation for your library from your TypeScript comments.
Prettier: as the code formatter for keeping your code style consistent and readable.
ESLint (typescript-eslint): for static code analysis.
Husky and Lint-staged: The template uses Husky and Lint-staged to run pre-commit hooks that ensure your code is formatted, linted, tested, and documented before committing.

How to use typescript-library-template?

I will mention just a couple of commands below. You can find more information in the README.md file.

Using typescript-library-template is very easy. You need to follow these steps:

Click the "Use this template" button on the GitHub repository page. This will create a new repository based on this template under your account.
Clone or download your new repository to your local machine.
Run npm install or yarn install to install all the dependencies.
Start developing your library by editing the src/index.ts file (or adding more files under src). Using AVA syntax, you can write unit tests using .spec.ts or .test.ts suffixes.
Run npm run build or yarn build to build your library. This will generate multiple output files under the dist folder.
Run the npm run test or yarn test to run all unit tests using AVA.
To generate a code coverage report under the coverage folder, run npm run cov.
Run npm run doc or yarn doc to generate API documentation using Typedoc. This will create an HTML file under docs/index.html.
Run npm run fix or yarn fix to format all source files using Prettier.
Commit your changes (it will activate a pre-commit hook that will perform test and linting/formatting)
Push your changes to GitHub.

How to customize typescript-library-template?

You can customize the typescript-library-template according to your preferences. Add files, change configurations, and remove whatever you find unimportant to your liking.

Geoguessr exploit with proxy (Fiddler on Windows)

Amel Spahić — Thu, 27 Oct 2022 19:53:40 +0000

There is a popular (and incredible) geographical game called Geoguessr, where players guess locations from Google Maps Street View imagery. I play it regularly, and I am sure many of you have seen those guys who can guess where they are just by a quick glimpse of the part of the grass or sky or just pixelated images. Well, you can do it, too.

Avoid using this method, but I would like to know about things in the real world.

How Geoguessr works?

As with any other web application, Geoguessr is a client-server application with client-side and server-side parts. The simplified explanation would be:

Through the application's user interface, the user sends a request to the web server across the Internet.
The web server forwards this request to the web application server.
After completing the specified task, the web application server generates the necessary data results.
The web application server then relays those results to the web server (requested information or processed data).
The client receives the requested information from the web server (tablet, mobile device, or desktop).
The requested information appears on the user's display.

Let's take, for example, a Classic Mode and find your map (World, Famous Places, United States, etc.). When you start a game, a browser will send an HTTP request to the endpoint: https://www.geoguessr.com/api/v3/games. The response will return the current location's latitude and longitude. That easy!

For different playing modes, the game will call other endpoints, e.q., Streaks will call https://www.geoguessr.com/api/v3/games/streak.

This begs the question: Well, we already have the information about the location; how to make something out of it without suffering? We can manually search for the
latitude and longitude on Google Maps, but it is time-consuming and slow. There is a better idea, and we call it - an INTERMEDIARY (Proxy).

Proxy?

There is no better explanation than in Section 2.3 of RFC 7230 (Request for Comments) of the Internet Engineering Task Force (IETF).

A "proxy" is a message-forwarding agent that is selected by the
client, usually via local configuration rules, to receive requests
for some type(s) of absolute URI and attempt to satisfy those
requests via translation through the HTTP interface. Some
translations are minimal, such as for proxy requests for "http" URIs,
whereas other requests might require translation to and from entirely
different application-level protocols. Proxies are often used to
group an organization's HTTP requests through a common intermediary
for the sake of security, annotation services, or shared caching.

Fiddler Classic

We want something easy to work with, easily configurable, and accessible. This is not a sponsored blog post, but Fiddler Classic is one of the best tools of this kind for Windows I have been working with.

Fiddler Classic is a specialized proxy server tool for troubleshooting online traffic from programs like browsers. It records and collects online traffic before sending it to a web server. The replies from the server are then sent back to the Fiddler program, which subsequently sends them back to the client. Fiddler's web debugging user interface shows the captured web traffic through a session list.

Capture and decrypt HTTPS traffic

By default, Fiddler Classic does not capture and decrypt secure HTTPS traffic. We need to enable HTTPS traffic decryption to capture data sent through HTTPS.

Go to Tools > Options > HTTPS
Ensure that Capture HTTPS CONNECTs and Decrypt HTTPS Traffic are selected

Now we can track HTTPS requests made from our browsers. Here is an example of Geoguessr game invocation, the same results as those from the browser image above:

FiddlerScript

One of Fiddler's most powerful features is FiddlerScript, which enables us to alter requests and responses "on the fly" to implement any desired action. It is built on JScript.NET, a.NET version of JavaScript, making it simple for web developers to use. Most .NET developers can design basic rules using FiddlerScript because the syntax is similar enough to C#.

Go to Rules > Customize Rules..., and it will open up the FiddlerScript editor (or Notepad if you don't have an editor installed)

Many things are happening in that file, but we will focus on the OnBeforeResponse session event method. It fires after the complete response has been read from the server and before the response is returned to the client (unless Streaming is enabled), so we can add some behavior.

Open Microsoft Edge pinned to the location

Once we start playing the game, we want our FiddlerScript to run a new browser session with the location taken from the response pinned on a map.

To retrieve latitude and longitude from the JSON response, we need to do some JSON decoding and get the latest result from the rounds parameter (the parameter rounds is an array of the current and previous results).

The only thing left now is to launch a fresh browser session (or another tab) if you launch the same browser as your current one. I use the Brave browser and launch Microsoft Edge (because it is on Windows by default). Our final OnBeforeResponse event will look like this:

static function OnBeforeResponse(oSession: Session) {
    if (m_Hide304s && oSession.responseCode == 304) {
        oSession["ui-hide"] = "true";
    }

    // Only GET method, otherwise, it will open the browser again once we POST the results
    if (oSession.RequestMethod == "GET" 
        && (oSession.uriContains("/api/v3/games/") 
        || oSession.uriContains("/api/v3/challenges/")
        || oSession.uriContains("/api/battle-royale/")
        ) 
    ) {

        var latLng = oSession.GetResponseBodyAsString();
        var oJson = Fiddler.WebFormats.JSON.JsonDecode(latLng);

        var rs = oJson.JSONObject["rounds"];
        var lat =  rs[rs.Count - 1]["lat"];
        var lng = rs[rs.Count - 1]["lng"];

        var zoomLevel = "8z";

        Utilities.LaunchHyperlink("microsoft-edge:https://maps.google.com/maps/place/"+lat+","+lng+"/@"+lat+","+lng+","+zoomLevel);
    }
}

Finally, you need to keep FiddlerClassic open for it to act as a proxy. See it in action:

Docker Persistence feat. MS SQL Server, PostgreSQL, MariaDB, MySQL, MongoDB

Amel Spahić — Sat, 21 May 2022 18:59:00 +0000

We did the initial project setup on containers in the previous "Demystifying Docker" series article. In this one, we will cover the database with persistence (examples with MS SQL Server, PostgreSQL, MySQL, MariaDB, Mongo) part. Still, first, we need to clarify a couple of things regarding the concept of the critical services (such as the database) running in containers.

Why NOT database in Docker?

TL;DR: Carefully consider using it in the production environment if your database is a critical service.

Docker was not created with persistence in mind, and one of the most appealing features of containers is their ability to be started and terminated at will. The data in the container is transient, and it is erased with the container.

Be careful using database services in containers in a PRODUCTION environment! If your database is not a critical service (which is rare), maybe you could try it in Docker; otherwise, you are entering a dangerous zone of having a headache in the (probably near) future.

If you are deploying on a cloud, the best option is to use cloud database services (AWS Relational Database Service (RDS), Azure Databases, Google Cloud Databases, Managed Databases on Digital Ocean, or an equivalent hosted database service of your cloud provider). This will simplify many management tasks such as updating minor versions, handling regular backups and even scaling up.

Why YES database in Docker?

TL;DR: For any environment (production with particular caution).

Let us suppose that we have a couple of different database engines in our projects, or have different versions of any (or all) of them, or your colleague has another operating system or with a different setup, or you need a quick start on the new developer joining the team, we can mention a lot more, but I am sure you get the point. You don't want to overload your working machine with different installations and versions when you can do it in one place easily through the docker CLI commands or the docker-compose specification. You can add or remove it without impacting the rest of the system.

As previously mentioned, containers don't exist with persistence in mind, but we usually want to have data even if we restart our container engine or working machine. Luckily, there are a couple of ways to preserve the data mentioned below.

Data persistence

There are two ways of Docker handling the persistence in general - volumes and bind mounts. Both allow you to mount a location on the host machine to a place in the container. This provides storage for the data even if the container is shut down, and there is no need to worry about the data being lost.

Bind mounts

Bind mount is not my preferred way of persisting data, but it still has its usage, so I will briefly run through the idea with PostgreSQL as an example. Bind mounts will mount a file or directory to the container from the host machine, and then it can be referenced via its absolute path. It relies on the host machine's filesystem having a specific directory structure. If not, it is necessary to explicitly create a path to the file or folder to place the storage. Additionally, bind mounts give us access to sensitive files, and we can change the host filesystem through processes running in a container, which is considered a security implication.

Usage

You can use the flags --mount and -v to use bind mounts on a container. The most noticeable difference between the two options is that --mount is more explicit and verbose, whereas -v is more of a shortcut for --mount. It combines all of the --mount options into a single field. To learn more, always reference the official documentation.

docker run --rm --name pgdb -e POSTGRES_PASSWORD=somepass --mount type=bind,source="$(pwd)",target=/var/lib/postgresql/data -p 2000:5432 -d postgres

Commands explanation:

docker run creates a writeable container layer over the specified image and then starts it using the selected command
--rm will automatically remove the container when it exits
--name pgdb assigns container name pgdb
-e POSTGRES_PASSWORD=somepass sets the required environment variable to use the PostgreSQL image, and it must not be empty or undefined. This environment variable specifies the superuser password for PostgreSQL.
--mount type=bind,source="$(pwd)",target=/var/lib/postgresql/data creates the type=bind mount between the current directory (pwd returns a current working directory name) and the default directory for the database files in Docker container /var/lib/postgresql/data
-p 2000:5432 maps the default PostgreSQL port 5432 to the host port 2000 (later, we will use port 2000 to connect to the database instance from the host machine). You can change port 2000 to some other port if that one is already in use.
-d will run the command in the detached mode (it won't lock our terminal). By design, containers started in detached mode exit when the root process used to run the container exits, unless you specify the --rm option. If you use -d with --rm, the container is removed when it exits or when the daemon exits, whichever happens first.
postgres is the base PostgreSQL image

Our PostgreSQL container is running, and now it is accessible from the appropriate database tool (such as DBeaver Community). I am using port 2000 and password somepass to connect to our database instance, as we defined earlier in the Docker CLI command.

My current working directory is ~/testmount, and running the previous Docker command will create the necessary database files in that folder.

Volume mounts (preferred way)

Docker fully manages docker volumes; therefore, they are unaffected by our directory structure or the host machine's operating system. When we utilize a volume, Docker creates a new directory in the host machine's storage directory, and Docker handles its content. For Docker volumes, storage is not associated with the container's life cycle and resides outside the container.

Some of the advantages:

kill containers as you need and still retain your data
attach volumes to multiple containers running at the same time
reuse storage across multiple containers (for example, one container writes to storage and another reads from storage)
volumes do not increase the size of the Docker containers that use them
you can use the Docker CLI to manage your volumes (for example, retrieving a list of volumes or deleting unused volumes).

All commands below follow the same pattern because we use persistence for the container unrelated to which database management system we use. The only things different are environment variables and target directories (because other databases usually use different file locations).

docker run --rm --name <custom-name> \
-e <ENV_VAR_1=some_value_1> \
-e <ENV_VAR_2=some_value_2> \ 
--mount type=volume,source=<custom-source-name>,target=<database-file-location-in-container> \
-p <host-port>:<container-database-port> \
-d <docker-image-with-or-without-tag>

Commands explanation:

docker run creates a writeable container layer over the specified image and then starts it using the selected command
--rm will automatically remove the container when it exits
--name custom-name assigns container name custom-name
-e ENV_VAR_1=some_value_1 -e ENV_VAR_2=some_value_2 sets the required environment variables to use for the database image. Different database management systems require different environment variables. You can find those in the documentation for the specific DBMS.
--mount type=volume,source=custom-source-name,target=database-file-location-in-container creates the type=volume with name custom-source-name and mounts it to the default directory for the database files in Docker container database-file-location-in-container
-p host-port:container-database-port maps the default database port (i.e., 5432 for PostgreSQL, 1433 for SQL Server, 3306 for MariaDB and MySQL, etc.). You can change the host-port to some free port on your host machine.
-d will run the command in the detached mode (it won't lock our terminal). By design, containers started in detached mode exit when the root process used to run the container exits, unless you specify the --rm option. If you use -d with --rm, the container is removed when it exits or when the daemon exits, whichever happens first.
docker-image-with-or-without-version is the base Docker image from the Docker Hub.

SQL Server

docker run --rm --name our-mssql -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=Str0ngP@ssword" -e "MSSQL_PID=Express" --mount type=volume,source=custommssql,target=/var/opt/mssql -p 7000:1433 -d mcr.microsoft.com/mssql/server:2019-latest

Environment variables:

ACCEPT_EULA confirms your acceptance of the End-User Licensing Agreement.
SA_PASSWORD is the database system administrator (userid = 'sa') password used to connect to SQL Server once the container is running. Important note: This password needs to include at least 8 characters of at least three of these four categories: uppercase letters, lowercase letters, numbers, and non-alphanumeric symbols.
MSSQL_PID is the Product ID (PID) or Edition that the container will run with. Acceptable values: Developer (this is the default if no MSSQL_PID environment variable is supplied), Express, Standard, Enterprise, EnterpriseCore.

A complete list of environment variables can be found here.

Connect using information defined above

Server Name - 127.0.0.1,7000
Authentication - SQL Server Authentication
Username - sa
Password - Str0ngP@ssword

PostgreSQL

docker run --rm --name our-postgresql -e POSTGRES_PASSWORD=Str0ngP@ssword --mount type=volume,source=custompostgres,target=/var/lib/postgresql/data -p 7001:5432 -d postgres:latest

Environment variables:

POSTGRES_PASSWORD - This environment variable is required for you to use the PostgreSQL image. It must not be empty or undefined. This environment variable sets the superuser password for PostgreSQL. The default superuser is defined by the POSTGRES_USER environment variable (if not present, the default user is postgres)

Complete list of environment variables can be found here.

Connect using information defined above

Server Host - localhost
Port - 7001
Username - postgres
Password - Str0ngP@ssword

MariaDB

docker run --rm --name our-mariadb -e MARIADB_ROOT_PASSWORD=Str0ngP@ssword --mount  type=volume,source=custommariadb,target=/var/lib/mysql -p 7002:3306 -d mariadb:latest

Environment variables:

MARIADB_ROOT_PASSWORD specifies the password that will be set for the MariaDB root superuser account. In the above example, it was set to Str0ngP@ssword password.

A complete list of environment variables can be found here.

Connect using information defined above

Server Host - localhost
Port - 7002
Username - root
Password - Str0ngP@ssword

MySQL

docker run --rm --name our-mysql -e MYSQL_ROOT_PASSWORD=Str0ngP@ssword --mount  type=volume,source=custommysql,target=/var/lib/mysql -p 7003:3306 -d mysql:latest

Environment variables:

MYSQL_ROOT_PASSWORD specifies the password set for the MySQL root superuser account. In the above example, it was set to the Str0ngP@ssword password.

A complete list of environment variables can be found here.

Connect using information defined above

Server Host - localhost
Port - 7003
Username - root
Password - Str0ngP@ssword

MongoDB

docker run --rm --name our-mongo -e MONGO_INITDB_ROOT_USERNAME=someuser -e MONGO_INITDB_ROOT_PASSWORD=Str0ngP@ssword --mount type=volume,source=custommongo,target=/data/db -p 7004:27017 -d mongo:latest

Environment variables:

MONGO_INITDB_ROOT_USERNAME and MONGO_INITDB_ROOT_PASSWORD will create a new user and set that user's password. This user is created in the admin authentication database and given the root role, a superuser role.

Note: This image will also create a volume for /data/configdb, so you will see an additional hex ID volume in your Docker volumes.

Connect using information defined above

URI - mongodb://someuser:Str0ngP@ssword@localhost:7004

Volumes

You can check your newly created volume with the command:

docker volume ls

Or you can check your running Docker Desktop instance under the Volumes tab.

Docker Compose

Every previously defined database management system can be run through the docker-compose instead of Docker CLI.

version: '3.4'

services:
  db:
    image: postgres:latest
    restart: always
    environment:
      - POSTGRES_PASSWORD=Str0ngP@ssword
    ports:
      - 7000:5432
    volumes:
      - custommssql:/var/lib/postgresql/data

volumes:
  custommssql:
    driver: local

Which one to use?

There are a few essential things to consider when determining whether to use volumes or bind mounts. You should utilize volumes if you want your storage or persistent layer to be fully controlled by Docker and accessed only through Docker containers and the Docker CLI.

On the other hand, Bind mounts are the perfect solution for the task if you need complete control of the storage and plan on letting other processes besides Docker access or modify the storage layer.

According to the Docker documentation, using volumes is the most straightforward approach to start persisting data in your Docker container. In general, bind mounts have additional restrictions.

Based on what we've observed thus far, this conclusion is not surprising. If you're ever unsure how to persist data in your Docker containers, using volumes is a wise rule of thumb.

A bind mount is distinguished because it can be accessed and modified by programs other than Docker. This can be advantageous for integrating Docker with other operations, but it can also be inconvenient if security is an issue.

Next Steps

Although we used databases as an example, we described two ways to keep data on Docker containers. The following article will show how to add the database service to the already created application with docker-compose and cover the integration part.

Demystifying Docker: .NET 6 on Docker + Docker Debugging

Amel Spahić — Sat, 16 Apr 2022 18:23:35 +0000

We start this series by building and debugging a .NET 6 Web API application in Docker containers. Many developers still avoid using Docker for development and debugging purposes, so I hope this will somehow reduce the gap of the unexplored. This is the first part, so come back later to check others where we add PostgreSQL with persistence, log to Elasticsearch, explore and visualize with Kibana to the existing setup, and much more.

This part of the tutorial will cover .NET 6, but the whole idea of containerized services is that we are not limited to just a particular technology. Instead of .NET, maybe you prefer writing applications in NodeJS or Python (in later examples), or you want to combine all these technologies, containers allow us to do so with the least effort.

What you will learn

After this series, you should be able to understand the significance of containerization with Docker and apply new knowledge to your projects. Also, I will try to keep up with best practices as much as possible so you can produce better software.

Prerequisites

Before we can even support this kind of story, we need to ensure that our machine and operating system keep the minimum requirements for the Docker engine.
A couple of things need to be installed on your machine before we start with this process:

.NET 6 SDK
Docker Engine
Visual Studio Code (You can use others, but this one is with most extensions)

Source

You can find the source code on GitHub. Every part of this series will have a separate branch, and all of them will be merged into the master branch.

https://github.com/amelspahic/blog-series/tree/part-1

Why Docker

Docker is a containerization technology for packaging your program and its dependencies into containers, ensuring that your application runs smoothly in any environment, including development, test, and production. Docker is a program that makes it easier to construct, deploy, and run containerized applications. With Docker in mind, the code running on your machine will function the same on every other machine.

.NET 6 Web API

Create a directory where we will hold everything related to our project.

# Create a new folder named `blog-series`
mkdir blog-series

# Change directory to the new directory
cd blog-series

Create a .NET 6 Web API application using .NET CLI in the terminal (if you use Visual Studio, you can create a new project from the template).

dotnet new webapi --no-https -o src/sample-app -n SampleAPI
cd src/sample-app

Command explanation:

dotnet new webapi will create a new Web API project (check additional templates with dotnet new --list
--no-https will turn off HTTPS (I use reverse proxy configuration anyways)
-o src/sample-app will output created boilerplate in the src/sample-app folder
-n SampleAPI is the name of our project SampleAPI.csproj

CLI will generate a sample boilerplate that we will use in this example. The project is ready to run, so for validation, run dotnet run from the src/sample-app folder (where we created our .NET 6 project).

dotnet run

The output will look like the image below, and it will show on which ports your application is available.

We can access the endpoint on the presented URL (mine is on localhost, port 5130), http://localhost:5130/WeatherForecast will get the sample response.

Visual Studio Code

At the start, we created a new folder, blog-series (maybe you named it differently), where we keep all our independent projects. Open it in Visual Studio Code.
You will be prompted to add build and debug assets for C# - choose Yes and VS Code will create another folder .vscode in which debugging and task configurations are stored.

If you are not prompted to generate build and debug assets, you can do it manually through the Command Palette (Ctrl+Shift+P) and run the .NET: Generate Assets for Build and Debug command.

Application containerization

Be sure to have the Docker extension installed so that we can use extension goodies. We could create our own, but we won't bother since this is a more convenient method.

This extension will expand VS Code Command Palette with additional options. Press Ctrl+Shift+P to open the Command Palette and search for Docker: Add Docker Files to Workspace... (start typing).

You will be prompted with different options, including:

Select Application Platform - choose .NET: ASP.NET Core
Select Operating System - choose Linux (this is the operating system of the container, not your machine, so definitely choose Linux)
Port - type in which container port your application will use (I prefer 5000)
Include optional Docker Compose files - choose Yes

The extension will create a couple of files and configurations:

Dockerfile
.dockerignore, which contains files and directories patterns to be excluded from the context
docker-compose.yml - a YAML file that defines the services and, with a single command, can spin everything up or tear it all down
docker-compose.debug.yml - same as docker-compose.yml, but with debugging configuration
VS Code tasks for building and running the container (in both debug- and release configuration, four tasks in total), and a debugging configuration for launching the container in debug mode.

Dockerfile

The previous step created a Dockerfile, a text document that contains all the commands to assemble an image in our src/sample-app folder. Visual Studio Code extension did a good job, but I added a comment above each step to better understand what happened.

# Initialize a new build stage and set the Base Image 
FROM mcr.microsoft.com/dotnet/aspnet:6.0-focal AS base
# Set the working directory
WORKDIR /app
# Expose the port 5000
EXPOSE 5000
# Sets the environment variable
ENV ASPNETCORE_URLS=http://+:5000

# Creates a non-root user with an explicit UID and adds a permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-dotnet-configure-containers
RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
USER appuser

# Initialize the new build stage and set the Build Image
FROM mcr.microsoft.com/dotnet/sdk:6.0-focal AS build
# Set the working directory for subsequent COPY and RUN commands
WORKDIR /src
# Copy the project file to the container filesystem /src folder
COPY ["src/sample-app/SampleAPI.csproj", "src/sample-app/"]
# Execute dotnet restore to restore dependencies specified in the .csproj file
RUN dotnet restore "src/sample-app/SampleAPI.csproj"
# Copy everything to the container filesystem
COPY . .
# Set the working directory for subsequent RUN commands
WORKDIR "/src/src/sample-app"
# Execute project build with all of its dependencies
RUN dotnet build "SampleAPI.csproj" -c Release -o /app/build

# Pick up where a previous build stage left off
FROM build AS publish
# Publish the application and its dependencies to a folder for deployment
# UseAppHost=false to disable the generation of the native executable.
RUN dotnet publish "SampleAPI.csproj" -c Release -o /app/publish /p:UseAppHost=false

# Pick up where a previous base stage left off
FROM base AS final
# Set the working directory for the subsequent COPY command
WORKDIR /app
# Copy from the publish image /app/publish directory to the container filesystem
COPY --from=publish /app/publish .
# Specify the command executed when the container is started
ENTRYPOINT ["dotnet", "SampleAPI.dll"]

If you are interested in more Docker references, please check the official documentation.

.dockerignore

Visual Studio Code Docker extension created a .dockerignore file which contains files and directories patterns to be excluded from the build context. This helps avoid unnecessarily sending large or sensitive files and directories to the daemon and potentially adding them to images using ADD or COPY.

**/.classpath
**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm
**/bin
**/charts
**/docker-compose*
**/compose*
**/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
README.md

docker-compose.yml and docker-compose.debug.yml

Our project needs to run multiple containers, and while we can do that separately, we want to have them configured in one place. That is the perfect place to introduce docker-compose. Even in scenarios where we use a single container, using Docker Compose provides a tool-independent configuration in a way that a single Dockerfile does not. Configuration settings such as volume mounts for the container, port mappings, and environment variables can be declared in the docker-compose YML files.

The Compose file is a YAML file defining services, networks, and volumes. Docker-compose and Docker-compose.debug are pretty much the same, with a slightly different setup for the debug configuration (i.e., setting the Development environment and attaching a debugger from the host machine).

version: "3.4"
# A service definition contains a configuration that is applied to each container
# started for that service, much like passing command-line parameters to docker run
services:
  # Define a service called 'sampleapi'
  sampleapi:
    # Specify the image to start the container from
    image: sampleapi
    # Configuration options that are applied at build time.
    build:
      # Either a path to a directory containing a Dockerfile
      context: .
      # Compose uses an alternate file to build with. A build path must also be specified.
      dockerfile: src/sample-app/Dockerfile
    # Expose ports (HOST:CONTAINER)
    ports:
      - 5000:5000
    # Set environment variables
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    # Mount host directories to container directories
    volumes:
      - ~/.vsdbg:/remote_debugger:rw

Please check the official Docker Compose documentation for more details on docker-compose specifications.

Starting the application

We have everything in place to spin up our application in Docker. We can do it in two ways using the command line or the Visual Studio Code Command Palette.

Command line

From the root folder, run the following command.

docker-compose up -d

Command explanation:

docker-compose up creates and starts containers defined in the docker-compose.yml file
-d detaches the execution from the terminal

Wait for the process to complete:

The error on the image is expected since we do not have any images created

Visual Studio Code Command Palette

After the Docker extension installation, we have additional options in the Command Palette (Ctrl+Shift+P). Pick which docker-compose file you want to run.

Another way is to right-click on docker-compose.yml or docker-compose.debug.yml and select Compose Up to reproduce the same behavior as the command-line execution.

Running the commands above will create an image with a sampleapi tag (image name from the docker-compose.yml), (re)create, start, and attach to a service container. You can run docker image ls to check if the image is successfully created. It will list the images we have on our machine. Also, docker ps will show running containers.

Our application should be up and running. We can access it on the port configured in the docker-compose file (in my case, 5000).

Debugging in Docker

If you came this far, you probably know how to use standard debugging techniques. Just act as if it is a typical application without container support.

We are skipping the part of standard debugging techniques and jump straight to Docker debugging. This one is more interesting as it brings you closer to the real-world container behavior.

From the Debug tab, select the Configuration dropdown and select Add Configuration
You will be prompted to choose a predefined configuration, so choose Docker: .NET Core Attach (Preview), and it will automatically add a predefined configuration to launch.json
Right-click on one of the docker-compose*.yml files and select Compose Up. If you go with the docker-compose.yml (without .debug), you will be asked to copy the debugger to the container. Choose Yes to be able to debug. If you are going with docker-compose.debug.yml, it will just spin up the service(s).
Open your browser and go to http://localhost:5000/weatherforecast. You should get the expected response.
Attach the debugger - you will need to choose container group and container
Add a breakpoint and refresh the http://localhost:5000/weatherforecast. If everything is configured correctly, the debugger should attach to your breakpoint.

Reloading changes

When changing your code, right-click on the docker-compose*.yml file, choose Compose Restart, and reattach the debugger.

Next steps

In the next part, we will add database support to our project in the containerized environment. We will focus on PostgreSQL, but a similar approach can be used for many other database systems (the main difference is in the code implementation).

Set up Dynamic DNS for Dynamic IP Addresses at Home (for free) + WireGuard Configuration

Amel Spahić — Fri, 31 Dec 2021 14:34:47 +0000

We already talked about how to Bring Your Home Network Anywhere For Free - Home VPN with Wireguard on Raspberry Pi + Pi-hole (Ubuntu Server 20.04 LTS). It is an awesome thing, especially if you have a static IP address, but if you are like most households in the world (including myself), your internet service provider (ISP) provides you with a dynamic IP address. It means that your home network's IP address changes frequently, even a couple of times a day, so your VPN connection configuration needs to be changed accordingly to keep up.

To solve that, we will utilize the Dynamic DNS solution.

Dynamic DNS

Dynamic DNS is a way of assigning a custom domain name that automatically updates even as the IP address changes. This system has been around long enough that there are workarounds for these kinds of issues. For the purpose of this post, I will use a free DDNS service - No-IP.

Go to the No-IP website and choose some cool hostname and domain:

You are required to sign up, so just populate the required fields and complete the registration (email confirmation and stuff). Your hostname will be pre-populated and set up for the currently logged-in IP address.

If you are not logged in from your home network, you need to change the address to point to your home network's IP address.

Once you are done setting up (don't forget to add username, No-IP will complain), we can set up our router to ping No-IP servers whenever our IP address changes.

Router Set Up

My home router is Technicolor CGA2121, but the configuration shouldn't be different for others, just browse a bit around through the configuration options.

After this is done, our router should automatically call No-IP API to change the pointing IP address.

WireGuard Configuration

When we set up WireGuard on our Raspberry Pi in the previous post, we selected the Public IP option with the current home IP address.

Current configuration:

$ cat /etc/pivpn/wireguard/setupVars.conf
PLAT=Ubuntu
OSCN=focal
USING_UFW=0
IPv4dev=eth0
install_user=ubuntu
install_home=/home/ubuntu
VPN=wireguard
pivpnPORT=51820
pivpnDNS1=10.6.0.1
pivpnDNS2=
pivpnHOST=<PUBLIC IP ADDRESS> # <- We will change this
INPUT_CHAIN_EDITED=0
FORWARD_CHAIN_EDITED=0
pivpnPROTO=udp
pivpnMTU=1420
pivpnDEV=wg0
pivpnNET=10.6.0.0
subnetClass=24
ALLOWED_IPS="0.0.0.0/0, ::0/0"
UNATTUPG=1
INSTALLED_PACKAGES=(net-tools iptables-persistent wireguard-tools qrencode)

Enter the edit mode for setupVars.conf file:

$ sudo nano /etc/pivpn/wireguard/setupVars.conf

Change pivpnHOST to point to your DDNS hostname which you created on No-IP - pivpnHOST=xxxx.ddns.net. Save with Ctrl+X, Y.

New clients you generate will use the new endpoint but you need to manually edit existing clients.
Open your configuration, for example, amel.conf, and update the line:

Endpoint = xxxx.ddns.net:51820

Final Words

I hope this tutorial will help you overcome issues with dynamic IP and help you improve your online being.

Deploy .NET 6 Web Application With GitHub Actions To Self-Hosted Linux Machine (Virtual Private Server, Raspberry Pi, etc.)

Amel Spahić — Fri, 31 Dec 2021 12:37:38 +0000

Let's suppose you are that kind of developer that holds code on GitHub but deploys his fantastic private application manually to some Linux based Virtual Private Server (VPS) you pay $5/month, or to home Raspberry Pi (because why not). You are not alone; a lot of us do that. There is no issue there, but it can be a much better experience.

More advanced users have some script to build to the output folder and then copy it to the server using Secure File Transfer Protocol (SFTP). Not bad, but still not perfect. Different branches - different rules (unless you push everything directly to the main/master branch. I won't tell anyone, but I know you, we will do the same here).

GitHub Actions

GitHub Actions are a new type of CI/CD (Continuous Integration, Delivery) server that has been released by GitHub recently. They integrate closely with GitHub, and the GitHub repository can link GitHub Actions, so they will only run when the GitHub repo is updated.

Running applications on self-hosted runners provides more control over the hardware, operating system, and software tools than GitHub-hosted runners. You may use self-hosted runners to build a customised hardware configuration with greater processing power or memory to execute bigger jobs, install programs available on your local network, and select an operating system that GitHub-hosted runners do not offer.

The process

I will be using Raspberry Pi with Ubuntu Server 20.04.3 LTS, but the same process we can perform on many other Linux distributions.

Foreword

If your application uses a local database to persist the data, that part is not covered here, but you can do it independently of this tutorial.

Step 1 - Installing .NET 6.0 SDK

There are always two parts of .NET applications - building and running ** them. That's why when you visit Download .NET 6.0 (Linux, macOS, and Windows), you will see at least two things - **SDK and Runtime. The former includes everything you need to build and run .NET Core applications, while the latter includes everything just to run them.

We will be building and restoring packages for our application on our machine, so we need to install the SDK part. It also includes the Runtime, so we don't need to install it separately.

From Microsoft documentation: Package manager installs are only supported on the x64 architecture. Other architectures, such as ARM, must install .NET by some other means such as with Snap, an installer script, or a manual binary installation.

Case 1 - CPU on your machine IS x64

If you have Linux operating system besides Ubuntu 20.04 LTS, you should visit Install .NET on Linux Distributions | Microsoft Docs documentation and act accordingly. Installing the SDK on Ubuntu 20.04 LTS on x64 architecture will need:

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb

sudo apt-get update
sudo apt-get install -y apt-transport-https
sudo apt-get update
sudo apt-get install -y dotnet-sdk-6.0

This step installed all the necessary dependencies, and we are ready to go.

Case 2 - CPU on your machine IS NOT x64 (e.g. Raspberry Pi)

There is a slightly different setup for Raspberry Pi because it has an ARM64 CPU. We need to install the Runtime manually or with a script - I prefer the script way (If you are a bit sceptical about the file, you can check the documentation here.

curl -sSL https://dot.net/v1/dotnet-install.sh | bash /dev/stdin -c 6.0

Wait for the process to complete, and the .NET SDK will put files in your current directory ~/.dotnet. To have dotnet in our PATH for a simple resolution, we can run:

sudo ln -s "$HOME/.dotnet/dotnet" "/usr/bin/dotnet"

If you don't want to add dotnet to the PATH, you will need to invoke the dotnet command from the directory $HOME/.dotnet/dotnet.

Check if dotnet installation completed without any issues:

dotnet --version

It is unnecessary to reboot your machine, but as I use Raspberry Pi as a DHCP server, I noticed some problems obtaining the IP address, but reboot solves it.

Since we installed .NET 6 SDK manually (using the script), we need to ensure that we have the required dependencies for the .NET by running:

 sudo apt install libc6 libgcc1 libgssapi-krb5-2 libicu66 libssl1.1 libstdc++6 zlib1g

Our machine is now ready to build and run .NET 6 applications, but this is still not the end. We need to let GitHub Actions know where to deploy, build, and run the code. Back to the GitHub repository.

Step 2 - Create a self-hosted runner

You can add self-hosted runners to a single repository. To add a self-hosted runner to a user repository, you must be the repository owner.

Navigate to the main page of the repository
Click Settings
Select Actions from the left sidebar
Select Runners submenu
Click on New self-hosted runner.

In the next step, you need to choose what type of operating system and CPU architecture. If you are using Raspberry Pi 3 and higher versions, you probably have ARM64, but anyways, you can check that quickly by running:

uname -m

If you see something like aarch64, it means an ARM64 CPU. If you are on VPS, it will probably show something like x86_64, which means your machine is running on Intel's 64-bit CPU. GitHub will provide you with the information needed to install the active-runner on your device. Follow it to the tee, but you can skip running the script with ./run.sh and Using your self-hosted runner part because we will cover it later.

The final screen should look like the image below. If you get, unexpected errors like An error occurred while sending the request or Resource temporarily unavailable, retry the command that caused it.

Run a self-hosted runner as a background service

In the previous configuration, we said to skip running ./run.sh because we want it to be a background process. Maybe someone likes it or uses it because of some security considerations, and it's OK, but the background process is my preferred way.

Action-runners come with an option for installing runners as a service in the background. It will enable it by default, so the next time your machine restarts, the service will be up and running.

sudo ./svc.sh install
sudo ./svc.sh start

Step 3 - Setup the application directory

We created the action runner that will listen for the changes from GitHub Actions. Now we need to create a directory that will hold the application files. You can set it anywhere available, but I like to keep everything in /var/www/<name_of_the_app>.

sudo mkdir /var/www/sample-app
# I will give the directory ownership to the user `ubuntu` and its group
sudo chown ubuntu:ubuntu /var/www/sample-app

Now we have a directory where we can output applications build assets.

Step 4 - Create a GitHub Actions workflow

We created a self-hosted runner on our machine; now, we need to create a GitHub Action workflow to perform our deployment.

Go to your GitHub repository
Select the Actions tab
Click on set up a workflow yourself.

GitHub will create a prepopulated file with a default filename which you can change. Replace the content of the file with the following:

name: CI

# Controls when the workflow will run
on:
  # Triggers the workflow on push request event for the master branch
  push:    
    branches: [master]

jobs:
  deploy:
    # Our previously created self-hosted runner
    runs-on: self-hosted

    strategy:
      matrix:
        dotnet: ["6.0.x"]

    # A sequence of tasks that will execute as part of the job
    steps:
      # Checks out repository so our job can access it
      - uses: actions/checkout@v2
      - name: Setup .NET Core SDK ${{ matrix.dotnet-version }}
        uses: actions/setup-dotnet@v1.7.2
        with:
          dotnet-version: ${{ matrix.dotnet-version }}

      - name: Install dependencies
        run: dotnet restore

      - name: Build
        run: dotnet build --configuration Release --no-restore

      # We will output publish files to the folder we previously created
      - name: Publish
        run: dotnet publish -c Release -o /var/www/sample-app

Once we commit the file, GitHub will place it in the .github/workflows/ folder in the root of our repository, and the workflow process will start right away (if we did everything correctly). You can see the output of the workflow in the Actions tab:

Step 5 - Create a background service for the application

You probably already figured out that we don't have the dotnet xxxx.dll command in our GitHub Actions workflow YAML file. Of course, there is a reason. When we start the application using the dotnet command, it is triggered only as a foreground process, and in our case, the GitHub Actions workflow will wait as long as the application is running and won't complete.

We could use something like nohup, but it would create a new process every time we run the application (not to mention that we didn't specify the port on which it should run). Also, if our machine reboots, our application will be down until we start it manually.

# It may look cool, but we don't want this
nohup dotnet SampleApp.dll > /dev/null 2>&1 &

While there might sometimes be better tools for the job, always consider using existing software first, and writing our service can give us a level of flexibility.

To solve the issue, we will create a Linux user service that will:

Run our application after reboot
Automatically restart the application if something internally happens and the application fails
Ability to restart the application from a deployment script (GitHub Actions)

We create services in the /etc/systemd/system directory, but we would need elevated privileges to control them by default. Sure, there are ways to run systemd services without using sudo by tampering with sudoers files or configuring PolKit, but we will approach it differently. In the end, we want to give our GitHub Actions user the ability to run and restart the service without using sudo or PolKit configurations.

We must locate user services in the ~/.config/systemd/user/ directory. Let's create it:

# Using -p argument to create the whole path if it does not exist
mkdir -p ~/.config/systemd/user/

First, you need to enable lingering for the user on which we installed GitHub Actions runner. In our case, that is the ubuntu user, and it is necessary because it allows users who are not logged in to run long-running services. Otherwise, the user services will start on user login instead of boot.

loginctl enable-linger ubuntu

Next, we need to create a systemd unit file for the user (the service). We can name it however we want, but I prefer logical names, such as name-of-the-application.service. Place it under the user's systemd instance.

nano ~/.config/systemd/user/sample-app.service

There are a couple of crucial things for which we prepared in previous steps, and we need to make sure we configure them correctly. The final configuration of sample-app.service should look like this:

[Unit]
Description=Example .NET 6 Application

[Service]
# This is the directory where our published files are
WorkingDirectory=/var/www/sample-app
# We set up `dotnet` PATH in Step 1. The second one is path of our executable
ExecStart=/usr/bin/dotnet /var/www/sample-app/SampleApp.dll --urls "http://0.0.0.0:5000"
Restart=always
# Restart service after 10 seconds if the dotnet service crashes
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=sample-app-log
# We can even set environment variables
Environment=ASPNETCORE_ENVIRONMENT=Production
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false

[Install]
# When a systemd user instance starts, it brings up the per user target default.target
WantedBy=default.target

We set --urls "http://0.0.0.0:5000" in ExecStart for Kestrel to listen on a specific port to test our application. I usually don't use Kestrel as an edge web server, but in a reverse proxy configuration with Nginx. For this tutorial, we will use Kestrel directly for brevity.

If you are interested in the reverse proxy configuration, the following link describes it in great detail.

After we are sure that we have the correct configuration, we can save the file with Ctrl+X, Y. Let's reload systemd and enable our service to run even after the reboot.

# Reload `systemd`
systemctl --user daemon-reload

# Enable the service at the next boot
systemctl --user enable sample-app.service

# Start it right away
systemctl --user start sample-app.service

It is pretty easy to create a service on Linux, but it does not mean that it is the best solution. This tutorial didn't cover security considerations regarding the usage of systemd services.

Step 6 - Update the GitHub Actions workflow file

Once we have our application service ready, we can update the GitHub Actions configuration to include our newly created service to run our application every time we push new code to the deployment branch.

Edit the GitHub Actions workflow file created in Step 4 under the <repository>/.github/workflows/ folder. As the last step of our workflow, we will add:

      - name: Restart the app
        run: |
          export XDG_RUNTIME_DIR=/run/user/$(id -u)
          systemctl --user restart sample-app.service

Our final workflow YAML file should look like this:

name: CI

# Controls when the workflow will run
on:
  # Triggers the workflow on push request event for the master branch
  push:    
    branches: [master]

jobs:
  deploy:
    # Our previously created self-hosted runner
    runs-on: self-hosted

    strategy:
      matrix:
        dotnet: ["6.0.x"]

    # A sequence of tasks that will execute as part of the job
    steps:
      # Checks out repository so our job can access it
      - uses: actions/checkout@v2
      - name: Setup .NET Core SDK ${{ matrix.dotnet-version }}
        uses: actions/setup-dotnet@v1.7.2
        with:
          dotnet-version: ${{ matrix.dotnet-version }}

      - name: Install dependencies
        run: dotnet restore

      - name: Build
        run: dotnet build --configuration Release --no-restore

      # We will output publish files to the folder we previously created
      - name: Publish
        run: dotnet publish -c Release -o /var/www/sample-app

      - name: Restart the app
        run: |
          export XDG_RUNTIME_DIR=/run/user/$(id -u)
          systemctl --user restart sample-app.service

Once you deploy the task, you can track its status in Actions tab on GitHub repository. Now you can access your web application over the assigned URL and port.

Final words

I hope this text will help you complete that 2% of "what’s left of the project" and make deployment easier. If you find anything unclear, please comment here, and we will try to solve it if possible. Thank you for reading!

Bring Your Home Network Anywhere For Free - Home VPN with WireGuard on Raspberry Pi + Pi-hole (Ubuntu Server 20.04 LTS)

Amel Spahić — Fri, 31 Dec 2021 12:32:40 +0000

In the previous blog post, I talked about setting up Ubuntu Server 20.04 LTS and Pi-hole DNS on Raspberry Pi. You can go through the process step by step following Block Ads, Tracking, and Telemetry With Pi-hole on Raspberry Pi (Ubuntu Server 20.04 LTS).

Having Pi-hole set up on our home network, we will have a much better internet browsing experience without ads and better control of available resources (if any). Also, maybe you have a network-attached storage (NAS) in your network and would want to have access from anywhere, or you just want a safe browsing experience when connected to public WiFis.

The setup above is limited only to your home network, and after a couple of days of browsing, you will think - why can't I bring this network setup wherever we go!? Well, YOU CAN. A logical presumption would be to have a way to connect to our home network from anywhere and browse through it. Even when you connect from the other side of the world.

Virtual Private Network

Virtual Private Network (VPN) allows us to connect our devices to another network over the internet in a secure manner. We can browse the internet using other computers' (server) internet connection.

I am sure you came across internet ads for paid services like ExpressVPN, NordVPN, Surfshark, etc. They are awesome without a doubt, you can fake your device's IP location and use some geographically limited services like Netflix, but it won't get you to your home network. And you have to pay for it. All VPNs use VPN protocols to create and secure your connection, so why shouldn't you, for your needs?

WireGuard or OpenVPN

Two most popular VPN protocols used today are WireGuard and OpenVPN. There is no specific reason why I choose one over the other, but it is said that WireGuard is much faster than OpenVPN and it consumes around 15% less data, handles network changes better and appears to be just as secure (I don't know who said it).

WireGuard (or OpenVPN) on Raspberry Pi

We could go through the manual installation instructions for WireGuard, but there is a great tool, PiVPN which allows us to install the desired VPN very easily.

curl -L https://install.pivpn.io | bash

The process will use sudo and install the necessary dependencies. Just wait for it to do its job. After installing the necessary packages, you will be prompted with graphical options:

We previously talked about setting up a static IP address on Ubuntu Server 20.04. PiVPN won't configure static IP for us because we are not using Raspbian OS for our Raspberry Pi.

Just accept default options, and be sure to select the WireGuard option when prompted.

You can change the default WireGuard port if necessary but have in mind that you will need it later, so make sure you remember it (I will use the default option, port 51820)

If you have a Pi-hole installation, PiVPN will detect it and ask if you want to use it as a DNS.

In the next steps, you will be prompted to use Public IP or DNS. Choose your public IP address.

If your ISP provides you with a dynamic IP address, there is a solution in the next post. For now, continue with this article.

Continue with the process and accept unattended upgrades to the server.

Just follow the process and accept to reboot Raspberry Pi after the installation, so everything is set up.

If you use Pi-hole as a DHCP server, you won't have an internet connection while Raspberry Pi is rebooting.

Port Forwarding

To be able to connect to your Raspberry Pi VPN server, we need to set up a port forwarding option on your router. I have Technicolor CGA2121, but you can find that on every router, under settings (or advanced settings, usually under the Application & Gaming option).

Adding VPN Client

To add a new VPN client user, use the integrated PiVPN command:

pivpn add

Choose your client name and hit ENTER.

You may have a warning to Run 'systemctl daemon-reload' to reload units, so just do it.

Now your client is ready to connect. You can find installation files here for different operating systems.

For Android and iOS devices, there is a WireGuard application on PlayStore/AppStore, so download it. To quickly set up WireGuard VPN, from your Raspberry Pi run:

pivpn -qr

You will have a QR code on the screen which you can read from your mobile phone to set it up.

Now when you leave your home network, you are always a flip of the switch away from it.

Pi-hole DNS Troubleshooting

If you installed PiVPN before Pi-hole, edit the PiVPN configuration with:

$ sudo nano /etc/pivpn/wireguard/setupVars.conf

Remove the pivpnDNS1=[...] and pivpnDNS2=[...] lines
Add this line pivpnDNS1=192.168.0.50 (your Pi-hole IP might be different) to point clients to the Pi-hole IP
Save the file with Ctrl+X, Y and exit
Run pihole -a -i local to tell Pi-hole to listen on all interfaces

Dynamic IP Address

If you are lucky enough or you are not sorry to pay for the static IP address, you can skip this part. Otherwise, here you can read how to Set up Dynamic DNS for Dynamic IP Addresses at Home for free.

Final Words

I hope this tutorial will help you set up your VPN communication and bring even more privacy, security, and comfort while browsing the internet.

Block Ads, Tracking, and Telemetry With Pi-hole on Raspberry Pi (Ubuntu Server 20.04 LTS)

Amel Spahić — Fri, 31 Dec 2021 12:15:59 +0000

While internet advertising is a significant source of revenue for your favorite websites, some individuals want to prevent it for a variety of reasons, including performance or privacy concerns. You could install blocking software on each of your devices, but the most efficient method is to use Pi-hole to establish a server that filters all of your web traffic at the local network level. #pihole

The best explanation on what is Pi-hole you can find on the Pi-hole® official website.

The Pi-hole® is a DNS sinkhole that protects your devices from unwanted content, without installing any client-side software.

Easy-to-install: our versatile installer walks you through the process and takes less than ten minutes
Resolute: content is blocked in non-browser locations, such as ad-laden mobile apps and smart TVs
Responsive: seamlessly speeds up the feel of everyday browsing by caching DNS queries
Lightweight: runs smoothly with minimal hardware and software requirements
Robust: a command-line interface that is quality assured for interoperability
Insightful: a beautiful responsive Web Interface dashboard to view and control your Pi-hole
Versatile: can optionally function as a DHCP server, ensuring all your devices are protected automatically
Scalable: capable of handling hundreds of millions of queries when installed on server-grade hardware
Modern: blocks ads over both IPv4 and IPv6
Free: open-source software which helps ensure you are the sole person in control of your privacy

This post should be as simple as possible and for everyone with a couple of bucks for the Raspberry Pi with an ethernet capability. An even better idea for those with a dust-collecting RPi somewhere in the room.

This post will use Raspberry Pi 4 with 4GB of RAM as an example, but it really does not matter since Pi-hole as long as you have a device with at least 512 MB of RAM and 2 GB of space.

Requirements

An internet connection
A computer with a microSD card reader
Raspberry Pi with ethernet capabilities
A microSD card (at least 8GB, 16GB is recommended)
A monitor with HDMI
A micro HDMI cable (if you have Raspberry Pi 4)
A USB keyboard

Process

Installing Operating System

We need to set up an operating system (OS) on our microSD card which will be used as persistence on Raspberry Pi. I will be using Ubuntu Server 20.04 LTS x64 for that purpose, but you can use any other Ubuntu version or Raspberry Pi OS.

The easiest way to format and install the required OS on a microSD card is by using the existing tool which can be downloaded from Raspberry Pi Imager.

Insert the microSD card into your computer, and start already installed Raspberry Pi Imager.

From the initial screen, select CHOOSE OS and get your preferred operating system. Ubuntu can be found under the Other general-purpose OS group.

Select the image, open the SD Card menu, and select the inserted microSD card. Click WRITE and wait a couple of minutes for the operating system to be downloaded, installed, and verified on microSD. Once the process is complete, you will be notified that you can safely remove the microSD card from the slot.

Booting Up

Make sure your Raspberry Pi is off before you insert the microSD card into the dedicated slot. Connect the ethernet cable to the router, keyboard, monitor, and finally Raspberry Pi power supply. Wait a bit and you will see the operating system booting up.

Ubuntu Server does not have graphical installation, but the process is pretty straightforward.

Once the system is booted up, you will see a login prompt, but don't do anything for now. Wait for a couple of minutes for everything to set up and then perform the login. The default username and password for Ubuntu Server 20.04 LTS is:

username: ubuntu
password: ubuntu

After the successful login, it will ask for a new password, so be sure to create a secure password and remember it. That is your sudo password which grants you administrative privileges, and it will be used frequently.

Update the OS

I know we did the clean install of the operating system, but it does not mean that the Raspberry Pi Imager has always up-to-date repositories. To update your system and packages with the latest patches, just run:

sudo apt update

sudo apt upgrade

If the Linux kernel is updated, you will need to reboot the system. You can do it with:

sudo reboot

Static IP

While we are at it, we should set up the static IP address for our device right from the start. It will be necessary for the Pi-hole later.

Ubuntu Server 20.04 LTS uses Netplan for the network configuration and can be found in /etc/netplan/ directory. There should be a file already in one of these forms:

01-netcfg.yaml
01-network-manager-all.yaml
50-cloud-init.yaml

Otherwise, if you cannot find the configuration file, you can try to generate a new one, executing:

$ sudo netplan generate

To configure a static IP address, you need to change the configuration in the file. This is how mine looks like:

# This file is generated from information provided by the datasource.  Changes
# to it will not persist across an instance reboot.  To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        eth0:
            dhcp4: no
            addresses: [192.168.0.50/24]
            gateway4: 192.168.0.1
            nameservers:
              addresses: [8.8.8.8,8.8.4.4]
    version: 2

This configuration indicates that the static IP address will be 192.168.0.50 and the gateway (routers') IP address is 192.168.0.1. Please check your router IP address so you can change values accordingly.

Remote Access

I don't like having Raspberry Pi connected to anything except the network, so I prefer using Secure Shell (SSH) to operate with Raspberry Pi. If you have a device on the same network, it is pretty easy. Don't disconnect the peripherals yet, we need some basic information. By default, SSH on Ubuntu Server obtained from Raspberry Pi Imager repository has SSH enabled. We can check this by checking the status with the command sudo systemctl status ssh:

$ sudo systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-12-25 12:54:10 UTC; 12min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 1733 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 1777 (sshd)
      Tasks: 1 (limit: 4435)
     CGroup: /system.slice/ssh.service
             └─1777 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Dec 25 12:54:09 ubuntu systemd[1]: Starting OpenBSD Secure Shell server...
Dec 25 12:54:10 ubuntu sshd[1777]: Server listening on 0.0.0.0 port 22.
Dec 25 12:54:10 ubuntu sshd[1777]: Server listening on :: port 22.
Dec 25 12:54:10 ubuntu systemd[1]: Started OpenBSD Secure Shell server.
Dec 25 12:54:36 ubuntu sshd[1966]: Accepted password for ubuntu from 192.168.0.183 port 63829 ssh2
Dec 25 12:54:36 ubuntu sshd[1966]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)

If our SSH has the status Active: active (running), we are ready to continue. We need the IP address of the Raspberry Pi which we can obtain by running the next command (it should be a static IP address that we specified earlier):

$ hostname -I
192.168.0.50

Since we already know our username is ubuntu, we can now safely disconnect everything from the Raspberry Pi except the ethernet cable.

I use PowerShell to connect from my Windows 11 machine, but you can use whatever command-line program you prefer.

Don't forget you need to be on the same network

To log in remotely, use previously obtained information about your hostname and IP address. The command format is ssh hostname@IPaddress

If you like this PowerShell look, there is a great blog post from @Scott Hanselman HERE

Unable to resolve host

If you get the following error

sudo: unable to resolve host ubuntu: No address associated with hostname

It means that your hostname is not configured for the IP address. To solve it, add the hostname in /etc/hosts file. To find the hostname, you can check it in the /etc/hostname file using cat command:

$ cat /etc/hostname
ubuntu # <-- this is your hostname

Now you can edit your hosts file

$ sudo nano /etc/hosts

Add your hostname, obtained with the previous command, to the row where localhost is defined:

127.0.0.1 localhost heregoesyourhostname # <- instead of this

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Pi-hole Installation

Installation of Pi-hole is pretty straightforward, and you can do it from your computer if you obtained information for the SSH connection.

$ curl -sSL https://install.pi-hole.net | bash

You will be prompted a couple of times to select configuration:

Every screen contains information about the configuration installation.

We already set up a static IP address, so we can just continue with the process. For the rest of the process, you can choose default options (including Web admin interface). Once we complete all the steps, we will have the final screen with the Web interface password.

You can access to Web admin interface by navigating to the IP address we specified through the browser

#<predefined static IP addres>/admin
192.168.0.50/admin

Change Password

Change your Pi-hole Web interface password using:

$ sudo pihole -a -p

Router Configuration

Once we have everything installed, it is necessary to tell our router to use Pi-hole as DNS. For this process, we need to have access to the network router (to which Raspberry Pi is connected to). A lot of ISPs allow this kind of configuration, but unfortunately, mine doesn't.

I have Technicolor CGA2121 DOCSIS 3.0 Wireless Gateway, and no option to manually change upstream DNS for DHCP connection (I don't have a static IP from my ISP). Thankfully, there is a way to override that behavior using Pi-holes' integrated DHCP server. For that purpose, we need to disable the DHCP server on the router and turn on DHCP on Pi-hole.

To disable the routers' DHCP, we need to log in and disable it manually.

Blocklists

Pi-hole comes with an optional blocklist that you could select during the installation process, and it is usually sufficient for most people.

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

This blocklist is well-maintained and provides effective blocking without interfering with routine operations. While this may be sufficient for many, users frequently discover that they want to add their own custom lists to improve blocking capabilities.

Blocklist Collections

The Firebog, DeveloperDan and you can check Reddit or Google Search.

The easiest way to update the blocklist is through the Web interface. From the blocklist websites above, select lists that you want to use (I suggest reading a bit about them, so you don't come across some unwanted behavior while browsing).

To update the blocklist on your Pi-hole, you need to paste selected lists into the Adlists group management tab.

After that, we need to update the Pi-hole Gravity.

Tip

Strange things will happen while browsing, for sure, especially if you put a lot of blocklists. To check if that something is because of the Pi-hole configuration, there is an option to disable the Pi-hole for some time.

While the Pi-hole is disabled, you can try what you intended, and after the countdown, everything will go back as it was before.

Final words

I hope this post helped you set up the Pi-hole for a better internet browsing experience, and I'm sure this will be one of the best improvements you could make for your home or business network.

Happy browsing without ads!

P.S. Read the documentation, please.