DEV Community: Adewale Azeez

Stateful React Pages using URLSearchParams and Local/Session Storage

Adewale Azeez — Thu, 04 Jan 2024 22:44:19 +0000

Note: This article speaks to creating react web pages that can retain their active state after a refresh or link share through the use of URL params and react state.

Table of content

What is a stateful page
Advantages of a stateful page
Methods to a stateful page madness
- Using URLSearchParams
  - Setup the URLSearchParams
  - Retrieving URL values
  - Adding or Modifying the URL values
  - Retrieving values from URLSearchParams to restore state
- Using Local or Session Storage
  - Adding value to storage
  - Retrieving values from the Storage API
  - Removing or modifying values in storage
  - Retrieving values from Storage to restore state
- Alternatives state storage methods
Helper implementations for URLSearchParams
- Functions to manage URL search params
- Usage Examples
  - Add some params entries
  - Invoke a callback after the URL has been updated
  - Delete an entry from the URL params
  - URLSearchParams to JSON object
  - URLSearchParams to URL search string
Glossary
References

What is a stateful page

First, what does stateful mean? According to redhat stateful allows the users to store, record, and return already established information and processes over the internet. "Stateful applications and processes allow users to store, record, and return to already established information and processes over the internet". In the context of this article, stateful refers to the capability to store, record, and return information to a webpage after refreshing the page or the copied link to the page is open in a different browser or device.

A stateful page will be able to render the UI and components as the last time the page information and processes are recorded even if the page is open in another browser or device.

Advantages of a stateful page

An advantage of making your pages and app stateful is giving the user the ability to resume on the page from their last interaction.
Another reason to make the page stateful is to be able to share a URL with another user and they can get the exact presentation as shown on your end.
For a page with a lot of data presentation it makes it sane to have a link to that presentation as it is, so the link can be revisited and still get your page.
A stateful page can also allow auto-fill of form on a page even if the page refresh or internet connection is loss, hence the user can resume the filling of the from with their previously entered values still on the page.

Methods to a stateful page madness

There are various ways to make a webpage stateful, two of the methods are storing the page state information in the URL params or using a local storage to store the page's current information. Both methods are described below:

Using URLSearchParams

React Router DOM provides an intuitive way of managing the URL search value of a webpage, it provides the useSearchParams hook that allows accessing the webpage URL search values and also changing the values without reloading the active webpage.

The useSearchParams hook provides the two variables the first for retrieving the URL search values, and the other for adding or modifying the URL search values.

Setup the URLSearchParams

The useSearchParams hook accepts a default search params value in the form of a string, Record, or URLSearchParams which will be useful in retrieving data from the webpage URL

Install the react-router-dom package

npm i react-router-dom

Initialize in your components

//...
import { useSearchParams } from "react-router-dom";
//...
const [searchParams, setSearchParams] = useSearchParams(window.location.search);

Retrieving URL values

An example is a webpage with the search params value: ?page=1&name=user-one. The return value is either a string or null for a non-existence key.

const page = searchParams.get("page"); // 1
const name = searchParams.get("name"); // user-one
const size = searchParams.get("size"); // null

Adding or Modifying the URL values

The second value from the useSearchParams hooks the URL search values can be modified to add a new search param or remove or modify existing search params.

searchParams.delete("page"); // => ?name=user-one
searchParams.set("size", 10); // => ?name=user-one&size=10
searchParams.append("status",["APPROVED", "REJECTED"]); // => ?name=user-one&size=10&status=APPROVED&status=REJECTED

Retrieving values from URLSearchParams to restore state

// UrlParamsExamplePage.tsx
import React from 'react';
import { useSearchParams } from "react-router-dom";
import { Helpers } from './Helpers';

function UrlParamsExamplePage() {
  const [searchParams, setSearchParams] = useSearchParams(window.location.search);
  const [page, setPage] = React.useState<number>(parseInt(searchParams.get("page") ?? "0"));

  return (
    <div style={{ display: "flex", flexDirection: "column", width: "fit-content" }}>
      <span>Page is {page}</span>
      <button type='button' onClick={() => Helpers.updateSearchParams("page", page+1, setSearchParams, () => {
        setPage(page+1);
      })}>Next Page</button>
      <br/>
    </div>
  );
}

export default UrlParamsExamplePage;

Full source code at UrlParamsExamplePage.tsx

Using Local or Session Storage

The browser local or session storage can also be used to store a webpage state. The Browser's local storage and session storage implementations provide API to store, retrieve, delete and modify data for a website. Local storage is different from session storage in the sense that session storage only retains the stored data in that browser tab while local storage retains the data across tabs for that website, but both share identical APIs.

The local and session storage does not require any setup as it is readily available in the JavaScript global window object.

Adding value to storage

const storage = window.localStorage ?? window.sessionStorage;
storage.setItem("page", "1");
storage.setItem("name", "user-one");

Retrieving values from the Storage API

The return value is either a string or null for a non-existence key.

const storage = window.localStorage ?? window.sessionStorage;
const page = storage.getItem("page"); // 1
const name = storage.getItem("name"); // user-one
const size = storage.getItem("size"); // null

Removing or modifying values in storage

The storage API provides the removeItem method to remove a key value, and also the clear method to remove all the entries for that website.

const storage = window.localStorage ?? window.sessionStorage;
storage.removeItem("page"); // delete a single entry
storage.clear();            // remove all entries for the website

Retrieving values from Storage to restore state

// StorageExamplePage.tsx
import React from 'react';

function StorageExamplePage() {
  const storage = window.localStorage ?? window.sessionStorage;
  const [page, setPage] = React.useState<number>(parseInt(storage.getItem("page") ?? "0"));

  return (
    <form style={{ display: "flex", flexDirection: "column", width: "fit-content" }}>
      <input placeholder="Name" defaultValue={storage.getItem("name") ?? ""} onChange={(e: any) => {
        storage.setItem("name", e.target.value);
      }}/>
      <span>Page is {page}</span>
      <button type='button' onClick={() => {
        storage.setItem("page", ""+(page-1));
        setPage(page-1);
      }}>Prev Page</button>
      <button type='button' onClick={() => {
        storage.setItem("page", ""+(page+1));
        setPage(page+1);
      }}>Next Page</button>
    </form>
  );
}

export default StorageExamplePage;

Full source code at StorageExamplePage.tsx

Alternatives state storage methods

Other alternatives to store data to be used to restore React webpages to React native mobile page states are.

Cache: The browser cache can store data with expiry time and can also share the data with other websites if needed.
Mobile - Async Storage: React native provides an async storage library that can be used to store data between app closing and reopening.
Mobile - SQLite: Since the mobile app has access to IO operation on the persistence storage, compact DB like SQLite will be helpful in quick storage and retrieval of data for page state restoration.
Mobile - File IO: Another alternative for mobile is to use the mobile native file access capability to create a custom file on the device to store state data, and read the file to retrieve the state data.

Helper implementations for URLSearchParams

Typescript's functions to manage URL params and examples on how to use the functions

Functions to manage URL search params

import { SetURLSearchParams } from "react-router-dom";
// ...
export type NoseurObject<T> = { [key: string]: T; };
export type URLSearchParamsValue = string | string[] | number | number[] | undefined | null;
// ...
function updateSearchParams(key: string, value: URLSearchParamsValue, setSearchParams: SetURLSearchParams | URLSearchParams, cb?: Function | undefined) {
    const fun = (prev: URLSearchParams) => {
        if (value === undefined || value === null) {
            prev.delete(key);
        } else if (Array.isArray(value)) {
            prev.delete(key);
            for (const v of value) prev.append(key, `${v}`);
        } else {
            prev.set(key, `${value}`);
        }
        return prev;
    };
    if (setSearchParams instanceof URLSearchParams) fun(setSearchParams);
    else setSearchParams(fun);
    cb && cb();
}

function normalizeUrlParams(urlSearchParams: URLSearchParams, keyLookupMap: NoseurObject<string> = {}, extraParams: NoseurObject<string | number> = {}) {
    const params: NoseurObject<any> = {};
    Object.keys(Object.fromEntries(urlSearchParams)).forEach((key: string) => {
        const lookupKey = keyLookupMap[key] ?? key;
        const values = urlSearchParams.getAll(key);
        if (values.length) {
            params[lookupKey] = values.length > 1 ? values : values[0];
        }
    });
    Object.keys(extraParams).forEach((key) => params[key] = extraParams[key])
    return params;
}

function urlParamsToSearch(urlSearchParams: URLSearchParams, only?: string[], includes?: NoseurObject<URLSearchParamsValue>) {
    only && only.length && Object.keys(Object.fromEntries(urlSearchParams)).forEach((key: string) => {
        if (!only.includes(key)) urlSearchParams.delete(key);
    });
    if (includes) {
        Object.keys(includes).forEach((key: string) => updateSearchParams(key, includes[key], urlSearchParams));
    }
    return urlSearchParams.toString();
}

Usage Examples

import { useSearchParams } from "react-router-dom";
//...
const [searchParams, setSearchParams] = useSearchParams(window.location.search);

Add some params entries

updateSearchParams("page", 1, setSearchParams);
updateSearchParams("name", "user-one", setSearchParams);
updateSearchParams("status", ["APPROVED", "REJECTED"], setSearchParams);

Invoke a callback after the URL has been updated

updateSearchParams("page", 2, setSearchParams, () => {
    // e.g. fetch data from page 2
});

Delete an entry from the URL params

updateSearchParams("name", null, setSearchParams);
updateSearchParams("name", undefined, setSearchParams);

URLSearchParams to JSON object

let search;
search = normalizeUrlParams(searchParams); // { "page": 2, "status": ["APPROVED", "REJECTED"] }

// with key lookup map
search = normalizeUrlParams(searchParams, { "page": "users_page" }); // { "users_page": 2, "status": ["APPROVED", "REJECTED"] }

// with extra params
search = normalizeUrlParams(searchParams, {}, {"size": 20, "name": "one"}); // { "page": 2, "status": ["APPROVED", "REJECTED"], "size": 20, "name": "one" }

URLSearchParams to URL search string

let searchString;
searchString = urlParamsToSearch(searchParams) // page=2&status=APPROVED&status=REJECTED&name=user-one

// only selected entries
searchString = urlParamsToSearch(searchParams, ["page", "name"]) // page=2&name=user-one

// include extra entry values
searchString = urlParamsToSearch(searchParams, [], {"size": 20}) // page=2&status=APPROVED&status=REJECTED&name=user-one&size=20

Glossary

API: Application Programming Interface, a method, routine, or procedure exposed by a component to allow usage or modification of its functionalities.
Stateful: The capability of a presentation or system to store, record, and return the system to its previous state.
Stateless: The absence of the capability of a presentation or system to retain information from previous interactions.

References

The sample project of this article is hosted at https://quickutils.github.io/stateful-ui/ and the source code at https://github.com/quickutils/stateful-ui

Logging in your API

Adewale Azeez — Wed, 22 Feb 2023 08:19:46 +0000

Logging in your API

This article will highlight some methods and best practices you can adopt in your service for proper logging and error tracing.

Table of content

What is logging
Choosing your log framework
What and when to log
Try and catch?
- Exception Handling
- Logging Exception For Some Weakly-Typed Language
- Almost say no to exception message
- Logging your exception
Data Safety
Cloud Logging
Glossary
References

What is logging

As copied from Wikipedia Logging is the act of keeping a log of events that occur in a computer system, such as problems, errors or just information on current operations. These events may occur in the operating system or in other software.

Choosing your log framework

Most API frameworks like spring, .NET, laravel comes equiped with
robust logging packages that you can use in your project. The list
below highlights some logging framework you can use in your services.

Java -> Logback, Log4j2, JDK (Java Util Logging), Slf4j, e.t.c.
C# -> Built in Logger, Log4Net, Serilog, NLog, e.t.c.
PHP -> Built in Logger, Monolog, Analog, KLogger, Log4PHP
C/C++ -> liblogax, Boost.Log, log4c, e.t.c.
NodeJS -> Pino, Winston, Bunyan, Npmlog, e.t.c.

What and when to log

When it come to loggin no information is too much to be logged and
there is no specific place to place a log just ensure that the logs
are properly classified.

There are 6 basic log classification

TRACE
DEBUG
INFO
WARN
ERROR
FATAL

When logging basic information that does not change on each call it can be classified as INFO log, if a logic or operation is likely going to cause an error in the service or a non fatal error occur then it can be classified as WARN log. For operations that cause an error that
breaks or stop the flow of an operation then it can be classified as ERROR log.

In the case of FATAL log the service is expected to end or shutdown,
a situation when a FATAL log is required is when the memory of the
system is completely used or corrupt a shutdown is expected cause it
will be almost impossible for the service to continuing function
properly after that.

Try and catch?

Exception plays an important part in logging as the error in the
exception catch block is logged to give the user an idea of what error
occur and possibly why it occur.

In strongly type language like Java, C# the try and catch logic
is robust enough to allow handling specific exceptions but in weakly
typed language like Javascript, Python, developers can become lazy in
using the language provided type check to determine what error is thrown
which in turn encorages generic and un-helpful logging. Then there is
languages like PHP that allows both strong and weak typing in special
case like try...catch block, for such languages always go for the strong
typing feature for safety and better error tracing.

Exception Handling

If you have a function or method that perform multiple operations, the
try and catch block should not be at the highlevel of function rather it should be at the last logic level such that when the exception is logged
it can easily be traced to the logic point.

An example of this is a method that accept a POST body, converts the body json string to object then store the object in the database.

BAD

public ResponseEntity<?> store(@RequestBody String body) {
    try {
        User user = mapper.readValue(body, User.class); // can throw JsonException
        // some long operation ...
        repository.save(user); // can throw ConstraintException
        // some other long operation ...
        return new ResponseEntity<>(HttpStatus.OK);
    } catch (Exception ex) {
        return new ResponseEntity<>(HttpStatus.INTERNAL_SERVER_ERROR);
    }
}

GOOD

public ResponseEntity<?> store(@RequestBody String body) {
    User user;
    try {
        User user = mapper.readValue(body, User.class); // can throw JsonException
        // some long operation ...
        repository.save(user); // can throw ConstraintException
        // some other long operation ...
        return new ResponseEntity<>(HttpStatus.OK);
    } catch (JsonException ex) {
        log.error("Error occur while processing body payload", ex);
        return new ResponseEntity<>(HttpStatus.BAD_REQUEST);
    } catch (ConstraintException ex) {
        log.error("Error occur while persisting to database", ex);
        return new ResponseEntity<>(HttpStatus.INTERNAL_SERVER_ERROR);
    }
    return new ResponseEntity<>(HttpStatus.OK);
}

EFFICIENT

public ResponseEntity<?> store(@RequestBody String body) {
    User user;
    try {
        user = mapper.readValue(body, User.class); 
    } catch (JsonException ex) {
        log.error("Error occur while processing body payload", ex);
        return new ResponseEntity<>(HttpStatus.BAD_REQUEST);
    }
    // some long operation ...
    try {
        repository.save(user);
    } catch (ConstraintException ex) {
        log.error("Error occur while persisting to database", ex);
        return new ResponseEntity<>(HttpStatus.INTERNAL_SERVER_ERROR);
    }
    // some other long operation ...
    return new ResponseEntity<>(HttpStatus.OK);
}

Now let analyse the trinity above, take note of the long operation
comment.

Bad: This prevent logging which error actually occur though it will catch any error even even those that we did not imagine occuring at the long operation blocks. It makes the service not debuggable as it will almost be difficult to trace the error down do where it actually occurs. And also a custom message cannot be returned to the caller.
Good: In the good block we take account of the two types of error that can occur but we have our other code logic (long operation) into tha try..catch block which is not neccessary.
Efficient: In this code block we only put a try catch around each logic which will spare the other logics (long operation) from being caught in the try..catch block.

Logging Exception For Some Weakly Typed Language

In languages that does not provide specific exception catching, it is
best practice to check what exception occur so as to produce proper log.

Take for example the code block below in Javascript and PHP, handles
general error that can occur in a try catch block.

Javascript

try {
    // operation 1 can throw JsonParseException
    // operation 2 can throw DatabaseException
} catch (err) {
    console.error("Error occur", err);
    return response.status(500);
}

try {
    # operation 1 can throw JsonParseException
    # operation 2 can throw DatabaseException
} catch ($e) {
    echo "Error occur: " .  $e->getMessage();
    return response->status(500);
}

The issue with the exception catch above is that, two type of exception is likely to occur but both of them are handled in the catch block and
if a response is expected from the caller a generic response is giving
which is not likely to be helpful to the caller. Therefore it important
to check the type/instance of that error to properly log the error and
send meaning ful response to caller.

Javascript

try {
    // operation 1 can throw JsonParseException
    // operation 2 can throw DatabaseException
} catch (err) {
    if (err instanceof JsonParseException) {
        console.error("Error while parsing Json body", err);
        return response.status(400);
    } else if (err instanceof DatabaseException) {
        console.error("Unable to persist to database", err);
        return response.status(500);
    }
    console.error("Unknown error occur", err);
    return response.status(500); 
}

try {
    # operation 1 can throw JsonParseException
    # operation 2 can throw DatabaseException
} catch ($e) {
    if ($e instanceof JsonParseException) {
        console.error("Error while parsing Json body", $e);
        return response->status(400);
    } else if ($e instanceof DatabaseException) {
        console.error("Unable to persist to database", $e);
        return response->status(500);
    }
    console.error("Unknown error occur", $e);
    return response->status(500); 
}

In strongly typed language like Java you can handle the two exceptions
in seperate catch blocks. And there is no need for the general
exception case since it unlikely to occur.

try {
    // operation 1 can throw JsonParseException
    // operation 2 can throw DatabaseException
} catch (JsonParseException ex) {
    log.error(ex.getMessage(), ex);
    return response.status(400);
} catch (DatabaseException ex) {
    log.error(ex.getMessage(), ex);
    return response.status(500);
}

Almost say no to eception message

When it comes to logging the exception's .getMessage(), or .message
method or fields are almost useless as a properper logging needs to
throughly output where error occur, why it occur and possible what
caused the error to happen. The message value of exceptions in this
case does not carry all the informations rather it only returns
what error occurs.

Hence it proper to log the entire stack trace of the exception.

Logging your exception

If the framework or logging framework used in your project provides
methods to log stack trace then you should use that instead of printing
the stack trace directly to your console. This allow the log
framework to categorize the exception and possibly sanitize the stack
trace before printing to the console.

For example In java spring rather than using ex.printStacktrace(),
invoke the error method in any of the provided log implementation e.g.

Logger log; // = any log implementation

try {
    //...
} catch (DatabaseException ex) {
    // bad
    ex.printStackTrace();
    // good
    log.error(ex.getMessage(), ex);
}

Data Safety

When it comes to logging it is important to be careful of the data
being sent to the log service or printed in the console. Information
like secrets, passwords should NEVER be logged rather these type
of information can be printed out as *** or in an encrypted value
that only the developer can decrypt the actual value.

A good example is a simple User class in spring if there is a need to
print out the class field ensure the secrets/password fields are
hidden or encrypted in the log.

class User {
    String username;
    String password;

    @Override
    public String toString() {
        return "[User { username=" + username  +", password=******}]";
    }
}

Cloud Logging

Most API frameworks come equipped with robust logging framework which
you can take advantage of when implementing logging in your service.
An example is the spring framework which have the cloud logging
feature to enable tracing log and fails between multiple services.

Configuring Spring Sleuth

For Spring Framework the cloud logging is provided through the
spring cloud package. Detailed walkthrough and version of spring
cloud that matches your spring or spring boot version can be seen
here

Glossary

Strongly Typed: A strongly typed programming language is one in which each type of data, such as integers, characters, hexadecimals and packed decimals, is predefined as part of the programming language.
Stack trace: This is the collection of stack records which stores the
procedure or method call information up until the stack trace is
invoked. Information stored in stack trace are the file name, line
number, column number e.t.c.
Weakly Typed:

References

Bash and Powershell remote scripts to install header-only C and C++ libraries from github repo.

Adewale Azeez — Thu, 11 Feb 2021 05:18:36 +0000

Managing dependencies for C and C++ projects
Mac, Unix, Linux Script
- Bash Script Options and Usage
- Bash Script Examples
Windows Script
- Powershell Script Options and Usage
- Powershell Script Examples

Managing dependencies for C and C++ projects

Managing dependency in C and C++ projects is usually tricky as there is no one standard registry where a library can be downloaded from. This might not be true for a Linux system where there is apt-get that installs libraries and programs from PPA but most of the libraries are outdated and it a lot of stress to add a library to PPA. Another approach is using Conan which requires the addition of the CMake build system into your project.

A better approach is to use a github as a package registry for C projects and install from each project repo since most libraries are distributed as source anyways, the closest project to achieving a smooth package registry with less hassle is vcpkg by Microsoft but this also requires the library to be added to the vcpkg port and have vcpkg installed on the system.

All the above ways of managing packages are all good but I love the way golang manages packages and is able to simply pull a package from the github repo and is immediately available for use in your go project, that the idea behind the two scripts I wrote and I went further such that no prerequisite is required to download a C, C++ library from github just the internet and the native shell for the operating systems, bash for Unix, Mac and Linux and Powershell for windows.

The purpose of the script is to be able to download, build (when needed), and install the library without any hassle at all using remote scripts. Currently, the script can only install headers only libraries from github but I had planned for it to be able to build non header-only library
if a CMakeLists.txt or Makefile is present in the repo root.

The source for the bash and Powershell scripts:

Mac, Unix, Linux Script

The bash script hosted at https://exoticlibraries.github.io/magic/install.sh can be used remotely or locally to install exotic libraries and any headers only library hosted on github into the /usr/local/include, /usr/include/ or any specified folder. The script accepts various argument to select what branch of the
repo to install, where to install the library, the temporary folder for download and extraction of the
repo archive e.t.c.

To use the script locally the bash script should be downloaded into a folder, it can also be invoked remotely using input character + curl.

To use locally replace bash <(curl -s https://exoticlibraries.github.io/magic/install.sh) in the examples
below with ./install.sh.

Bash Script Options and Usage

The script accepts various options, to view the help message use the -h or --help option.

bash <(curl -s https://exoticlibraries.github.io/magic/install.sh) -h

-h, --help: Displays the script help message and exit.
-all: Install all the exotic libraries. existing by default. This still downloads the archive though, so it advisable not to specify a library if it not to be installed and overidden.
--dontclean: Do not delete the libraries downloaded and extracted archive from the temporary folder.
--installfolder=: Specify where to install the library. The default location is /usr/local/include/ and if the script is executed with superuser (sudo/su) privilege, the default is /usr/include/. For example to change the installation folder to the current directory add the option --installfolder=./
--tmpfolder=: Change where the libraries archive are downloaded and extracted into, the default location is /tmp/

Bash Script Examples

The default github repo branch is main, to select which branch to install from add @branch-name after the library name e.g. libcester@dev, metaref@main, MrFrenik/gunslinger@master, https://github.com/nothings/stb@master* e.t.c.

To install all the exotic libraries:

bash <(curl -s https://exoticlibraries.github.io/magic/install.sh) -all

To install selected exotic libraries, e.g. libxtd, libcester, libmetaref

bash <(curl -s https://exoticlibraries.github.io/magic/install.sh) libxtd libcester libmetaref

To install non exotic library from a github repo, specify just the repo-owner/repo-name or the absolute github link to the repo. E.g. to install gunslinger and stb from master branch

bash <(curl -s https://exoticlibraries.github.io/magic/install.sh) MrFrenik/gunslinger@master https://github.com/nothings/stb@master

To install libcester and gunslinger into the currenct folder with a custom temporary directory

bash <(curl -s https://exoticlibraries.github.io/magic/install.sh) --installfolder=./ --tmpfolder=../temp/ libcester@main MrFrenik/gunslinger@master

Windows Script

The Powershell script hosted at https://exoticlibraries.github.io/magic/install.ps1 can be used remotely or locally to install exotic libraries and any header-only library hosted on github into the detected compilers include paths or any specified folder. The following compilers are auto-detected and the libraries are installed into their corresponding include paths:

Clang
GCC
MSVC

It is required that the gcc is present in PATH.

The script accepts various argument to select what branch of the repo to install, where to install the library, the temporary
folder for download and extraction of the repo archive e.t.c.

To use locally replace & $([scriptblock]::Create((New-Object Net.WebClient).DownloadString("https://exoticlibraries.github.io/magic/install.ps1"))) in the examples below with ./install.ps1.

Powershell Script Options and Usage

The script accepts various options, to view the help message use the -H or --Help option.

& $([scriptblock]::Create((New-Object Net.WebClient).DownloadString("https://exoticlibraries.github.io/magic/install.ps1"))) -H

-H, --Help: Displays the script help message and exit.
-All: Install all the exotic libraries.
--DontClean: Do not delete the libraries downloaded and extracted archive from the temporary folder.
--InstallFolder=: Specify where to install the library. The default location is the include path for detected compilers. For example to change the installation folder to the current directory add the option --installfolder=./
--TmpFolder=: Change where the libraries archive are downloaded and extracted into, the default location is C:\Users\<username>\AppData\Local\Temp

Powershell Script Examples

The default github repo branch is main, to select which branch to install from add @branch-name after the library name
e.g. libcester@dev, metaref@main, MrFrenik/gunslinger@master, https://github.com/nothings/stb@master e.t.c.

To install all the exotic libraries:

& $([scriptblock]::Create((New-Object Net.WebClient).DownloadString("https://exoticlibraries.github.io/magic/install.ps1"))) -All

To install selected exotic libraries, e.g. libxtd, libcester, libmetaref

& $([scriptblock]::Create((New-Object Net.WebClient).DownloadString("https://exoticlibraries.github.io/magic/install.ps1"))) libxtd libcester libmetaref

To install non exotic library from a github repo, specify just the repo-owner/repo-name or the absolute github link to the repo. E.g. to install gunslinger and stb from master branch

& $([scriptblock]::Create((New-Object Net.WebClient).DownloadString("https://exoticlibraries.github.io/magic/install.ps1"))) MrFrenik/gunslinger@master https://github.com/nothings/stb@master

To install libcester and gunslinger into the currenct folder with a custom temporary directory

& $([scriptblock]::Create((New-Object Net.WebClient).DownloadString("https://exoticlibraries.github.io/magic/install.ps1"))) --InstallFolder=./ --TmpFolder=../temp/ libcester@main MrFrenik/gunslinger@master

Shebang, Hebang, and Webang lines. Execute any type of file from the terminal

Adewale Azeez — Sun, 10 Jan 2021 16:15:14 +0000

This article will explain how to execute a file from the terminal without having to type out long commands and arguments every time. Most people working on Unix and Linux based OS will know what a shebang #! line is, this article will explain why there should be more type of bang lines and how to create them.

Table of content

How operating system identifies executable
Shebang
Hebang and Webang
Why more type of bang lines
Writing custom bang lines program
- Powershell
- Bash

How operating system identifies executable

There are various ways by which an operating system identifies a file as an executable. On platforms like Unix, Linux, and Mac the executable are mostly identified by its magic number which consists of byte string while on windows any file ending with .exe, .cmd or .com is identified as an executable, and for other types of file, it relies on the file association in the registry to identify what program to execute the file with.

An example of file association on windows is the .txt extension associated with any text editor you selected in the Open with dialog. Or the .html extension which is associated with Edge by default until you change the default executable to associate with the extension and the .py files associated with Python IDLE on windows.

The magic numbers for some binary like png, elf, pdf can be seen in this Wikipedia list. The magic number can also be used to detect a file MIME type properly instead of relying on the file extension.

Shebang

According to Wikipedia, "In computing, a shebang is the character sequence consisting of the characters number sign and exclamation mark (#!) at the beginning of a script. It is also called sha-bang, hashbang, pound-bang, or hash-pling".

From the explanation of magic number in the last section, shebang can simply be explained as a human-readable version of the magic number, it is used by the os procedures for executing a file to determine whether the file should be executed directly or as a binary.

When the shebang line is specified at the top of the file the os procedures will execute the command in the shebang line without executing the file itself. The procedure will then pass the absolute path to the file to the binary specified on the shebang line.

Example of a shebang line below:

#!/usr/bin/env python3

print("Hey through shebang")

Executing the file from the terminal with the command ./test.py and python3 test.py will output the same text.

Hebang and Webang

From the explanation of the shebang line above it has a huge limitation which is it cannot be used in source code or files that do not use # as a comment or has a special use-case for the character #, enters hebang, and webang lines.

I had the idea for hebang when I need to run unit tests for my C projects and every time I had to go to the terminal to run long commands. Using shebang is not possible in C as the # is used for macro preprocessing, since // is used for comments I decided to have another type of shebang but using // instead of #. After a while when I tried to run a C source that targets C90 and below with my custom hebang command I get the error C++ style comments are not allowed in ISO C90 and according to the C90 standard // is not a valid comment the only valid comment is /* */ so I had to add another type of bang line which I call webang.

These two new terms are my attempt to provide an alternative solution to shebang limitation, hebang, and webang lines are to bring the same functionality of shebang and more into other files that do not use # as a comment. The hebang uses //! as its prefix and webang use /*!.

The example below defines a hebang and webang line for a C file.

Hebang line

//!gcc {0} -I. -I../../include/ -o out; ./out

#include <stdio.h>

int main(int agvc, char **argv)
{
    printf("Hello through hebang and webang!!!\n");
}

Webang line

/*!gcc {0} -I. -I../../include/ -o out; ./out */

#include <stdio.h>

int main(int agvc, char **argv)
{
    printf("Hello through hebang and webang!!!\n");
}

Why more type of bang lines

Normally shebang will work fine for files using # as a comment and executed with an interpreter, but for a compiled language like C, Java, Rust that needs to be compiled, sometimes linked before the generated binary can be executed shebang might not work as it only appends the file path at the end of the bang string.

With a custom shebang, hebang, and webang script multiple commands can be written in the bang line for any type of file, various steps like compilation, linking, and execution can be executed with just one bang line.

Writing custom bang lines scripts

The basics for writing your own bang line command are:

Read the first line of the file
Check the first two chars in the first line,
- if it equals #! that a shebang line
- if it equals // and the next char is ! that a hebang line
- if it equals /* and the next char is ! that a webang line
If the first two chars do not match above send the file to the os for execution using the appropriate logic of any language you implementing in.
If it matches any of the bang line strip the bang prefix and in case of webang remove the comment suffix */
Then execute the remaining string with the absolute file path appended to the string from the bang line.
But in my case, I did not append the file path to the bang command instead I allow the use of string format so I can accept the file at any part of the bang string. e.g. gcc {0} -o out; ./out my command replaces {0} with the path to the file been executed and appends the remaining arguments from the terminal to the expanded string, therefore the command above will be expanded into gcc test_bang.c -o out; ./out.

Give your command/script any name and use it to execute your files. By following the command above you can also use the command/script to execute any type of file and not the ones with bang lines only.

For the example above, My command name is $ to make sure it not intrusive and not noticeable in the terminal.

$ test_bang.c --help

The bang line in the file test_bang.c will expand to:

gcc test_bang.c -I. -I../../include/ -o out; ./out --help

Another example of executing a Javascript source (test_bang.js) using a hebang line

//*!node {0}

console.log("Hello " + process.argv[2]);

> $ test_bang.js dev.to
> Hello dev.to

My first attempt at creating the bang line script is using Batch file as part of the Cronux collection, I later rewrite the commands in the collection using Powershell for sanity. I had to write the bash version of the script as I don't have so much time to port Cronux to Unix, Linux, and Mac.

You can save any of the sources below as $ and start using hebang and shebang in your files, e.g. if you save as bang.ps1 or bang.sh

bang python.py
bang test_bang.c

Powershell

Bash

I hope this article helps maximize your productivity.

Hosting a maven repository on Github: site-maven-plugin

Adewale Azeez — Mon, 27 Jan 2020 16:36:04 +0000

This article provides a step by step guide on publishing a maven project from GitHub using site-maven-plugin.

Prerequisites

Project repository on Github
Java and Maven
Familiarity with Maven configuration

Create a repository for your maven project on Github if you don't have one already and push your maven project. This method will use the site-maven-plugin to push the artifacts to Github.

This approach does not cause conflict with gh-pages and other branch as the artifacts is pushed to the mvn-repo branch of your repository, every time the mvn deploy command is executed the new build artifact will be published.

First, the maven artifact has to be deployed to a temporary location in the build/target directory. Add the following repository to the project pom.xml:

<distributionManagement>
    <repository>
        <id>internal.repo</id>
        <name>Temporary Staging Repository</name>
        <url>file://${project.build.directory}/mvn-repo</url>
    </repository>
</distributionManagement>

Add the maven-deploy-plugin configuration to your pom.xml:

<plugin>
    <artifactId>maven-deploy-plugin</artifactId>
    <version>2.8.2</version>
    <configuration>
        <altDeploymentRepository>internal.repo::default::file://${project.build.directory}/mvn-repo</altDeploymentRepository>
    </configuration>
</plugin>

After adding the above repository and plugin to your pom.xml execute mvn deploy, the maven repository will be deployed to the directory target/mvn-repo/. The next step is to deploy to github mvn-repo branch.

Configure github authentication information in ~/.m2/settings.xml to enable site-maven-plugin push to github. if the file settings.xml does not exist it should be created.

There are several ways to authenticate github, add one of the following configuration to your ~/.m2/settings.xml.

using your github username and password:

<settings>
  <servers>
    <server>
      <id>github</id>
      <username>GitHubLogin</username>
      <password>GitHubPassw0rd</password>
    </server>
  </servers>
</settings>

using OAUTH2TOKEN:

<settings>
  <servers>
    <server>
      <id>github</id>
      <password>OAUTH2TOKEN</password>
    </server>
  </servers>
</settings>

Then add the following to your pom.xml:

<properties>
  <github.global.server>github</github.global.server>
</properties>

The last step is to configure site-maven-plugin to push your local staging repo target/mvn-repo/ to your remote mvn-repo branch.

<plugin>
    <groupId>com.github.github</groupId>
    <artifactId>site-maven-plugin</artifactId>
    <version>0.11</version>
    <configuration>
        <message>Maven artifacts for ${project.version}</message> <!-- git commit message -->
        <noJekyll>true</noJekyll>  <!-- disable webpage processing -->
        <outputDirectory>${project.build.directory}/mvn-repo</outputDirectory> <!-- matches distribution management repository url above -->
        <branch>refs/heads/mvn-repo</branch> <!-- remote branch name -->
        <includes><include>**/*</include></includes>
        <repositoryName>YOUR-REPOSITORY-NAME</repositoryName> <!-- github repo name -->
        <repositoryOwner>THE-REPOSITORY-OWNER</repositoryOwner> <!-- organization or user name  -->
    </configuration>
    <executions> <!-- run site-maven-plugin's 'site' target as part of the build's normal 'deploy' phase -->
      <execution>
        <goals>
          <goal>site</goal>
        </goals>
        <phase>deploy</phase>
      </execution>
    </executions>
</plugin>

repositoryOwner value is is the parent name of the repo, if the repo is owned by a github organization the value will be the organization name and if the repo is owned by a user the value will be the username e.g. for the repo keyvaluedb/key-value-db-java, repositoryName value will be key-value-db-java and repositoryOwner value will be keyvaluedb.

Execute mvn deploy to upload your artifact to github. The mvn-repo branch will be created if it does not exist.

Visit github.com in your browser, select the mvn-repo branch, and verify that all your binaries are uploaded.

Hooray. Your maven project is now available to be used in other projects.

Every time you run mvn clean deploy command on your project, the latest artifacts will be uploaded to github.

Other maven project using your project can be configured to pull your artifact from github. Add the following snippet to the pom.xml files that depend on your project.

<dependency>
    <groupId>YOUR.PROJECT.GROUPID</groupId>
    <artifactId>ARTIFACT-ID</artifactId>
    <version>VERSION</version>
</dependency>

<repository>
    <id>ARTIFACT-ID</id>
    <url>https://raw.github.com/REPOSITORYOWNER/REPOSITORY-NAME/mvn-repo/</url>
</repository>

After adding the properties to the pom.xml the project will automatically download your maven repository jars from github.

Managing Windows PFX certificates through PowerShell

Adewale Azeez — Fri, 17 Jan 2020 15:16:38 +0000

The certificate is a container for the public key. It includes the public key, the server name, some extra information about the server, and a signature computed by a certification authority (CA).

Table of content

Prerequisites
What is PFX Certificate
Logical stores
PKI Module
- Export-PfxCertificate
- Get-Certificate
- Get-PfxData
- Import-PfxCertificate
- New-SelfSignedCertificate
Finding and Selecting a certificate
Creating a certificate
Installing a pfx certificate
Add a certificate as trusted
Exporting and Importing certificate
- Exporting pfx certificate
- Importing pfx certificate
Export a Certificate from pfx
Deleting a certificate
- Deleting with thumbprint
- Find and Delete a certificate using property match
Windows Certificate Manager
References

Prerequisites

Windows Vista, Windows Server 2008, or newer operating system. The examples shown use Windows 10 Pro version 1909
PowerShell 5.X and above
Familiarity with PowerShell

What is a PFX Certificate

A .pfx file which should not be confused with .cert is a PKCS#12 archive; this is a bag that can contain a lot of objects with optional password protection. It usually contains a certificate (possibly with its assorted set of CA certificates) and the corresponding private key. While PFX can contain more than one certificates a .cert file contains a single certificate alone with no password and no private key. Within Windows, all certificates exist in logical storage locations referred to as certificate stores.

The most common usage of a PFX certificate is code signing and creating a Software Publisher Certificate.

Logical stores

Logical stores are virtual locations that map to certificate physical paths. Powershell uses the Cert PSDrive to map certificates to the physical stores.

The Certificates Microsoft Management Console (MMC) logical store labeling is different from the Cert PSDrive store labeling. The table below shows the comparison between both:

CERT:	CERTIFICATES MMC
My	Personal
Remote Desktop	Remote Desktop
Root	Trusted Root Certification Authorities
CA	Intermediate Certification Authorities
AuthRoot	Third-Party Root Certification Authorities
TrustedPublisher	Trusted Publishers
Trust	Enterprise Trust
UserDS	Active Directory User Object

PKI Module

MakeCert.exe utility, which is part of the Microsoft .NET Framework SDK and Microsoft Windows SDK is used to create a self-signed certificate. On a system with no Windows SDK installed, the cmdlets in Powershell PKI module are used to manage the certificates.

To list all the commands available in the PKI module, run the following command:



Get-Command -Module PKI

The following commands are available from the PKI module.



CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Cmdlet          Add-CertificateEnrollmentPolicyServer              1.0.0.0    PKI
Cmdlet          Export-Certificate                                 1.0.0.0    PKI
Cmdlet          Export-PfxCertificate                              1.0.0.0    PKI
Cmdlet          Get-Certificate                                    1.0.0.0    PKI
Cmdlet          Get-CertificateAutoEnrollmentPolicy                1.0.0.0    PKI
Cmdlet          Get-CertificateEnrollmentPolicyServer              1.0.0.0    PKI
Cmdlet          Get-CertificateNotificationTask                    1.0.0.0    PKI
Cmdlet          Get-PfxData                                        1.0.0.0    PKI
Cmdlet          Import-Certificate                                 1.0.0.0    PKI
Cmdlet          Import-PfxCertificate                              1.0.0.0    PKI
Cmdlet          New-CertificateNotificationTask                    1.0.0.0    PKI
Cmdlet          New-SelfSignedCertificate                          1.0.0.0    PKI
Cmdlet          Remove-CertificateEnrollmentPolicyServer           1.0.0.0    PKI
Cmdlet          Remove-CertificateNotificationTask                 1.0.0.0    PKI
Cmdlet          Set-CertificateAutoEnrollmentPolicy                1.0.0.0    PKI
Cmdlet          Switch-Certificate                                 1.0.0.0    PKI
Cmdlet          Test-Certificate                                   1.0.0.0    PKI

The cmdlets used in this article are explained below:

Export-PfxCertificate

The Export-PfxCertificate cmdlet exports a certificate or a PFXData object to a Personal Information Exchange (PFX) file. By default, extended properties and the entire chain are exported.

Get-Certificate

The Get-Certificate cmdlet can be used to submit a certificate request and install the resulting certificate, install a certificate from a pending certificate request, and enroll for the directory service protocol LDAP.

Get-PfxData

The Get-PfxData cmdlet extracts the contents of a Personal Information Exchange (PFX) file into a structure that contains the end entity certificate, any intermediate and root certificates.

Import-PfxCertificate

The Import-PfxCertificate cmdlet imports certificates and private keys from a PFX file to the destination store.

New-SelfSignedCertificate

The New-SelfSignedCertificate cmdlet creates a self-signed certificate for testing purposes. Using the CloneCert parameter, a test certificate can be created based on an existing certificate with all settings copied from the original certificate except for the public key.

Finding and Selecting a certificate

In Powershell, the Cert: PSDrive is used to list the certificates in a particular store. To list the two locations under the Cert: PSDrive, run the following command:



Get-ChildItem -Path Cert:

The certificate locations for the CurrentUser and LocalMachine will be shown. To list the certificates under the LocalMachine Trusted Root Certification Authorities, run the following command:



Get-ChildItem -Path Cert:\LocalMachine\Root\

The returned object will be the certificates that can be modified, delete or exported.

The Get-ChildItem cmdlet can be combined with Where-Object to find specific certificates. The command below will find certificate supposedly issued by Microsoft by checking the Subject property of the certificates:



Get-ChildItem -Path Cert:\LocalMachine\Root\ | Where-Object {$_.Subject.Contains("Microsoft")}

The certificates listed will have part of it Subject value contains Microsoft.

Creating a certificate

To create a certificate the values of –DnsName (name of a DNS server) or the -Subject and -CertStoreLocation (the certificate store in which the generated certificate will be placed) will have to be specified.

To create a certificate for the DNS test.com and install it in the list of Personal certificate on a system, run the following command:



New-SelfSignedCertificate -DnsName test.com -CertStoreLocation cert:\LocalMachine\My

The command creates a new certificate and installs it into the personal store location of the computer. On opening certlm.msc the certificate will appear in the Personal section. The output will show the details of the newly created certificate including its thumbprint:



PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\My

Thumbprint                                Subject
----------                                -------
644F29EA276A639A28F82137F8132D6F49C1664A  CN=test.com

A self-signed certificate is generated with the following settings by default:

Cryptographic algorithm: RSA;
Key length: 2048 bit;
Acceptable key usage: Client Authentication and Server Authentication;
The certificate can be used for Digital Signature, Key Encipherment;
Validity period: 1 year.

The validity of the generated certificate is limited to one year. To extend the certificate, a proper Date object with extended validity period should be specified with the -notafter switch. To issue a certificate for 5 years, run the command below:



$expiry_year = (Get-Date).AddYears(5)
New-SelfSignedCertificate -DnsName test.com -notafter $expiry_year -CertStoreLocation Cert:\LocalMachine\My

The command below generates a certificate with value Test using the Subject property only and sets its expiry year to 3 years after the creation date:



$expiry_year = (Get-Date).AddYears(3)
New-SelfSignedCertificate -Subject Test -notafter $expiry_year -CertStoreLocation Cert:\LocalMachine\My

New certificates can only be created in the personal store locations, Cert:\LocalMachine\My or Cert:\CurrentUser\My.

Execute the help command below to view all other parameters accepted by the New-SelfSignedCertificate cmdlet:



help New-SelfSignedCertificate -Full

To export the certificate to pfx, read this section

Installing a pfx certificate

The certificate can be installed in a different store location, each store location has its specific purpose but in this article, we will install into four locations Cert:\CurrentUser\My, Cert:\LocalMachine\My, Cert:\CurrentUser\Root and Cert:\LocalMachine\Root. My location is the personal store for untrusted application and Root is the store for trusted certificates.

The command used to install a common certificate is different from the command to install a PFX certificate. The Powershell Cmdlet Import-PfxCertificate is used to install a pfx certificate.

To install a PFX certificate to the current user's personal store, use the command below:



Import-PfxCertificate -FilePath ./TestPFXCert.pfx -CertStoreLocation Cert:\CurrentUser\My -Password testpassword

To install into the system personal location change the store location in the command above from Cert:\CurrentUser\My to Cert:\LocalMachine\My

To install a PFX certificate the current user's trusted store, use the command below:



Import-PfxCertificate -FilePath ./TestPFXCert.pfx -CertStoreLocation Cert:\CurrentUser\Root -Password testpassword

Installing into the current user Root location will pop up a dialog to confirm the certificate installation but installing into the system Root location will not pop up any dialog.

To install into the system trusted location change the store location in the command above from Cert:\CurrentUser\Root to Cert:\LocalMachine\Root

Add a certificate as trusted

An untrusted certificate is pretty much as dangerous and suspicious than no certificate at all, therefore, it is important to add our certificate to the trusted path.

Certificates can only be generated in the Cert:\LocalMachine\My store location. Since any certificate in this location is marked as untrusted by default, to make it secure it has to be moved from Cert:\LocalMachine\My to Cert:\LocalMachine\Root.



Move-Item -Path $cert.PSPath -Destination "Cert:\LocalMachine\Root"

Now the certificate will be marked as trusted. When used to sign a script or application the script will be marked secure for execution.

Check for duplicate certificate

To avoid re-adding an already existing certificate into the certificate store location, the already existing certificate should be checked and removed before adding a new certificate with the same subject. Powershell automatically creates a virtual drive path for the cert: label and behaves like a normal directory. The snippet below checks the cert:\LocalMachine\My path of the certificate store to find a certificate using its Subject value.



$cert_name = "Test Cert"
ForEach ($cert in (ls cert:\LocalMachine\My)) {
    If ($cert.Subject -eq "CN=$cert_name") {
        //the certificate can be deleted or edited in here
    }
}

Exporting and Importing certificate

The commands to import and export a pfx certificate are different from commands to export and import a regular certificate. The Import-PfxCertificate cmdlet is used for importing a passworded pfx certificate and Export-PfxCertificate cmdlet for exporting a certificate from its store location to a new file location.

Exporting pfx certificate

To export a pfx certificate a password is needed for encrypting the private key.

In the example below, we use the Subject property to find the certificate to be exported by selecting the certificate whose Subject value equals Test,



$certificate = Get-ChildItem -Path Cert:\LocalMachine\My\ | Where-Object {$_.Subject -match "Test"}

After selecting the certificate it can be exported from its store location to a folder with the command below:



$password= "@OurPassword1" | ConvertTo-SecureString -AsPlainText -Force
Export-PfxCertificate -Cert $certificate -FilePath $env:USERPROFILE\Documents\TestCert.pfx -Password $password

Importing pfx certificate

Downloaded or exported pfx certificates can be installed using the Import-PfxCertificate cmdlet. The password for the certificate and location are required for importing.

The command below imports the pfx certificate we exported earlier:



$password= "@OurPassword1" | ConvertTo-SecureString -AsPlainText -Force
Import-PfxCertificate -Exportable -Password $password -CertStoreLocation Cert:\LocalMachine\My -FilePath $env:USERPROFILE\Documents\TestCert.pfx

The -Exportable switch marks the private key as exportable.

Export a Certificate from pfx

To export certificate from a pfx file, the combined cmdlet Get-PfxCertificate and Export-Certificate is used. Get-PfxCertificate is used to locate the pfx certificate and Export-Certificate is used to export the specified certificate to a FilePath.

This process is not seamless as it requires the user to input the password for the pfx certificate.

The following command exports a DER encoded (binary) certificate from a pfx file.



Get-PfxCertificate -FilePath $env:USERPROFILE\Documents\TestCert.pfx | 
Export-Certificate -FilePath $env:USERPROFILE\Documents\TestCert.cer -Type CERT

Note the -Type switch which is used to indicates the type of certificate to be exported, in this example a DER encoded (binary) CERT certificate. There are other types that can be exported. They are listed below.



-Type <CertType>
    Specifies the type of output file for the certificate export as follows. 
     -- SST: A Microsoft serialized certificate store (.sst) file format which can contain one or more certificates. This is the default value for multiple certificates. 
     -- CERT: A .cer file format which contains a single DER-encoded certificate. This is the default value for one certificate. 
     -- P7B: A PKCS#7 file format which can contain one or more certificates.

Deleting a certificate

To remove a certificate, the Remove-Item command in Powershell can be used. Before a certificate can be deleted its thumbprint id must be known or the certificate object itself identified. The certificate path can be iterated through, using the snippets above to find the object or thumbprint.

Deleting with thumbprint

The snippet below uses the certificate thumbprint for deletion...



Get-ChildItem Cert:\LocalMachine\My\D20159B7772E33A6A33E436C938C6FE764367396 | Remove-Item

Get-ChildItem is a Powershell command to get file/folder information, one of its aliases is ls. It simply uses the thumbprint to get the certificate object and pipe it as input to the Remove-Item command.

Find and Delete a certificate using property match

A certificate can be searched in the store location using the Where-Object command that accepts a conditional statement that matches the certificate we are looking for.



Get-ChildItem Cert:\LocalMachine\My |
Where-Object { $_.Subject -match 'Test' } |
Remove-Item

Notice we use the Subject property of a certificate, the other properties are Thumbprint and SerialNumber.

Windows Certificate Manager

The Windows Certificate Manager (certmgr.msc) is a Windows essential GUI application to manage certificates. Search for certmgr in the Start menu to open the Windows Certificates MMC. The landing view provides an overview of all the logical stores.

Certificates can be installed, delete, import and export from the Windows Certificate Manager.

References

PKIClient
Digital Certificates
System Store Locations
Convert .pfx to .cer
How to Sign PowerShell Script with a Code Signing Certificate
Managing Certs with Windows Certificate Manager and PowerShell

Enabling and disabling USB Disk Drive using C#

Adewale Azeez — Wed, 04 Dec 2019 16:18:50 +0000

Working with device drivers from a managed environment like .NET can be messy and sometimes not possible without making a call to unmanaged native code APIs. In this post, I will show how to enable and disable USB disk drive in C# using ManagementObjectSearcher and Native setupapi.

Before a device can be enabled or disable we need to know the device GUID and its instance path.

Getting device GUID and instance path

GUID(Globally Unique Identifier) is also known as UUID(Universally Unique Identifier) is a 128-bit integer number used to identify resources. Every device has hard-coded GUID but to get the GUID from C# we can query the device group using ManagementObjectSearcher. To use the ManagementObjectSearcher class you will need to add the System.Management reference to the project.

In our case we need to get the detail of disk drive hence we will query one of the Win32 class DiskDrive, CIM_LogicalDevice or CIM_LogicalElement. Querying DiskDrive will only give information on currently enabled Disk Drive but we need to get all drives on the system including those with error, disabled, mounted e.t.c. hence we use the CIM_LogicalDevice class which is the supertype of the DiskDrive class.

Add the following namespaces in your code.

using System;
using System.Collections.Generic;
using System.Management;

Create the static method to get all known logical drive from the system

static List<ManagementBaseObject> GetLogicalDevices()
{
    List<ManagementBaseObject> devices = new List<ManagementBaseObject>();
    ManagementObjectCollection collection;
    using (var searcher = new ManagementObjectSearcher("root\\CIMV2", 
                          @"Select * From CIM_LogicalDevice"))
                          collection = searcher.Get();
    foreach (var device in collection)
    {
        devices.Add(device);
    }
    collection.Dispose();
    return devices;
}

Note the class we choose to query in the line @"Select * From CIM_LogicalDevice")) you can change from CIM_LogicalDevice to other device class you wish to get in case you want to fetch detail of other kind of devices e.g to get all known keyboard you can change it to CIM_Keyboard, @"Select * From CIM_Keyboard")). You can see all the list of device classes at https://docs.microsoft.com/en-gb/windows/win32/cimwin32prov/cim-wmi-provider.

Iterating over our devices to get GUIDs and instance paths

To get the properties of the returned devices we need to use the GetPropertyValue method of ManagementBaseObject to extract the value of a property of the device. The device classes have different properties hence the doc should be consulted before trying to get property value from a ManagementBaseObject. In our case, we need the properties of CIM_LogicalDevice specifically DeviceID which holds the instance path and ClassGuid which holds the GUID.

var usbDevices = GetLogicalDevices();
foreach (var usbDevice in usbDevices)
{
    Console.WriteLine("Name: {0}, Status: {1}, Instance Path: {2}, GUID: {3}",
                       usbDevice.GetPropertyValue("Caption"),
                       usbDevice.GetPropertyValue("Status"),
                       usbDevice.GetPropertyValue("DeviceID"),
                       usbDevice.GetPropertyValue("ClassGuid"));
}

Disabling and Enabling the drive using it GUID and Instance path

After getting the GUID and Instance path of the device to be enabled or disabled we can make a call to the Win32 unmanaged SetupAPI to interact with the device drivers.

The following native setup API methods are required to enable and disable a device:

public static extern bool SetupDiEnumDeviceInfo(SafeDeviceInfoSetHandle deviceInfoSet, int memberIndex, ref DeviceInfoData deviceInfoData);
public static extern bool SetupDiCallClassInstaller(DiFunction installFunction, SafeDeviceInfoSetHandle deviceInfoSet, [In()] ref DeviceInfoData deviceInfoData);
public static extern SafeDeviceInfoSetHandle SetupDiGetClassDevs([In()] ref Guid classGuid, [MarshalAs(UnmanagedType.LPWStr)] string enumerator, IntPtr hwndParent, SetupDiGetClassDevsFlags flags);
public static extern bool SetupDiGetDeviceInstanceId(IntPtr DeviceInfoSet, ref DeviceInfoData did, [MarshalAs(UnmanagedType.LPTStr)] StringBuilder DeviceInstanceId, int DeviceInstanceIdSize, out int RequiredSize);
public static extern bool SetupDiDestroyDeviceInfoList(IntPtr deviceInfoSet);
public static extern bool SetupDiSetClassInstallParams(SafeDeviceInfoSetHandle deviceInfoSet, [In()] ref DeviceInfoData deviceInfoData, [In()] ref PropertyChangeParameters classInstallParams, int classInstallParamsSize);

To enable/disable a device the GUID is used to fetch SafeDeviceInfoSetHandle of the device from the SetupDiGetClassDevs native method.

Since the GUID is for a group of devices i.e more than one device will have the GUID so far the devices fall under the same device class. The native method GetDeviceInfoData is used to get all the devices under the GUID.

The instance id will determine the actual device we want to change its state, the GetIndexOfInstance is invoked with diSetHandle, the array of GetDeviceInfoData and the instance ID to get the index of the device in the list of all the devices returned by the GetDeviceInfoData method.

public static void SetDeviceEnabled(Guid classGuid, string instanceId, bool enable)
{
    SafeDeviceInfoSetHandle diSetHandle = null;
    try
    {
        diSetHandle = NativeMethods.SetupDiGetClassDevs(ref classGuid, null, 
                      IntPtr.Zero, SetupDiGetClassDevsFlags.Present);
        DeviceInfoData[] diData = GetDeviceInfoData(diSetHandle);
        int index = GetIndexOfInstance(diSetHandle, diData, instanceId);
        EnableDevice(diSetHandle, diData[index], enable);
    }
    finally
    {
        if (diSetHandle != null)
        {
            if (diSetHandle.IsClosed == false)
            {
               diSetHandle.Close();
           }
           diSetHandle.Dispose();
        }
    }
}

The SafeDeviceInfoSetHandle, the array of DeviceInfoData and the condition(true/false) is sent into our EnableDevice method to change the device state. The EnableDevice method is implemented below. We make a call on the native method in setupdi SetupDiSetClassInstallParams to change the state of the device

private static void EnableDevice(SafeDeviceInfoSetHandle handle, 
                                 DeviceInfoData diData, bool enable)
{
    PropertyChangeParameters @params = new PropertyChangeParameters();
    @params.Size = 8;
    @params.DiFunction = DiFunction.PropertyChange;
    @params.Scope = Scopes.Global;
    if (enable)
    {
        @params.StateChange = StateChangeAction.Enable;
    }
    else
    {
        @params.StateChange = StateChangeAction.Disable;
    }
    bool result = NativeMethods.SetupDiSetClassInstallParams(handle, ref diData, 
                                           ref @params, Marshal.SizeOf(@params));
    if (result == false) throw new Win32Exception();
    result = NativeMethods.SetupDiCallClassInstaller(DiFunction.PropertyChange,
                                                     handle, ref diData);
    if (result == false)
    {
        int err = Marshal.GetLastWin32Error();
        if (err == (int)SetupApiError.NotDisableable)
            throw new ArgumentException("Device can't be disabled (programmatically).");
        else if (err >= (int)SetupApiError.NoAssociatedClass && 
                err <= (int)SetupApiError.OnlyValidateViaAuthenticode)
            throw new Win32Exception("SetupAPI error: " + ((SetupApiError)err).ToString());
        else
            throw new Win32Exception();
        }
    }
}

The SETUPAPI native method full source code is at https://github.com/thedarkprojects/devjammer/blob/master/NativeSetupDiLib.cs the file can be placed in your project without the need for extra dependency.

The EnableDevice method can be called with the GUID and Instance id gotten from ManagementObjectSearcher query above and our boolean to enable or disable the drive. E.g. to disable an already active disk drive connect with USB (external hard disk, card reader, MTP Device), we can check the list of Logical Drive and disable the drive that contains USB and is enabled

var usbDevices = GetLogicalDevices();
foreach (var usbDevice in usbDevices)
{
    if (usbDevice.GetPropertyValue("DeviceID").ToString().Contains("USB") && 
        usbDevice.GetPropertyValue("Status").ToString().Contains("OK") ) 
    {
        Guid mouseGuid = new Guid(usbDevice.GetPropertyValue("ClassGuid").ToString());
        string instancePath = usbDevice.GetPropertyValue("DeviceID").ToString();
        DeviceHelper.SetDeviceEnabled(mouseGuid, instancePath, true)
    }
}

If the state of the device does not change en exception is thrown. If no exception occurs in the method the state of the device has changed successfully.

When building the project ensure CPU target corresponds to the CPU architecture of your system as the SETUPAPI is conscious of CPU architecture. The functionalities will not work on an x86 system if the build target is x64 and it won't work on x64 if the target build is x86. If you are using a 64bit system ensure your CPU target build is x64 and if your system is 32bit ensure your build target is x86. Also, you have to run the app in Administrator Mode for it to work.

Applications

The ManagementObjectSearcher query can be used to get pretty much any or all types of devices on the Windows system. With a careful combination of the ManagementObjectSearcher queries and the NativeSetupDiLib you can implement a custom Device control application that can block access to system e.g blocking the USB port by listening for device changes and if not allowed in your application you immediately disable the device connected in the USB port. It can also be used to bypass device control by anti-virus programs by continual enabling of a device if the dev control program disable it (be careful).

If you are choosing to perform the device enabling and disabling from command line by writing a system script (batch/powershell) you can check out the devcon tool provided by Microsoft which also uses the SETUP APIs. For the devcon program to work ensure you download the CPU architecture version for your system e.g. devcon-x64 if your PC is 64bit.