DEV Community: grant horwood

curl: getting performance data with curl

grant horwood — Mon, 15 Sep 2025 15:41:30 +0000

the api endpoint you've written is slow. everyone says so. but how slow? there are lots of packages we could install to inspect an url's performance, but why bother with all that when curl can get us the performance metrics we need?

in this post, we'll be going over using curl to get data like execution time, download speed and header values.

a user waits patiently for an url that was not profiled for performance

getting total time using `--write-out`

most of us, when we use curl to inspect an url, just throw the -v switch on it for 'verbose' and then sift through the headers and body using grep, looking for the data we want. this works in a banging-two-rocks-together, paleolithic kind of way, but we can craft a much more elegant and powerful solution by using the --write-out parameter.

as the name implies, write-out is about filtering and formatting curl's output. the write-out argument accepts a string that defines the output format using pre-defined variables to represent various response data.

let's look at an example of how we would get the total execution time of a request:

curl -s \
-X GET \
-o /dev/null \
--write-out "Total time: %{time_total} seconds" \
https://example.ca/path/to/endpoint

here we see that we provided write-out with a format string that included the variable %{time_total}. this variable gets interpolated in the output with the time, in seconds, that the entire process took; from the moment the request was initiated to the instant the last byte of the response was downloaded.

the output looks similar to this:

Total time: 0.897665 seconds

that's useful information!

note that this command also suppresses all other output to keep thing nice and clean. we use the -s switch for 'silent' operation, and throw away the body of the response with -o /dev/null. it just makes everything nicer and easier to read.

more timing data

getting the total round-trip time of our call is good and useful, but we might want a more detailed accounting.

fortunately, curl provides with seven different timing measurements. of these, there are four that are particularly useful:

time_namelookup: how long, in seconds, it took to do the name lookup. ie. dns.
time_pretransfer: the time, in seconds, the entire request portion of the call took; all that dns and connection and ssl stuff.
time_starttransfer: the time, in seconds, from the start of the call to right before the first byte of the response was transferred.
time_total: how long, in seconds, the whole call took

let's look at a sample curl call that gets all these data points:

curl -s \
-X GET \
-o /dev/null \
--write-out "time_namelookup %{time_namelookup}\ntime_pretransfer %{time_pretransfer}\ntime_starttransfer %{time_starttransfer}\ntime_total %{time_total}" \
https://example.ca/ | column -t

you'll notice that we added a pipe out to column -t to format the output into columns. this is just a little bit of display sugar to make things easier on the eyes. the output looks like this:

time_namelookup     0.001698
time_pretransfer    0.171207
time_starttransfer  0.298703
time_total          0.587096

looking at this output and knowing what these data points represent allow us to do a little bit of subtraction math to learn some important things.

how long our server took to calculate the result: since we know that time_pretransfer is the time from the start of the call to the end of the request, and that time_starttransfer is the time from the start to very beginning of the response, we can determine that the time the server took to calculate the response is (approximately) time_starttransfer - time_pretransfer.

for instance, in the above example, our time_pretransfer was 0.171207 and our time_starttransfer was 0.298703 meaning that our time-on-server was about 0.127496 seconds.

this value is mildly accurate and will obvious change from one request to the next. however, it is good data to have for making comparisons.

how long our download took: we can also use subtraction to determine how long it took for the response data to get downloaded from the server to our test machine by subtracting time_starttransfer, the time to the first response byte, from time_total. however, there are better ways to inspect download speed.

speeds and sizes

if we want to know how much data we're downloading and how fast that download is, we can get that with the following write-out variables:

size_download: the size of the total download, in bytes.
size_header: the size of just the headers downloaded, in bytes.
speed_download: the speed of the download in bytes/second.

curl -s \
-X GET \
-o /dev/null \
--write-out "size_download %{size_download}\nsize_header %{size_header}\nspeed_download %{speed_download}" \
https://example.ca/ | column -t

the output of this example call looks something like:

size_download   922306
size_header     704
speed_download  1186138

response codes and headers

getting timing data is great for telling us what happened, but if we want to get some clues as to why, we're going to want to inspect things like response codes and headers.

getting the response code

if you've ever noticed that an endpoint suddenly got a lot faster because it started returning a 404, you'll understand that knowing the response code of a request is important.

we can get the response code with the %{response_code} variable:

curl -s \
-X GET \
-o /dev/null \
--write-out "response_code %{response_code}" \
https://example.ca/ | column -t

it outputs what we would expect:

response_code  200

getting headers

we can extract individual headers from the response by using the write-out variable

%header{<header name>}

the <header name> here is the name of the header in all lowercase and without the trailing colon.

if we wanted to test what our url's cloudflare cache status was, for instance, we could do:

curl -s \
-X GET \
-o /dev/null \
--write-out "cf-cache-status %header{cf-cache-status}" \
https://example.ca/

and get as output something like

cf-cache-status MISS

getting all the headers as json

we can get all the response headers in json format with %{header_json}. note that this is singular: header, not headers.

curl -s \
-X GET \
-o /dev/null \
--write-out "header_json %{header_json}" \
https://example.ca/

the output looks like:

header_json {
    "server":["nginx/1.18.0 (Ubuntu)"],
    "date":["Thu, 11 Sep 2025 19:06:12 GMT"],
    "content-type":["text/html"],
    "content-length":["178"],
    "connection":["keep-alive"],
    "location":["https://example.ca/"]
}

we can then inspect the data in the output using a tool like jq.

using a format configuration file

of course, typing out a long write-out format every time we want to curl an endpoint is annoying. we can save time and keystrokes by storing output formats in configuration files.

first, let's create a format file:

time_namelookup %{time_namelookup}\n
time_pretransfer %{time_pretransfer}\n
time_starttransfer %{time_starttransfer}\n
time_total %{time_total}\n
size_download %{size_download}\n
size_header %{size_header}\n
speed_download %{speed_download}\n

this file holds an output format and is the same as if we had used it directly in the command.

once we have our format file, we can supply it to the write-out argument using the @ directive, which tells curl to read input from a file

curl -s \
-X GET \
-o /dev/null \
--write-out "@/path/to/file" \
https://example.ca/

the output looks like what we would expect:

time_namelookup 0.037403
time_pretransfer 0.206697
time_starttransfer 0.241542
time_total 0.556894
size_download 922308
size_header 704
speed_download 1656164

format configuration files are good things because they allow us to easily standardize formats and write our commands more accurately and qucklly.

conclusion

curl may not be the most sophisticated profiling tool out there, but it is still incredibly powerful and almost universally available. having a better understanding of its capabilities will always pay off.

php: a curl cheatsheet

grant horwood — Tue, 09 Sep 2025 16:45:40 +0000

php's implementation of curl is very powerful. unfortunately, it's also frustratingly complex and the official docs can be terse and dense. every php developer who uses curl frequently accumulates, over time, a directory of tried-and-true snippets for reference; a 'cheatsheet', basically. this is mine.

a php dev copies and pastes useful php curl examples from a blog post

flyover

this article is going to cover (nearly) all of the basic use cases of php curl and provide examples that can be copy-pasted and modified. the topics are:

an overview of a curl request
a basic `GET` request
reusing the pointer for multiple requests
building and sending a query string
getting the HTTP code with `curl_get_info`
connecting to a different port
error handling

treating http errors as curl errors

sending headers
getting headers

functions to extract response headers

basic auth
sending cookies
getting cookies
cookie jars

getting cookies into a file
sending cookies from a file
sending and getting cookies using a file

sending POST requests
sending json via POST
sending form data via POST
sending PUT, PATCH and DELETE
downloading a file
uploading a file
using ftp

running ftp commands
uploading a file via ftp
downloading a file via ftp

an overview of a curl request

a curl script consists of four basic stages:

intialization: creating the curl handle using curl_init()
setting options: using curl_setopt() to set all the options for the request; things like headers and body data and more.
execution: running the curl request with curl_exec() and, optionally, getting the response data
close: closing the handle we made with curl_init()

there are, of course, other optional components that we may use. things like curl_getinfo() to get data about a response or the CURLFile object for crafting file uploads. but the core functionality of a curl request is built with of the four basic components.

a basic `GET` request

running a basic http GET request is fairly straightforward. we initialized curl using curl_init() with the url as an argument and set the CURLOPT_RETURNTRANSFER option to true so that we can get the response body.

$url = "http://example.ca/api/bands";

// initialize curl with the target url
$ch = curl_init($url);

// set CURLOPT_RETURNTRANSFER so we get a response body
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// run the curl request and get response body
$result = curl_exec($ch);

// close curl
curl_close($ch);

optionally, we can specify the url with the CURLOPT_URL option rather than as an argument for curl_init().

$url = "http://example.ca/api/bands";

// initialize curl without an url
$ch = curl_init();

// set the url with CURLOPT_URL
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);

reusing the curl pointer for multiple requests

we can reuse a curl handle to make multiple requests by changing the url with the CURLOPT_URL option and rerunning curl_excec(). here, we create one handle, then make two different GET requests to two different urls.

$url_bands = "http://example.ca/api/bands";
$url_albums = "http://example.ca/api/albums";

// initialize curl without an url
$ch = curl_init();

// run first request
curl_setopt($ch, CURLOPT_URL, $url_bands);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);

// run second request
curl_setopt($ch, CURLOPT_URL, $url_albums);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);

curl_close($ch);

building and sending a query string

sending a query string is straightforward: we append it to the url of the request.

// url with query string
$url = "http://example.ca/api/bands?name=bratmobile";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);

query strings are annoying to build manually. fortunately, php provides the http_build_query function to turn arrays into query strings.

$args = [
    'artist' => 'monk, thelonious',
    'years' => [
        1957,
        1963,
    ],
];

// convert array to query string
$query_string = http_build_query($args);

// append to url
$url = "http://example.ca/api/bands?".$query_string;

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);

getting HTTP code with `curl_getinfo`

the http response code can be extracted from the curl handle with curl_getinfo() like this:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);

$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE); // 200

curl_getinfo() can also provide other useful information. popular data includes:

curl_getinfo($ch, CURLINFO_CONTENT_TYPE);
curl_getinfo($ch, CURLINFO_HTTP_VERSION);
curl_getinfo($ch, CURLINFO_PRIMARY_IP);
curl_getinfo($ch, CURLINFO_SIZE_DOWNLOAD);

connecting to a different port

the port can be set with the CURLOPT_PORT option:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// option to set the port
curl_setopt($ch, CURLOPT_PORT, 8080);

$result = curl_exec($ch);
curl_close($ch);

error handling

curl can throw errors; it's a good idea to test for and handle them.

we do this by calling curl_errno() to get the error number raised by curl_exec(). if the result of curl_errno() is 0, no error has ocurred.

if we have an error number greater than zero, we can get the human-readable error message with curl_error():

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);

// test for error
if(curl_errno($ch)) {
    $error_message = curl_error($ch);
    // handle errors
}

curl_close($ch);

there are a lot of potential curl errors. popular ones are:

CURLE_UNSUPPORTED_PROTOCOL (1)
CURLE_URL_MALFORMAT (3)
CURLE_COULDNT_RESOLVE_HOST (6)
CURLE_COULDNT_CONNECT (7)
CURLE_TOO_MANY_REDIRECTS (47)

treating http errors as curl errors

by default, http errors are not treated as errors by curl. we can change that by setting the CURLOPT_FAILONERROR to true. this will cause curl_errno to return 22 (CURLE_HTTP_RETURNED_ERROR) if the http server returns a code of 400 or greater.

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// treat any HTTP code >=400 as a curl error 
curl_setopt($ch, CURLOPT_FAILONERROR, true);

$result = curl_exec($ch);

// test for error
if(curl_errno($ch)) {
    $error_message = curl_error($ch);
    // handle errors
}

curl_close($ch);

sending headers

headers can be set with the CURLOPT_HEADER option. the option value passed here is an array of headers as strings:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// set the request headers
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Content-Type: application/json',
    'X-My-Special-Header: foo',
]);

$result = curl_exec($ch);
curl_close($ch);

getting headers

response headers and body are returned all in one string. we can extract a string of just the headers using substr.

first, we need to explicitly tell curl to return the headers with the option CURLOPT_HEADER.

extracting headers requires us to find the length in bytes of the header block with:

$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);

we can then use that length to write a substr command to get a string from character 0 to the the header size. this is all the headers as a string.

to get the body content, we again use substr to get a string from the end of the headers section to the end of the returned string.

it looks like this:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to get response body
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// option to get response headers
curl_setopt($ch, CURLOPT_HEADER, true);

$result = curl_exec($ch);
curl_close($ch);

// size of the headers in bytes
$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);

// string of all headers
$headersString = substr($result, 0, $headerSize);

// get body
$body = substr($result, $headerSize);

headers we get this way are a string, which is not particularly useful.

to turn that string of headers into an array of header values keyed by header names, we can do a little bit of work with explode, array_map and array_combine:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);

$result = curl_exec($ch);
curl_close($ch);

$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$headersString = substr($result, 0, $headerSize);

// get array of all header lines
$headerLines = array_values(array_filter(explode(PHP_EOL, $headersString), fn($l) => preg_match('/^[A-Za-z0-9\-]+:/', $l)));

// get header names
$headerKeys = array_map(fn($h) => substr($h, 0, strpos($h, ":")), $headerLines);

// get header values
$headerValues = array_map(fn($h) => trim(substr($h, strpos($h, ":") + 1)), $headerLines);

// combine to array of [name] = value
$headersArray = array_combine($headerKeys, $headerValues);

functions to extract response headers

here is a convenience function for converting header strings returned by curl into nice, usable arrays:

/**
 * Extract response headers from curl result as array keyed by header name
 *
 * @param string $result Return from curl_exec()
 * @param CurlHandle $ch  Return from curl_init()
 * @return array<string,string>
 */
function getCurlResponseHeaders(string $result, CurlHandle $ch): array {
    $headersString = substr($result, 0, curl_getinfo($ch, CURLINFO_HEADER_SIZE));
    $headersLines = array_values(array_filter(explode(PHP_EOL, $headersString), fn($l) => preg_match('/^[A-Za-z0-9\-]+:/', $l)));
    return array_combine(array_map(fn($h) => substr($h, 0, strpos($h, ":")), $headersLines),
        array_map(fn($h) => trim(substr($h, strpos($h, ":") + 1)), $headersLines));
}

sample output of this function will look something like this:

Array
(
    [Server] => nginx/1.18.0 (Ubuntu)
    [Content-Type] => application/json
    [Transfer-Encoding] => chunked
    [Connection] => keep-alive
    [Set-Cookie] => woot=bar
    [Cache-Control] => no-cache, private
    [Date] => Wed, 20 Aug 2025 22:18:43 GMT
    [Access-Control-Allow-Origin] => *
)

if you have two or more of the same header key, those elements will be overwritten with the last value.

this function sets header values as arrays, not strings, to allow more than one value per header key.

/**
 * Extract response headers from curl as associated array keyed by header name of arrays of header values
 *
 * @param  string $result The return from curl_exec
 * @param  CurlHandle $ch The return from curl_init
 * @return array<string, <array string>>
 */
function getCurlResponseHeadersArrays(string $result, CurlHandle $ch): array {
    $headersString = substr($result, 0, curl_getinfo($ch, CURLINFO_HEADER_SIZE));
    $headersLines = array_values(array_filter(explode(PHP_EOL, $headersString), fn($l) => preg_match('/^[A-Za-z0-9\-]+:/', $l)));
    $headerKeys = array_unique(array_map(fn($h) => substr($h, 0, strpos($h, ":")), $headersLines));

    $headersArray = [];
    foreach($headersLines as $h) {
        $headersArray[substr($h, 0, strpos($h, ":"))][] = trim(substr($h, strpos($h, ":") + 1));
    }

    return $headersArray;
}

output looks like this. notice that there are multiple values for the Set-Cookie header:

Array
(
    [Server] => Array
        (
            [0] => nginx/1.18.0 (Ubuntu)
        )

    [Content-Type] => Array
        (
            [0] => application/json
        )

    [Set-Cookie] => Array
        (
            [0] => favourite_year=1993
            [1] => preferred_format=EP
        )

)

both of these functions are available as gists.

basic auth

making requests to sites protected by basic auth can be done by setting the CURLOPT_HTTPAUTH option to CURLAUTH_BASIC and setting the CURLOPT_USERPWD option with the username and password:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// set to use basic auth
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);

// set basic auth username and password
curl_setopt($ch, CURLOPT_USERPWD, "<username>:<password>");

$result = curl_exec($ch);
curl_close($ch);

alternately, we can send an Authorization: header with the username and password base64 encoded:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// set base64 of username and password in header
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Authorization: Basic '. base64_encode("<username>:<password>")
]);

$result = curl_exec($ch);
curl_close($ch);

sending cookies

we can send cookies to a site as a string of name=value pairs using the CURLOPT_COOKIE option.

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// set cookies as string
curl_setopt($ch, CURLOPT_COOKIE, "favourite_year=1993;preferred_format=EP");

$result = curl_exec($ch);
curl_close($ch);

or, if we prefer, we can send them using the Cookie header:

$url = "http://example.ca/api/cookies";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// set cookies as string using headers
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Cookie: favourite_year=1993;preferred_format=EP',
]);

$result = curl_exec($ch);
curl_close($ch);

getting cookies

cookies set by a site are returned in the Set-Cookie header. This requires us to extract them using, for instance, the getCurlResponseHeadersArrays function shown in the section on getting headers above.

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to get response body
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// option to get response headers
curl_setopt($ch, CURLOPT_HEADER, true);

$result = curl_exec($ch);
curl_close($ch);

/**
 * Extract response headers from curl as associated array of arrays containing header values
 *
 * @param  string $result The return from curl_exec
 * @param  CurlHandle $ch The return from curl_init
 * @return array<string, <array string>>
 */
function getCurlResponseHeadersArrays(string $result, CurlHandle $ch): array {
    $headersString = substr($result, 0, curl_getinfo($ch, CURLINFO_HEADER_SIZE));
    $headersLines = array_values(array_filter(explode(PHP_EOL, $headersString), fn($l) => preg_match('/^[A-Za-z0-9\-]+:/', $l)));
    $headerKeys = array_unique(array_map(fn($h) => substr($h, 0, strpos($h, ":")), $headersLines));

    $headersArray = [];
    foreach($headersLines as $h) {
        $headersArray[substr($h, 0, strpos($h, ":"))][] = trim(substr($h, strpos($h, ":") + 1));
    }

    return $headersArray;
}

// extract cookies from headers as array
$cookiesArray = getCurlResponseHeadersArrays($result, $ch)['Set-Cookie'];

the output of this example is two cookies:

Array
(
    [0] => favourite_year=1993
    [1] => preferred_format=EP
)

cookie jars

persisting cookies between requests is probably something we will need to do. this can be done by saving cookies to the filesystem in files known colloquially as 'cookie jars'

getting cookies into file

to save the cookies a site has sent to a file, we set the CURLOPT_COOKIEJAR to the path of the file we wish to use:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to set file where downloaded cookies are stored
curl_setopt($ch, CURLOPT_COOKIEJAR, "/path/to/cookiejar");

$result = curl_exec($ch);
curl_close($ch);

if we look at the contents of a cookie jar, we can see the cookie data.

# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

example.ca  FALSE   /api/   FALSE   0   preferred_format    EP
example.ca  FALSE   /api/   FALSE   0   favourite_year  1993

like the comments say, don't edit this file.

sending cookies from a file

to send cookies from a file, we set the CURLOPT_COOKIEFILE option to the path of the file we wish to use:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to set file where cookies to send are stored
curl_setopt($ch, CURLOPT_COOKIEFILE, "/path/to/cookiejar");

$result = curl_exec($ch);
curl_close($ch);

note: the file we send cookies from is defined by CURLOPT_COOKIEFILE. the file we receive cookies into is CURLOPT_COOKIEJAR.

sending and getting cookies using a file

we can send and receive cookies using files in the same request using the same file. this is a good strategy for constantly keeping cookies fresh.

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to set file where cookies to send are stored
curl_setopt($ch, CURLOPT_COOKIEFILE, "/path/to/cookiejar");

// option to set file where downloaded cookies are stored
curl_setopt($ch, CURLOPT_COOKIEJAR, "/path/to/cookiejar");

$result = curl_exec($ch);
curl_close($ch);

sending `POST` requests

we can explicitly set the http method to POST in two ways: using the CURLOPT_POST or CURLOPT_CUSTOMREQUEST options.

to use CURLOPT_POST:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to set http method as POST
curl_setopt($ch, CURLOPT_POST, true);

$result = curl_exec($ch);
curl_close($ch);

important: using the option CURLOPT_POST automatically sends the header Content-Type: application/x-www-form-urlencoded

if we want to avoid sending that header we can use the CURLOPT_CUSTOMREQUEST option like so:

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to set http method as POST
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');

$result = curl_exec($ch);
curl_close($ch);

sending json via `POST`

sending json data via post is done by setting the CURLOPT_POSTFIELDS option with the json data and setting the Content-Type: application/json header:

$url = "http://example.ca/api/bands";

// build json for request body
$data = ['name' => 'bratmobile'];
$json = json_encode($data);

$ch = curl_init($url);

// option to set body of request
curl_setopt($ch, CURLOPT_POSTFIELDS, $json);

// header to set request content type as json
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Content-Type: application/json',
]);

$result = curl_exec($ch);
curl_close($ch);

setting CURLOPT_POSTFIELDS automatically sets the http method to POST.

sending form data via `POST`

form data can be sent via curl by setting the option CURLOPT_POSTFIELDS with the form data. form data is formatted like a query string. we can use http_build_query here to help.

$args = [
    'artist' => 'monk, thelonious',
    'years' => [
        1957,
        1963
    ]
];

// convert array to form data
$postFields = http_build_query($args);

$url = "http://example.ca/api/bands";

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// option to set post fields
curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields);

$result = curl_exec($ch);
curl_close($ch);

setting the CURLOPT_POSTFIELDS option automatically sets the method to POST and adds the request header Content-Type: application/x-www-form-urlencoded

sending `PUT`, `PATCH` and `DELETE`

the PUT, PATCH and DELETE methods can be sent using the CURLOPT_CUSTOMREQUEST option

$url = "http://example.ca/api/bands";

$ch = curl_init($url);

// option to set http method as PUT
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT'); // or PATCH or DELETE

$result = curl_exec($ch);
curl_close($ch);

downloading a file

downloading a file requires a file write handler created by fopen() to be passed to the option CURLOPT_FILE:

$url = "http://example.ca/api/bands/bratmobile.jpg";

// file pointer for downloaded file. 
$outputFp = fopen("/path/to/image.jpg", "wb");

$ch = curl_init($url);

// option to set file pointer where downloaded file will be saved
curl_setopt($ch, CURLOPT_FILE, $outputFp);

$result = curl_exec($ch);
curl_close($ch);

uploading a file

uploading a file requires creating a CURLFile object with the path and mimetype of the file to upload and then assigning that object to the CURLOPT_POSTFIELDS option.

note that postname here is the name of the file as the receiving server sees it.

$url = "http://example.ca/api/bands";

// path of file to upload
$path = "/path/to/file.txt";

// get mime type of file from content
$mime = mime_content_type($path);

// the key of the file. usually 'file'
$postname = 'file';

// create CURLFile to upload in post fields
$curlFile = new CURLFile($path, $mime, $postname);

$ch = curl_init($url);

// option to set CURLFile in post fields
curl_setopt($ch, CURLOPT_POSTFIELDS, [$curlFile->postname => $curlFile]);

$result = curl_exec($ch);
curl_close($ch);

using ftp

php has an existing suite of ftp commands that are almost certainly more useful than curl. consider using those instead.

running ftp commands

we can log into an ftp server by setting the CURLOPT_USERPWD option to the username and password of the account separated with a colon.

sending commands to the ftp server is done with the CURLOPT_CUSTOMREQUEST option.

multiple ftp commands can be run per connection.

$url = "ftp://ftp.example.ca";

// ftp username and password
$username = "someftpuser";
$password = "secretpassword";

$ch = curl_init($url);

// set username and password
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// optionally set verbose output for debugging
// curl_setopt($ch, CURLOPT_VERBOSE, true);

// execute HELP command
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'HELP');
$result = curl_exec($ch);

// execute CWD command to cd to a directory
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'CWD /path/to/dir');
$result = curl_exec($ch);

// execute LIST command to ls the current directory
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'LIST');
$directoryList = curl_exec($ch);

// output results of LIST
print_r($directoryList);

curl_close($ch);

important note: sending the HELP command to the ftp server will return a list of valid commands. only those commands will be accepted by the ftp server.

an example of the output from the HELP command is:

214-The following commands are recognized.
 ABOR ACCT ALLO APPE CDUP CWD  DELE EPRT EPSV FEAT HELP LIST MDTM MKD
 MODE NLST NOOP OPTS PASS PASV PORT PWD  QUIT REIN REST RETR RMD  RNFR
 RNTO SITE SIZE SMNT STAT STOR STOU STRU SYST TYPE USER XCUP XCWD XMKD
 XPWD XRMD

note: setting the CURLOPT_VERBOSE option to true will output all ftp traffic and is useful in inspecting/debugging. some commands, such as HELP may not return output and can only be read when the verbose option is turned on.

uploading a file via ftp

uploading a file via ftp requires several steps:

set the CURLOPT_UPLOAD option to true
set the CURLOPT_INFILE option to a file pointer of the file created by fopen
set the CURLOPT_INFILESIZE option to the size of the file from filesize()
set the url of the request to the path and name of the uploaded file

// full path to file to upload
$file = "/tmp/filetoupload";

// open file and get pointer
$fp = fopen($file, "r");

// build url including path of target file
$url = "ftp://ftp.example.ca/path/to/dir/".basename($file);

// ftp username and password
$username = "someftpuser";
$password = "secretpassword";

$ch = curl_init($url);

// set username and password
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// option to set upload
curl_setopt($ch, CURLOPT_UPLOAD, true);

// option to set pointer to file to upload
curl_setopt($ch, CURLOPT_INFILE, $fp);

// option to set size of file
curl_setopt($ch, CURLOPT_INFILESIZE, filesize($file));

$result = curl_exec($ch);

curl_close($ch);

downloading a file via ftp

downloading a file from an ftp server involves making an ftp request to the file and then writing the result of curl_exec to a file.

// path to remote file on server
$remoteFile = "/TEST_OUT/sampleout";

// path to file where the downloaded file will be saved
$localFile = "/tmp/downloadedfile";

// open download file and get pointer
$fp = fopen($localFile, "w");

// url of ftp server including path of file to download
$url = "ftp://ftp.avpbooks.com".$remoteFile;

// ftp username and password
$username = "avpftpuser";
$password = "RadialContentsGopherKelp98!3";

$ch = curl_init($url);

// set username and password
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$result = curl_exec($ch);

// write result to file
fwrite($fp, $result);

curl_close($ch);

nginx: useful basic auth

grant horwood — Wed, 13 Aug 2025 14:42:37 +0000

basic auth doesn't get the love it deserves. sure, it's not as fancy as other authentication models out there, it's basic auth, after all. but if you want to stand up some fast authentication to limit your staging site to the dev team or restrict api access to a few users, basic auth is a good choice. best of all, we can implement the whole thing in nginx so we don't have to modify our site or api's authentication.

in this article, we'll be going over setting up basic auth in nginx and getting it to do useful, practical things like restrict by specific url locations, set access by username, design a way to assign roles to users, and pass usernames into our site's code.

basic auth is so adequate for fast ad-hoc authentication right now

the flyover

this article is divided into two main parts: creating and managing user accounts with htpasswd, and configuring nginx with basic auth to restrict or allow access.

creating accounts with htpasswd
- installing apache2-utils to get htpasswd
- creating a new htpasswd file
- adding accounts to htpasswd
- adding htpasswd accounts non-interactively
- deleting htpasswd accounts
- choosing a better hashing algorithm
configuring nginx with basic auth
- protecting the entire site
- excluding locations from basic auth
- requiring auth for only certain locations
- adding further restrictions by username
- restricting by user role
- getting the username into our code

creating accounts with htpasswd

basic auth stores user accounts in a flat file as a list of name/password pairs. login credentials need to be added to this file manually. we are going to do this using the htpasswd utility.

installing apache2-utils to get htpasswd

the htpasswd command is part of the apache2-utils package. since we're using nginx, it may seem odd to need a tool from that other webserver, but nginx's basic auth feature is designed to mimic apache's older implementation. if we really want to, we can manually create a hashed password using openssl passwd -apr1 or similar and then hand edit our flat file, but why bother when htpasswd is quick, easy and free?

every major distro has an apache2-utils package. on ubuntu-like systems we can install with:

sudo apt install apache2-utils

if, for some reason, we need to build from source, we can do that from a docker file

creating a new .htpasswd file

once we have htpasswd installed, we can use it to create our flat file and add a user. the format for the command to do this is:

sudo htpasswd -c </path/to/.htpasswd> <username>

here, we run htpasswd as root and pass it the -c switch to 'create' a new password file. the arguments are the full path to where we want our file to live and the username of the account we are creating.

this command is interactive: htpasswd will prompt us for the password we want to assign to our user. for example, to create an account for the user 'molly':

sudo htpasswd -c /etc/apache2/.htpasswd molly
New password: *******
Re-type new password: *******
Adding password for user molly

the traditional location for our flat file of user accounts is:

/etc/apache2/.htpasswd

we can put it anywhere we want, but we probably shouldn't mess with tradition.

if we look at our newly-created .htpasswd file, we can see the account:

$ cat /etc/apache2/.htpasswd
molly:$apr1$LONuzAbJ$zZFjMjRDM6v3FkolReLfq0

each account lives on one line and consists of two fields: the username, in plaintext, and the hashed password. the fields are separated by a colon.

if we look at the password hash, we see that it starts with $apr1$ . this indicates which hashing algorithm was used. in this case 'apr1', which is an apache-specific algorithm that uses md5. there are other, better, hashing choices we can use. we'll go over that later.

adding accounts to htpasswd

the -c switch creates a new flat file. if you already have a file and run htpasswd -c it will delete your file and replace it. not good.

to add new accounts to the password file, we simply omit the -c switch.

sudo htpasswd </path/to/.htpasswd> <username>

note that if we want to create a user with a username that contains a space, we need to enclose the username in quotes. for instance:

sudo htpasswd /etc/apache2/.htpasswd "allison wolfe"

adding htpasswd accounts non-interactively

by default, htpasswd prompts us for a password. this is both boring and difficult to script.

if we want to add an account as one command, with the password as an argument, we can use the -b switch.

sudo htpasswd -b </path/to/.htpasswd> <username> <password>

deleting htpasswd accounts

if we want to delete an account, one option is to manually edit our .htpasswd file. the other option is to run htpasswd with the -D switch:

sudo htpasswd -D </path/to/.htpasswd> <username>

choosing a better hashing algorithm

the default hashing algorithm that htpasswd uses is 'apr1'. this is an apache-specific implementation of md5 that does an iteration of hashes using a series of random salts. is it better than raw md5? yes. but that doesn't mean it's a good or secure choice.

you’re not using md5 to hash passwords, right

a better option is bcrypt, which we can use by passing the -B switch:

sudo htpasswd -bB </path/to/.htpasswd> <username> <password>

if we want to make our bcrypt password even harder to brute-force, we can also add the -C argument to set the amount of computing time required to create the hash. the longer the computing time, the longer it takes an attacker to try every possible password.

sudo htpasswd -bB -C <time> </path/to/.htpasswd> <username> <password>

the -C argument takes an integer value for time. the minimum is 4, the default value is 5, and the maximum is 17.

if we use the maximum -C value, htpassword will take several seconds to run. for instance, this command took over twelve seconds on my machine:

sudo htpasswd -b -B -C 17 /etc/apache2/.htpasswd allison mysecretpassword

note: the password hash needs to be run every time a user authenticates. if we choose a long hashing time when we create a password it will affect user performance.

when we look at our htpasswd file after generating accounts with bcrypt hashing, we see that the algorithm field has changed:

molly:$2y$05$2k3HbYVvpiwW0.6D5zCFVOQK/RxjquivUGcr8uyC039e0qYaef1TC
allison:$2y$17$8ODsatSNUJImOsPMnSoe0uFMkpDBcnHatc9XXOY3/N857ZyVyYPmO

here, the algorithm is listed as '2y', which is bcrypt. there is also a field for the computer time value: $05$ for molly and $17$ for allison.

configuring nginx with basic auth

once we have an .htpasswd flat file, we can use it to require authentication to access our nginx-served site. we do this by modifying nginx's configuration files.

at it's core, basic auth is enabled by adding two lines:

auth_basic           "<login message>";
auth_basic_user_file /path/to/.htpasswd;

the first line here, basic_auth, sets the message that is shown in the login form. the second line, basic_auth_user_file sets the path to our .htpassword file.

protecting the entire site

to protect our entire site, we add our basic auth directives to the server block of nginx configuration:

server {

    server_name example.ca;
    root "/var/www/html/example/";

    index index.html index.htm index.php;

    charset utf-8;

    # basic auth configuratoin
    auth_basic           "Login required";
    auth_basic_user_file /etc/apache2/.htpasswd;

    location /private/ {
        try_files $uri $uri/ =404;
    }

    location /public/ {
        try_files $uri $uri/ =404;
    }

    access_log /var/log/nginx/access.log combined;
    error_log  /var/log/nginx/error.log error;

    listen 80;
}

with this config, both the private and public locations, as well as any other uri, requires a successful login to access.

excluding locations from basic auth

if we want to exclude a location or locations from requiring authentication, we can use the directive

auth_basic off;

at the location level.

here, we've turned off basic auth for the public location.

server {

    server_name example.ca;
    root "/var/www/html/example/";

    index index.html index.htm index.php;

    charset utf-8;

    # basic auth configuratoin
    auth_basic           "Login required";
    auth_basic_user_file /etc/apache2/.htpasswd;

    # authentication required
    location /private/ {
        try_files $uri $uri/ =404;
    }

    # no authentication required
    location /public/ {
        # turn off auth basic for this location
        auth_basic off;

        try_files $uri $uri/ =404;
    }

    access_log /var/log/nginx/access.log combined;
    error_log  /var/log/nginx/error.log error;

    listen 80;
}

all other locations require login.

requiring auth for only certain locations

if we configure basic auth at the location level, instead of the server level, authentication will only be required for the location.

here, we have set auth for the private and alsoprivate locations but not for public.

server {

    server_name example.ca;
    root "/var/www/html/example/";

    index index.html index.htm index.php;

    charset utf-8;

    # authentication required
    location /private/ {
        # basic auth configuratoin
        auth_basic           "Login required";
        auth_basic_user_file /etc/apache2/.htpasswd;

        try_files $uri $uri/ =404;
    }

    # authentication required
    location /alsoprivate/ {
        # basic auth configuratoin
        auth_basic           "Login required";
        auth_basic_user_file /etc/apache2/.htpasswd;

        try_files $uri $uri/ =404;
    }

    # no authentication required
    location /public/ {
        try_files $uri $uri/ =404;
    }

    access_log /var/log/nginx/access.log combined;
    error_log  /var/log/nginx/error.log error;

    listen 80;
}

adding further restrictions by username

normally, basic auth is a pretty blunt instrument: you're either authenticated and allowed to make a request or you aren't.

if we want more fine-grained control, we can limit access to specific users by their username by using the $remote_user variable.

$remote_user is a built-in variable that contains the username of the user. we can combine that with an if statement in a location block to fine tune our permissions. in this example, we have three locations:

public: no authentication.
private: any authenticated user who is not molly
molly: only molly

let's look:

server {

    server_name example.ca;
    root "/var/www/html/example/";

    index index.html index.htm index.php;

    charset utf-8;

    auth_basic           "Login required";
    auth_basic_user_file /etc/apache2/.htpasswd;

    # all authenticated users except molly permitted
    location /private/ {

        # check remote user and deny molly access
        if ($remote_user = 'molly') {
            add_header Content-Type text/html;
            return 403 '<html><body>Molly is not allowed</body></html>';
        }
        try_files $uri $uri/ =404;
    }

    # only molly permitted
    location  /molly/ {

        # check remote user and allow only molly
        if ($remote_user != 'molly') {
            add_header Content-Type text/html;
            return 403 '<html><body>Only Molly is allowed</body></html>';
        }
        try_files $uri $uri/ =404;
    }

    # no authentication required
    location /public/ {
        # turn off auth basic for this location
        auth_basic off;

        try_files $uri $uri/ =404;
    }

    access_log /var/log/nginx/access.log combined;
    error_log  /var/log/nginx/error.log error;

    listen 80;
}

here, in the private location, we test the value of $remote_user using a pretty basic if statement. if it is 'molly', we return a 403 error code and some html. all other authenticated users are permitted access:

if ($remote_user = 'molly') {
    add_header Content-Type text/html;
    return 403 '<html><body>Molly is not allowed</body></html>';
}

likewise, in the molly location, we can test for the 'not molly' condition:

if ($remote_user != 'molly') {
    add_header Content-Type text/html;
    return 403 '<html><body>Only Molly is allowed</body></html>';
}

the add_header and return directives here are how we output html from nginx. if we decide (wisely) that hardcoding content in our nginx config file is a bad idea, we can return content from a file using the rewrite directive like so:

if ($remote_user = "molly") {
    rewrite ^ /path/to/molly_not_allowed.html break;
}

restricting by user role

handling permissions by user name is useful stuff, but it can get clumsy and difficult to maintain pretty fast. it would be much more convenient if we could assign 'roles' to our users and then handle access by that.

doing this is a two-step process:

assigning users to roles using the map directive
testing the role

the 'assigning users to roles' is the new material here, so we'll look at that first. let's look at an example:

# map user names to roles
map $remote_user $role {
  molly     "dev";
  allison   "dev";
  erin      "client";
}

here, we use nginx's map directive to map a username to a role and assign that role to a variable. if you read the article on restricting nginx by ip address this should be familiar. if not, we'll go over it:

the map function takes three arguments:

the value to test: this is $remote_user, the internal variable we've used before to get the username of the authenticated user. this is the value that we will be testing in the list of test cases below.
the variable to set: the variable that will hold the result of our mapping procedure. in this case it is $role
the list of test cases: a list of tuples, essentially. if the left value of a tuple matches the value we are testing ($remote_user) then the right value is assigned to the variable we want to set ($role)

knowing this, we can see that if the value of $remote_user is 'molly', then our map call will assign the value 'dev' to the variable $role. we can then use $role in an if statement in our location blocks.

let's look at a full configuration file:

# map user names to roles
map $remote_user $role {
  molly     "dev";
  allison   "dev";
  erin      "client";
}

server {

    server_name example3.ca;
    root "/var/www/html/example3/";

    index index.html index.htm index.php;

    charset utf-8;

    auth_basic           "Login required";
    auth_basic_user_file /etc/apache2/.htpasswd;

    # role 'dev' only
    location /devonly/ {
        if ($role != 'dev') {
            add_header Content-Type text/html;
            return 403 '<html><body>Development only</body></html>';
        }

        try_files $uri $uri/ =404;
    }

    # role 'client' only
    location /client/ {
        if ($role != 'client') {
            add_header Content-Type text/html;
            return 403 '<html><body>Client only</body></html>';
        }

        try_files $uri $uri/ =404;
    }

    # authentication required
    location /staging/ {
        try_files $uri $uri/ =404;
    }

    # no authentication required
    location /public/ {
        # turn off auth basic for this location
        auth_basic off;

        try_files $uri $uri/ =404;
    }


    access_log /var/log/nginx/access.log combined;
    error_log  /var/log/nginx/error.log error;

    listen 80;
}

here, we see that in the devonly location we test the value of $role and if it is not dev, we deny access.

handling access by user role this way does require a lot of direct intervention on our behalf. we need to manually edit our nginx config file and restart the daemon every time we want to make a change. that's not the most convenient thing in the world, but if we have a small number of users and don't want to fiddle with the authentication model our site or api is using, it can be effective.

getting the username into our code

one of the great things about basic auth is that you can add it on top of your site or api's already-existing authentication structure. you can keep your site or api's login model the way it is; your code never even needs to know that basic auth is being used.

but sometimes that might be more of a bug than a feature. maybe we want our app to know the basic auth username of our visitors.

we can do this using 'fastcgi params'.

the fastcgi_param directive is a way to assign a value to a variable in our nginx configuration and then pass that variable to the fastcgi server, allowing our code to read it. if we're using php, the value is accessible in the $_SERVER array.

fastcgi_param takes two arguments: first, the name of the fastcgi parameter we want to declare and, second, the value we want to give it. to assign $remote_user to the fastcgi parameter REMOTE_USER we would do this:

fastcgi_param REMOTE_USER $remote_user;

once we have assigned that parameter, we can read it from the $_SERVER array like so:

$_SERVER['REMOTE_USER']

let's look at an example configuration for this:

server {

    server_name example.ca;
    root "/var/www/html/example/";

    index index.html index.htm index.php;

    charset utf-8;

    auth_basic           "Login required";
    auth_basic_user_file /etc/apache2/.htpasswd;

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass unix:/var/run/php/php8.3-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;

        # set $remote_user in $_SERVER['REMOTE_USER']
        fastcgi_param REMOTE_USER $remote_user;

        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_intercept_errors off;
        fastcgi_buffer_size 16k;
        fastcgi_buffers 4 16k;
        fastcgi_connect_timeout 300;
        fastcgi_send_timeout 300;
        fastcgi_read_timeout 300;
    }

    access_log /var/log/nginx/access.log combined;
    error_log  /var/log/nginx/error.log error;

    listen 80;
}

if we look inside the location ~ \.php$ block we can see the fastcgi parameter assignment.

conclusion

basic auth isn't a replacement for a real and robust login model and if you try to use it like one, you will be disappointed. however, if you need to put together a fast or ad hoc authorization system or need to add some 'extra' auth to restrict some or all of your site for internal users, it is a powerful tool. if you even casually dabble in ops, having a strong understanding of basic auth is a good thing.

php: manipulating file pointers with fseek and ftell

grant horwood — Wed, 23 Jul 2025 14:40:33 +0000

php developers generally don't put a lot of thought into reading files from disk. we usually just call file or file_get_contents or similar, shovel the whole thing into ram, and then fight with the resulting array or string. if we want to do simple things with small files, it works okay. but php can do a lot more.

in this article, we're going to look at php's file pointer and how to manipulate it using commands like fseek and ftell, and by the end we'll be able to build a pure php version of linux's tail -f command.

no one knows we can do file pointer manipulation with fseek and ftell

the sample file

any discussion of file reading needs a sample file to do that reading on. for all the examples in this article we'll be using this short, ten-line file of bands i own records by. it looks like this:

bratmobile
coltrane, john
dumb
eno, brian
fall, the
gryce, gigi
hella
idles
die kreuzen
johnson, linton kwesi

a short overview of the file pointer

in php, when we want to open a file for reading we typically use fopen:

$fp = fopen("/path/to/file", "r");

the return value from fopen is a file pointer.

file pointers keep track of where we are in the file and advance as we read data from the file. when we first create our pointer with fopen(), it is set to the beginning of the file. if we read one character from the file with fgetc(), the pointer advances one character. if we read a line with fgets(), the pointer moves to the beginning of the next line.

we can see this behaviour in action by using two loops to read parts of our file:

$fp = fopen("/path/to/bands","r");

// call fgets() three times
for($i = 0; $i < 3; $i++) {
    print fgets($fp);
}

// call fgets() three times, again
for($i = 0; $i < 3; $i++) {
    print fgets($fp);
}

the second for loop here does not read the first three lines again. instead, it picks up at line four and continue on from there. our output looks like this:

bratmobile
coltrane, john
dumb
eno, brian
fall, the
gryce, gigi

this is because the file pointer advances one line every time fgets() gets called, regardless of which loop it is in.

rewinding the file pointer

reading from a file only moves the pointer in one direction: forward. there's no way to use fgets or fgetc to move our pointer backwards.

if we want to start reading our file from beginning again, we have to explicitly set our pointer to the start of the file with rewind. let's look:

$fp = fopen("/path/to/bands","r");

// call fgets() three times
for($i = 0; $i < 3; $i++) {
    print fgets($fp);
}

// set the file pointer to the beginning of the file
rewind($fp);

// call fgets() three times, again
for($i = 0; $i < 3; $i++) {
    print fgets($fp);
}

here, we read the first three lines from our file, then set our pointer back to the file start and read the first three lines again. our output is:

bratmobile
coltrane, john
dumb
bratmobile
coltrane, john
dumb

setting the file pointer with `fseek`

if we want more control over our file pointer, we can use fseek.

fseek allows us to set our pointer to an exact byte in the file. to do this, we need to tell fseek two things:

where we are starting from, aka 'whence'
how many bytes to move the pointer, aka 'offset'

let's look at an example:

$fp = fopen("/path/to/bands", "r");

// move our pointer four bytes
fseek($fp, 4, SEEK_SET);

here, we are moving our file pointer four bytes from the beginning of our file.

the 'whence' argument here is the third one. we have three possible values we can pass:

SEEK_SET the start of the file
SEEK_END the end of the file
SEEK_CUR the current position of the file pointer

with fseek we can move our pointer anywhere we want to. for instance:

// advance pointer 10 bytes from current position
fseek($fp, 10, SEEK_CUR);

// move pointer to 15 bytes from the end of the file
fseek($fp, -15, SEEK_END);

// set point to 5 bytes from the start of the file
fseek($fp, 5, SEEK_SET);

let's look at that in action. here, we move the file pointer four bytes in from the start of the file, skipping the first four characters of 'bratmobile' and output the remainder of the line. then we move the pointer twelve bytes from the end of the file and output.

$fp = fopen("/tmp/bands","r");

// move pointer four bytes from start of file and output line
fseek($fp, 4, SEEK_SET);
print fgets($fp);

// move pointer twelve bytes from end of file and output line
fseek($fp, -12, SEEK_END);
print fgets($fp);

the results are:

mobile
linton kwesi

doing something useful with `fseek`

manipulating our file pointer like this is a great trick and it will make us very popular at parties, but it's not particularly useful. let's change that.

we're going to build a function that moves our file pointer to n lines from the end of the file so that we can do something like output the last five or ten (or whatever) lines. if you've ever used the linux tail command, you get the idea.

our approach is going to be:

loop backwards from the end of the file one byte at a time, using fseek
test each byte to see if it's the line end character PHP_EOL
count our line ends
when we get to n lines, return the file pointer

let's look at the implementation:

/**
 * Updates a file pointer $fp to be n bytes from the end of the file
 * so that outputting to eof prints the last $lineCount lines.
 *
 * @param mixed $fp
 * @param int $lineCount How many lines back from the end of file
 * @return mixed The file pointer resource
 */
function windToTailStart(mixed $fp, int $lineCount): mixed {
    $position = -1;
    $lineCounter = 0;
    do {
        fseek($fp, $position--, SEEK_END);
        if(fgetc($fp) == PHP_EOL) {
            $lineCounter++;
        }
    }
    while($lineCounter < $lineCount);
    return $fp;
}

the heart of the action here is the do while loop. this loop keeps running until the number of PHP_EOL characters we have looped over matches $lineCount, the number of lines we want our file pointer to be from the end.

inside the loop, we move our our pointer one byte back from the end of the file with the line:

fseek($fp, $position--, SEEK_END);

as we went over above, the SEEK_END argument tells fseek to start from the end of the file, and the $position argument is the number of bytes to add to the starting point. since we're moving backwards here, that number is negative.

we test to see if the byte we're inspecting is a line end character by using getc to get one char. if that byte is PHP_EOL, the line end, we increment $lineCounter. when $lineCounter equals the number of lines we want, we stop the loop and return the file pointer, which is now at the beginning of the line that is $lineCount from the end of the file.

once we have our windToTailStart function written, we can implement tail fairly easily:

// open the file
$fp = fopen("/path/to/bands","r");

// move file pointer to 3 lines from the end
$fp = windToTailStart($fp, 3); 

// output everything from $fp to the end of the file
while(!feof($fp)) {
    print fgets($fp);
}

when we run this, we get the last three lines of our 'bands' file. it looks like this:

idles
die kreuzen
johnson, linton kwesi

inspecting our pointer with `ftell`

setting our file pointer is great, but we probably also want to get it; find out exactly where our pointer is. we can do that with ftell.

ftell takes one argument, our pointer, and returns the number of bytes the pointer is away from file start. let's look:

$fp = fopen("/path/to/bands","r");

print ftell($fp); // 0

// move pointer ahead 10 bytes
fseek($fp, 10, SEEK_SET);

print ftell($fp); // 10

here we opened our file and immediately called ftell on our pointer. the result, not surprisingly, is 0. the beginning of the file.

if we wind the pointer ahead 10 bytes with fseek and call ftell again, the number we get is 10. exactly what we expect.

let's look at a slightly more practical example:

$fp = fopen("/path/to/bands","r");

while(!feof($fp)) {
    print fgetc($fp);
    if(ftell($fp) == 12) {
        break;
    }   
}

here, we've written a short block of code that outputs the first 12 bytes of our file and then exits. the implementation is basically looping over the file, outputting each character and then running ftell. when ftell returns 12, we know we're done and call break, exiting the loop. the output looks like this:

bratmobile
c

that's ten bytes for 'bratmobile', one for PHP_EOL, and 'c', the first letter in john coltrane's name.

a short note about file bounds

back in our windToTailStart function, we moved our file pointer up n lines from the file end. this works great if the file we're winding over has n or more lines in it, but if it does not then we end up pushing our file pointer past the start of the file. the results when we do this are not great.

we can protect against that by using ftell to check if we are at the start of the file. let's look:

function windToTailStart(mixed $fp, int $lineCount): mixed {
    $position = -1;
    $lineCounter = 0;
    do {

        // guard against running past start of file
        if(ftell($fp) == 0) {
            return $fp;
        }

        fseek($fp, $position--, SEEK_END);
        if(fgetc($fp) == PHP_EOL) {
            $lineCounter++;
        }
    }
    while($lineCounter < $lineCount);
    return $fp;
}

here, we've added a small if statement to return our file pointer if we've reached 0, the beginning of the file. problem solved and disaster averted.

putting it all together: writing `tail -f` in php

now that we have a grip on fseek and ftell, let's leverage them to do a moderately powerful file-reading task: implementing linux's tail -f functionality in php.

for those not familiar with tail -f, it allows us to 'watch' a file. we run tail -f /path/to/file and the command waits for new data to be added to that file and then immediately outputs it. it's great for watching things like logs in real time.

to write this, we're going to use ftell to find the end of our file and 'bookmark' it. then, in a loop, we're going to continuously check the end of the file with fseek. if our new end-of-file is different than our 'bookmark', we'll know new data has been added to the file. we can then output the new lines.

let's look at the implementation:

function tailf(string $file) {

    // open file at the end
    $fp = fopen($file, 'r');
    fseek($fp, 0, SEEK_END);

    // infinite loop to keep checking file
    while(true) {
        // bookmark current end of file
        $tell = ftell($fp);

        // wait for 0.1 seconds
        usleep(100000);

        // move pointer to end of file
        fseek($fp, 0, SEEK_END);

        // if the end of file is not the same as the bookmar there is new data
        if($tell != ftell($fp)) {
            // move pointer to bookmark
            fseek($fp, $tell, SEEK_SET);

            // output everything from bookmark to end of file
            while(!feof($fp)) {
                print fgets($fp);
            }
        }
    }
}

as we see here, we can monitor if new data has been added to our file by getting the pointer for the file end with ftell, waiting for 0.1 seconds then seeing if our new file end is different than the previous one. if we have new data, we move the file pointer back to the old end of file and output from there.

for those who are interested, there is a gist of the full implementation of tail -f in php.

a more comprehensive example of tail -f functionality is available in my internal-use logger package.

wrapping up

convenience functions like file_get_contents are great; they're powerful and easy-to-use. but, ultimately, they exist for convenience. the underpinning of every file operation in php is the file pointer, and once we understand how to track and manipulate that we have far greater control over our file handling.

linux: looking under the hood of neofetch

grant horwood — Mon, 07 Jul 2025 15:21:22 +0000

a couple of months ago, some semi-famous video game vlogger i've never heard of decided to get up on the linux soap box and rally his followers to ditch windows.

this was one of those viral-level, meme-type event things, and the result was a literal flood of screenshots of 'riced' desktops. and while each of those arch-newbie screeners was different than the last, they all had one common element: a terminal running neofetch.

this article is not going to be about neofetch specifically, but rather about the concepts and commands under neofetch's hood. we're not going to rebuild this command, or go over it line-by-line. instead, we're going to look at the basic tools we can use to inspect our linux system. by the end, we should be able to learn everything we want to about our linux-like system using standard, built-in commands and files. no neofetch required.

neofetch has always been a stack of linux commands, files and variables in a trenchcoat

inspecting our os with `uname`

uname stands for 'unix name' and, not surprisingly, it's job is to give us the name of our operating system. but it also does a lot more. let's start with the name:

$ uname -o
GNU/Linux

the -o switch here tells uname to tell us about the 'operating system'. the response is simple, but accurate: GNU/Linux.

if we want to get some basic information on our system's 'processor', we use the -p switch.

$ uname -p
x86_64

getting the exact kernel version we're running is accomplished with -r.

$ uname -r
6.12.10-76061203-generic

files with more useful os information

the data from uname is correct, but not particularly useful. if we want something more detailed, there are some files we can look at, though.

the `os-release` file

the /etc/os-release file contains a lot of data about our system. this file is part of the systemd standard and has been around since 2012, so if you're running a moderately modern linux-like operating system with systemd, it will be there.

let's look at an example of its contents:

NAME="Pop!_OS"
VERSION="22.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 22.04 LTS"
VERSION_ID="22.04"
HOME_URL="https://pop.system76.com"
SUPPORT_URL="https://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=jammy
UBUNTU_CODENAME=jammy
LOGO=distributor-logo-pop-os

from this file, we can see that the user here is running pop_os 22.04. that's some useful information!

the `lsb-release` file

an alternative to the os-release file is /etc/lsb-release. the 'lsb' here stands for 'linux standard base' which was an attempt made many years ago to unify and standardize all sorts of things across the various linux distros. stuff like the filesystem hierarchy, library versions, run levels and the like.

like most attempts at unification and standardization, it fell apart and lsb hasn't been a going concern for over ten years now. however, there's still a lot of lsb stuff lingering around in modern distros, and it's useful.

let's look at some sample output of /etc/lsb-release:

DISTRIB_ID=Pop
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Pop!_OS 22.04 LTS"

exactly the sort of data we want.

our system may also include the binary lsb_release (mileage may vary). if it does, we can run it like so:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Pop
Description:    Pop!_OS 22.04 LTS
Release:    22.04
Codename:   jammy

‘linux standard base’ was abandoned in 2015

learning about memory with `free`

memory is complicated stuff. the traditional way to find out about free and used memory was to look in the /proc/meminfo file, but it's an absolute mess in there.

if you want to stick with tradition, you can use grep to get just the 'good bits' from /proc/meminfo. something like this:

$ cat /proc/meminfo | grep -i "MemTotal\|MemFree"
MemTotal:       45141040 kB
MemFree:         6225932 kB

the alternative is to use the free command, which reports on 'free' memory:

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            43Gi        11Gi       6.0Gi       396Mi        25Gi        27Gi
Swap:           19Gi       6.5Gi        13Gi

here we have passed the -h switch, which stands for 'human readable', ie megabytes and gigabytes instead of just raw kb.

there are two rows in the standard output: 'Mem' for physical, installed memory, and 'Swap' for swap space. if you don't have a 'Swap' row, you should probably fix that. we'll focus on physical memory.

the two columns we're probably the most interested in are 'total' (the total amount of installed memory), and 'free'.

'free' is calculated by adding the 'used' memory and the 'buffer/cache' memory and subtracting it from the total. this means that 'used' plus 'buffer/cache' plus 'free' adds up to our total memory:

$ free | grep 'Mem' | awk '{print $2 " = "$3" + "$4" + "$6}'
45141040 = 12235120 + 6080968 + 26824952
$ expr 12235120 + 6080968 + 26824952
45141040

and it does.

inspecting our hardware with `lshw`

the command lshw's long name is 'hardware listener', but everybody just says 'list hardware' instead because that's what it does: it lists all our machine's hardware.

let's run it with the -short argument:

lshw -short

the output of this is a short description of every piece of hardware in our box. everything from our motherboard to our mouse to our cd burner, if we're old enough to still have such a thing. if we want to, we can apply a little grep to filter for the information we need. for instance, to get just our processor and cdrom drive we could do:

$ sudo lshw -short | grep -i "processor\|cdrom"
/0/4                               processor      Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
/0/100/1d/1/1/3/0.0.0  /dev/cdrom  disk           DVDRAM GP65NB60

if we run lshw -short and look a the top line of the output, we see that the columns have these headings:

H/W path        Device      Class          Description
======================================================

if we want to do a more detailed exploration of our hardware, we will use the value under 'Class'. so, for instance, for our cpu, the class is 'processor'.

to inspect the hardware class, we use the -C switch and the class name like so:

$ lshw -C processor
  *-cpu                     
       product: Intel(R) Core(TM) i7-4820K CPU @ 3.70GHz
       vendor: Intel Corp.
       physical id: 1
       bus info: cpu@0
       version: 6.62.4
       size: 1200MHz
       capacity: 3900MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts vnmi md_clear flush_l1d cpufreq
       configuration: microcode=1070

that's a lot of information about our cpu.

we can do this with any class that lshw lists. for instance, if we want info on our video card, we can query the class 'display' (with a little bit of grep to cut out the noise)

$ lshw -C display | grep "product\|vendor"
       product: GP104 [GeForce GTX 1070]
       vendor: NVIDIA Corporation

getting info about hard disks with `df`

we can get some basic, but useful information about our hard drives using the df command. 'df' here stands for 'disk free' because most people use it to find out how much free space they have, but it does more than that. let's look at some truncated sample output:

$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
tmpfs                        4.4G  2.5M  4.4G   1% /run
/dev/sda1                    1.8T  791G  946G  46% /
192.168.1.70:/volume1/music  3.6T  457G  3.2T  13% /mnt/music

in this example we've run df with the -h switch to use 'human readable' units.

from this we can see that there is one 1.8 terabyte hard drive mounted on the root partition and it's 46% full. there's also a network drive mounted on mnt/music.

for most purposes, this level of detail is sufficient. however, if we want a bit more, we can use lsblk to list all our 'block devices'.

lsblk

the output of this command will show drives (including things like our cdrom) and their partitions. it will also show every single snap we have installed as a device.

if we want to know the filesystems of our drives, we can pass the --fs switch

lsblk --fs

finding our screen resolution

if we want to get our current screen resolutions and all possible supported resolutions, we can use xrandr.

as that 'x' at the beginning of the name implies, xrandr is an xwindow tool. if we are using wayland, instead, we'll need to find an alternate solution such as wlr-randr

running xrander gives us the following output:

$ xrandr
Screen 0: minimum 8 x 8, current 2560 x 1440, maximum 32767 x 32767
DVI-D-0 disconnected (normal left inverted right x axis y axis)
HDMI-0 connected primary 2560x1440+0+0 (normal left inverted right x axis y axis) 697mm x 392mm
   3840x2160     60.00 +  59.94    50.00    30.00    29.97    25.00    23.98  
   2560x1440     59.95* 
   1920x1080     60.00    59.94    50.00    29.97    23.98  
   1680x1050     59.95  
   1600x900      60.00  
   1440x900      59.89  
   1280x1024     75.02    60.02  
   1280x800      59.81  
   1280x720      60.00    59.94    50.00  
   1152x864      75.00  
   1024x768      75.03    70.07    60.00  
   800x600       75.00    72.19    60.32    56.25  
   720x576       50.00  
   720x480       59.94  
   640x480       75.00    72.81    59.94

here we can see that our current resolution is 2560x1440. there's also a lengthy output of supported resolutions.

deducing our desktop environment

lots of people use graphical desktop environments in linux, so finding out what's being run is probably important. we can get that information from the $XDG_CURRENT_DESKTOP environmental variable:

$ echo $XDG_CURRENT_DESKTOP
pop:GNOME

once we know what desktop environment is being run, we can call it directly to get the version number:

$ gnome-shell --version
GNOME Shell 42.9

for kde, we would use:

plasmashell --version

other desktop environments will have their own commands that accept the --version argument.

getting the current shell

the default shell of the current user is stored in the $SHELL environmental variable. we can access it like this:

$ echo $SHELL
/usr/bin/fish

if we want to retrieve the version of the shell, we can use $SHELL as an executable and pass the --version argument

$ $SHELL --version
fish, version 3.7.1

if we need to get the shell for a user other than the one we are logged in as, we can always retrieve it from the /etc/passwd file:

$ cat /etc/passwd | grep ghorwood
ghorwood:x:1000:1000:grant horwood:/home/ghorwood:/usr/bin/fish

finding up time with `uptime`

it's a ridiculous point of pride for some linux users that they never have to reboot their machines. we can find out how long our box has been up and running with the uptime command:

$ uptime -p
up 2 weeks, 4 days, 18 hours

eighteen days isn't bad.

Conclusion

neofetch is a useful and fun tool, but learning how it finds data about our machine gives us a better understanding of our operating system.

🔎 this post originally appeared in the grant horwood technical blog

php: writing command-line applications with macrame. pt 2

grant horwood — Wed, 19 Feb 2025 14:49:20 +0000

in this series, we've been looking at how to write command line applications in php using macrame. in the previous installment, we covered the basic structure of macrame scripts, getting user input both as text and from interactive menus, parsing command-line arguments, and styling our output text.

the sample application we're building is a script that fetches a list of a mastodon user's followers and outputs the data in a nicely-formatted table. it looks like this:

the sample script in action

the flyover

in this installment we'll be looking at:

outputting array data as a nicely-formatted ascii table
running a function in the background while we show users an animated spinner
writing safely to files
paging long output
basic notice level output

before we start, the (skipable) mastodon stuff

our example script scrapes a list of one user's followers from a mastodon instance. this, of course, requires us to make some calls to the mastodon api.

there are the two functions our script has that do this:

mastodonUserId(): accepts a mastodon user's username and calls the instance to get the user id
mastodonFollowers(): accepts the user's id and returns a list of follower data.

we won't be covering this functionality in this post. the full script is available as a github gist.

outputting arrays in a nice table format

macrame allows us to output array data in formatted ascii tables. if you've ever selected stuff from mysql on the command line, it looks like that.

let's look at the tableFollowers function that accepts an array of follower data from mastodon and returns it as a table:

function tableFollowers(array $followers, Macrame $macrame): string
{
    // extract the 'acct' and 'display_name' fields from each element for use as table rows
    $data = array_map(fn ($f) => [$f->acct, $f->display_name], $followers);

    $headers = ['Account', 'Display Name'];

    // build the ascii table and return
    return $macrame->table($headers, $data)->get();
}

we can gloss over the array_map call here; all it does is extract the account and display names of the followers from the array of data from mastodon. the important line here is:

return $macrame->table($headers, $data)->get();

here, we used macrame's table() method to create a macrame table object. the table() method takes two arguments: an array of column headers, and an array of arrays that represent the rows of our table.

once we've made our table object, we use it to create and return our table by calling the get() method. if we instead wanted to just write our table to the screen, we could call write() instead.

if we were creating this table using hardcoded data, it might look something like this:

$headers = ['Account', 'Display Name'];
$data = [
    ['ghorwood', 'grant horwood'],
    ['thephp', 'The PHP Foundation'],
];

$table = $macrame->table($headers, $data)->get();

the value in our $table variable would be a string that looks like this:

+----------+--------------------+
| Account  | Display Name       |
+----------+--------------------+
| ghorwood | grant horwood      |
| thephp   | The PHP Foundation |
+----------+--------------------+

a nice, clean table.

macrame also allows us to set the alignment of columns and change the border style if we so wish. for instance, if we want to right-align our Account column and apply a solid border, we could write:

$table = $macrame->table($headers, $data)
                 ->right(0)
                 ->solid()
                 ->get();

here, we added a call to the right method to right-align the first column (column zero). we also applied the solid method to set the border style to a solid line. the output is:

┌──────────┬────────────────────┐
│  Account │ Display Name       │
├──────────┼────────────────────┤
│ ghorwood │ grant horwood      │
│   thephp │ The PHP Foundation │
└──────────┴────────────────────┘

macrame's table feature is based on the the tabletown library and is designed to handle multibyte content, output rows with line breaks as multi-row cells, and deal with tabs as proper tab stops.

a full overview of the table feature is available on the table documentation page.

running code in the background and showing an animated spinner.

it's nice to show our users an animated spinner while we do time-intensive tasks in the background so they know that the script hasn't hung and is actually doing things. doing this in php, however, is not very straightforward since it requires us to run two different functions at the same time -- one to do the work, the other to show the spinner -- and php is single threaded. macrame's spinner feature allows us to do this by forking a separate process.

let's look at how we could show an animated spinner to our users while we fetch and build our table of followers. this code lives in the main body of our script.

$getFollowersTable = function (string $username, string $instance, Macrame $macrame) {
    // call mastodon to get the user id for the username
    $userId = mastodonUserId($username, $instance, $macrame);

    // call mastodon to get the followers for the user id
    $followers = mastodonFollowers($userId, $instance, $macrame);

    // format the array of followers into a table
    $followersTable = tableFollowers($followers, $macrame);

    return $followersTable;
};

/**
 * Run the $getFollowersTable() function in the background and display an animated
 * spinner to the user while it is running.
 */
$followersTable = $macrame->spinner('cycle 2')
                       ->prompt('fetching ')
                       ->speed('fast')
                       ->run($getFollowersTable, [$username, $instance, $macrame]);

the spinner features allows us to show an animated spinner to our user while a function is run in the background, so the first step is to construct that function.

here, we have created an anonymous function and assigned it to the variable $getFollowersTable. in the body of the function, we're calling mastodon to get a list of followers and formatting the response as a table. the function accepts three arguments and returns a string.

the run method on spinner is where we actually run the function. run accepts two arguments. the first is the anonymous function, and the second is an array containing the values we want to pass as arguments to that function.

calling run will automatically execute our anonymous function in the background while showing the animated spinner to the user. when the anonymous function terminates, run will return its return value. in our example, the string containing the table of followers will be set in $followersTable.

the spinner feature allows a fair amount of customization. to change the type of animation we display, we can pass the name of the animation to spinner as an argument. there are about forty animations to choose from. if no argument is passed to spinner, the default animation is used.

we can also add a string that will preceed the animation by adding an optional call to prompt. in the example above, we are outputting the word 'fetching' in front of the spinner.

lastly, we can adjust the speed of the animation with the optional speed method. we can pass one of four strings to speed: 'slow', 'medium', 'fast', or 'very fast'.

there is a full overview of the spinner feature in the manual.

writing to files safely

when we write php for the web, we don't normally need to put a lot of thought into file access. we control the server, after all, so we're confident about things like the directory structure, permissions, how much disk space we have and so on. that's not the case for command line scripts that are going to be run on some random user's computer.

macrame's file feature has error checking for common read and write problems built in. let's look at how we would write a feature for our script that allows users to output the list of mastodon followers to a file:

// get optional outfile file path argument
$outfile = $macrame->args('outfile')->first();

if ($outfile) {
    $macrame->file($outfile)->write($followersTable);
}

here, we're accepting a path to an output file as a command line arg. we covered how to do this in part one of this series. we're then taking that file path and writing our table of followers to it.

the file method here takes as an argument the path to the file we want to write to. the write method's argument is the contents we wish to write. if we wanted to append to the file, we would use the append method instead.

when we call write or append, macrame checks first to see if we can write to the file. the checks performed are:

if the path is valid
permissions on the file if the file exists
permissions on the directory if the file is to be created
if there is sufficient disk space on the target partition

if any of these checks fail, error messages will be printed to the screen and the method will return false. if everything goes well, true is returned and our content is written to file.

we can also perform the checks manually if we want to override the default behaviour:

// bool. Test if file can be written to
if(!$macrame->file('/path/to/file')->writable()) {
    // error. cannot write due to permissions
}

if(!$macrame->file('/path/to/file')->enoughSpace('content')) {
    // error. cannot write due to lack of disk space
}

here, we're checking the permissions of our target file (or directory if the file does not exist yet), and confirming that we have sufficient space to write '$content'.

reading files operates similarly to writing them:

$contents = $macrame->file('/path/to/file')->read();

if there are any errors on this attempted read, they will be displayed to the user and the method will return null.

macrame file paths are automatically expand the ~ character to the user's home directory.

temporary files

if we want to create temporary files that are automatically deleted when the script terminates, we can apply the deleteOnExit method:

$macrame->file('/path/to/file')->deleteOnExit()->write('contents');

doing this flags the file to be removed when our script calls:

$macrame->exit();

the full list of file features is covered in the manual

paging long output

our list of followers might be too long to fit all in one screen. we could just let the output zip by and let whoever's running our script use the mouse wheel to get to the top, but that's rude.

macrame allows us to page long output with the page method. let's apply paging to our table of followers:

$macrame->text($followersTable)->page();

here, we create a text object of our table string using text and then, instead of calling write to print it to the screen, we call page to print it to the screen one page at a time.

our user's will be able to advance the output one page using the <SPACE> bar. hitting <RETURN> will move it forward one line.

outputting notices

keeping users informed of how the script is running is a good thing to do. if a tasks is completed successfully, it's nice to output an 'OK' message. if there's an error, we should show them some 'ERROR' text, preferably with some red in there.

macrame provides a number of convenience methods for text to allow us to write notices to the screen.

$macrame->text('things went well')->ok();
$macrame->text('things did not go well')->error();

the first line here prints our message to the screen with a nice, green OK. the second one has a red ERROR.

these convenience methods follow (loosely) the levels outlined in RFC 5424. there's a full list of them in the documentation

putting it all together

now that we have a grip on building command line interfaces, we can assemble the mastodon follower app.

#!/usr/bin/env php 
<?php

/**
 * Example Macrame script that fetches mastodon followers.
 *
 * https://macrame.fruitbat.studio/Installation.html
 * https://github.com/gbhorwood/Macrame
 */

require __DIR__ . '/vendor/autoload.php';

use Gbhorwood\Macrame\Macrame;

/**
 * Instantiate a Macrame object.
 * The argument is the name of the script as seen by ps(1)
 */
$macrame = new Macrame("Example Macrame Script");

/**
 * Enforce that the script only runs if executed on the command line
 */
if ($macrame->running()) {

    // ENTRY POINT

    /**
     * Validate that the host system can run Macrame scripts. Exit on failure
     */
    $macrame->preflight();

    /**
     * Handle the -v or --version arguments
     */
    if ($macrame->args('v')->exists() || $macrame->args('version')->exists()) {
        $macrame->text('1.0')->write();
        $macrame->exit();
    }

    /**
     * Handle the --instance= argument if present, or poll user for instance
     * with dynamic menu if not.
     */
    $instance = $macrame->args('instance')->first();
    if (!$instance) {
        $instance = menuInstance($macrame);
    }

    /**
     * Handle the --username= argument if present, or poll user for username
     * with text input if not
     */
    $username = $macrame->args('username')->first();
    if (!$username) {
        $username = inputUsername($instance, $macrame);
    }

    /**
     * Read the value of the --outfile= argument if any
     */
    $outfile = $macrame->args('outfile')->first();

    /**
     * A function to fetch an ascii table of followers from mastodon for the user
     * defined by $username and $instance.
     * @param  string $username
     * @param  string $instance
     * @param  Macrame $macrame
     * @return string
     */
    $getFollowersTable = function (string $username, string $instance, Macrame $macrame) {
        // call mastodon to get the user id for the username
        $userId = mastodonUserId($username, $instance, $macrame);

        // call mastodon to get the followers for the user id
        $followers = mastodonFollowers($userId, $instance, $macrame);

        // format the array of followers into a table
        $followersTable = tableFollowers($followers, $macrame);

        return $followersTable;
    };

    /**
     * Run the $getFollowersTable() function in the background and display an animated
     * spinner to the user while it is running.
     */
    $followersTable = $macrame->spinner('cycle 2')
                           ->prompt('fetching ')
                           ->speed('fast')
                           ->run($getFollowersTable, [$username, $instance, $macrame]);

    /**
     * If the --outfile= argument was passed, write the followers table to the file
     * otherwise write the followers table to STDOUT, paged to screen height
     */
    if ($outfile) {
        $macrame->file($outfile)->write($followersTable);
    } else {
        $macrame->text($followersTable)->page();
    }

    // exit cleanly
    $macrame->exit();
}

/**
 * Display a dynamic menu of instances to the user, return
 * the selected text.
 *
 * @param  Macrame $macrame
 * @return string
 */
function menuInstance(Macrame $macrame): string
{
    $header = "Select your instance";

    // list of options in the menu
    $instances = [
        'phpc.social',
        'mastodon.social',
        'mstdn.ca',
    ];

    // display the menu, return the selected text
    return $macrame->menu()
                   ->erase() // erase the menu after selection
                   ->interactive($instances, $header);
}

/**
 * Display a text input to the user, return the input text.
 *
 * @param  string $instance
 * @param  Macrame $macrame
 * @return string
 */
function inputUsername(string $instance, Macrame $macrame): string
{
    // the prompt for the text input, with bold styling using tags
    $prompt = $macrame->text("<!BOLD!>Username<!CLOSE!> (for $instance): ")
                      ->get();

    /* alternate method to apply bold styling:
    $prompt = $macrame->text('Username ')
                      ->style('bold')
                      ->get()."(for $instance): ";
     */

    // read one line of user input, return the text
    return $macrame->input()
                   ->readline($prompt);
}

/**
 * Convert the followers data returned by the mastodon instance into an ascii
 * table and return.
 *
 * @param  array $followers The array returned by mastodonFollowers()
 * @param  Macrame $macrame
 * @return string
 */
function tableFollowers(array $followers, Macrame $macrame): string
{
    // extract the 'acct' and 'display_name' fields from each element for use as table rows
    $data = array_map(fn ($f) => [$f->acct, $f->display_name], $followers);

    $headers = ['Account', 'Display Name'];

    // build the ascii table and return
    return $macrame->table($headers, $data)->get();
}

/**
 * Fetch the id of the user $username from the mastodon instance $instance
 *
 * @param  string $username
 * @param  string $instance
 * @param  Macrame $macrame
 * @return int
 */
function mastodonUserId(string $username, string $instance, Macrame $macrame): int
{
    $url = "https://$instance/api/v1/accounts/lookup?acct=$username";

    // make the api call and handle errors
    try {
        return get($url, $macrame)->id;
    } catch (\Exception $e) {
        // display error text and exit the script
        $macrame->text("User '$username' not found on '$instance'")->error();
        $macrame->exit();
    }
}

/**
 * Fetch the array of the followers for the user $userId from the mastodon instance $instance
 *
 * @param  string $userId
 * @param  string $instance
 * @param  Macrame $macrame
 * @return int
 */
function mastodonFollowers(int $userId, string $instance, Macrame $macrame): array
{
    $url = "https://$instance/api/v1/accounts/$userId/followers?limit=80";

    try {
        return get($url, $macrame);
    } catch (\Exception $e) {
        $macrame->text("Followers for '$username' not found on '$instance'")->error();
        $macrame->exit();
    }
}

/**
 * Execute curl GET call to $url and return result
 *
 * @param  string $url
 * @param  Macrame $macrame
 * @return mixed
 */
function get(string $url, Macrame $macrame)
{
    $headers = [
        'Accept: application/json',
    ];

    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    $result = curl_exec($ch);
    $header = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);

    if ($header !== 201 && $header !== 200) {
        curl_close($ch);
        $macrame->text("Call to $url returned $header")->warning();
        throw new \Exception();
    }

    curl_close($ch);

    return json_decode($result);
}

the complete source for this project is also available as a gist.

🔎 this post originally appeared in the grant horwood technical blog

php: powerful sorting with usort

grant horwood — Fri, 07 Feb 2025 16:03:39 +0000

i recently worked on a rescue project where the original dev wanted to sort an array of item objects first by manufacturer name and then by price, descending. their 'solution' was an eighty-line foreach mess of fiddling with array keys. it was difficult to read, more difficult to modify, and didn't even work correctly lots of the time. i replaced the whole thing with a five-line usort call.

usort stands for 'user-defined sort'. it allows us, as devs, to write our own sorting rules (plural!) and makes working with arrays of complex data easier.

in this post, we're going to take a thorough look at sorting arrays with usort. we'll go over the basics of how the function works, write some ascending and descending sorts on integers and strings, and then cover how to sort on multiple values.

best other sorting functions can do for you is one rule on an array of primitives

our sample array

all the examples we'll be looking at will be using an array of objects that represents a small, but well-curated, collection of vinyl record albums. it looks like this.

$albums = [
    (object)[
        'artist' => 'Bratmobile',
        'title' => 'Pottymouth',
        'year' => 1993,
    ],
    (object)[
        'artist' => 'Monk, Thelonious',
        'title' => 'Brilliant Corners',
        'year' => 1957,
    ],
    (object)[
        'artist' => 'Fugazi',
        'title' => 'In on the kill taker',
        'year' => 1993,
    ],
    (object)[
        'artist' => 'Monk, Thelonious',
        'title' => '5 by Monk by 5',
        'year' => 1959,
    ],
];

the basics of `usort`

usort takes two arguments: the array that we want to sort, and a callable function that defines the rule(s) we want to use for sorting.

the array argument is passed by reference. this means that the array is modified in place. usort does not return a new, sorted, array. rather, the original array is sorted.

the callable argument is a function that, itself, takes two arguments. these two arguments represent two arbitrary elements of the array we want to sort, and the return value of the callable is an integer that indicates which of those two arguments should be considered 'higher' in the sort order.

this integer must be one of:

0 if the two arguments are equal
1 if the left argument is greater than the one on the right
-1 if the right argument is greater

we can do any comparison we want in the body of callable. this is the power of usort.

let's look at some examples.

basic sorting: albums by year

let's start with doing some straightforward sorting on a nice, clean integer value: the year our albums were released.

here's what that would look like:

// sorting callable
$sortByYearAsc = function($a, $b) {
    if($a->year == $b->year) {
        return 0;
    }

    if($a->year > $b->year) {
        return 1;
    }

    if($a->year < $b->year) {
        return -1;
    }
};

// calling usort
usort($albums, $sortByYearAsc);

the thing to pay attention to here is how we've constructed our callable.

we've created an anonymous function and assigned it to a variable. our function accepts two arguments, representing two arbitrary elements: the individual album objects in the array we're sorting.

in the body of the function, we do our comparison testing on the year element of the album objects. if the year of the left argument is higher, we return 1. if the right argument is higher, -1, and if the two years are equal, we return 0.

if we print_r() our sorted array it will be sorted by year, exactly as we expect:

Array
(
    [0] => stdClass Object
        (
            [artist] => Monk, Thelonious
            [title] => Brilliant Corners
            [year] => 1957
        )

    [1] => stdClass Object
        (
            [artist] => Monk, Thelonious
            [title] => 5 by Monk by 5
            [year] => 1959
        )

    [2] => stdClass Object
        (
            [artist] => Bratmobile
            [title] => Pottymouth
            [year] => 1993
        )

    [3] => stdClass Object
        (
            [artist] => Fugazi
            [title] => In on the kill taker
            [year] => 1993
        )

)

ascending and descending sorts

our current callable sorts our albums by year ascending, ie. oldest first.

if we want to change to a descending order -- put the newest albums at the top -- all we have to do is reverse our comparisons in our callable:

$sortByYearDesc = function($a, $b) {
    if($a->year == $b->year) {
        return 0;
    }

    if($a->year > $b->year) {
        return -1;
    }

    if($a->year < $b->year) {
        return 1;
    }
};

here, we've changed it so that instead of 1 being returned if the left year is greater and -1 if the right year is greater, the inverse is true. we've basically just switched the 1 and -1.

the result is that the comparisons still work, just in reverse, and we get our descending sort.

enter the 'spaceship operator'

our sorting callables work, but they are ugly. three if statements in a row? that's code smell.

sure, we could tighten them up with an else if and a default return value, but there's a much cleaner solution: the spaceship operator

let's look at an example:

$a <=> $b;

the spaceship operator itself is the <=>. if you squint, it kind of looks like a crude ascii art of a spaceship. hence the name.

when we compare two values with the spaceship operator we get zero if both sides are equal, 1 if the left side is greater, and -1 if the right side is greater. this is exactly the sort of comparison we want to do for usort()!

let's rewrite our callable to use the spaceship operator:

$sortByYearAsc = function($a, $b) {
    return $a->year <=> $b->year;
};

this does everything that our ugly, three-if statement function did and is much cleaner and terser.

if we want to, we can tighten this up even more by rewriting it as an arrow function

$sortByYearAdc = fn($a, $b) => $a->year <=> $b->year;

and, if we want to change the sort order from ascending to descending, all we need to do is switch the operands: swap $a and $b.

sorting on strings

nobody sorts their album collection by release year. what we probably want to do is sort alphabetically by artist.

fortunately, php has some built-in functions for doing string comparisons:

strcmp compares two strings
strcasecmp compares two strings without case sensitivity

both these functions take two arguments: the strings to compare. and both behave like the spaceship operator, returning 1 if the left argument is greater, -1 if the right argument is greater, and 0 if the arguments are the same.

let's look at how we would write a callable to sort on artist using strcasecmp:

$sortByArtistAsc = fn($a, $b) => strcasecmp($a->artist, $b->artist);

usort($albums, $sortByArtistAscSpaceship);

in our callable here we have replaced our spaceship operator comparing years with a call to strcasecmp to compare artists. the results are precisely what we want:

Array
(
    [0] => stdClass Object
        (
            [artist] => Bratmobile
            [title] => Pottymouth
            [year] => 1993
        )

    [1] => stdClass Object
        (
            [artist] => Fugazi
            [title] => In on the kill taker
            [year] => 1993
        )

    [2] => stdClass Object
        (
            [artist] => Monk, Thelonious
            [title] => 5 by Monk by 5
            [year] => 1959
        )

    [3] => stdClass Object
        (
            [artist] => Monk, Thelonious
            [title] => Brilliant Corners
            [year] => 1957
        )

)

sorting on multiple elements

people who really care about how their albums are organized may argue about a lot of things (is "iggy pop" filed under "i" or "p"? we have opinions), but there is one thing everyone agrees on: albums are filed alphabetically by artist, and the albums of each artist are filed in chronological order of release.

this means that to properly sort our array of albums, we need to sort by artist first, and then by year. we're looking here to replicate the functionality of sql's

ORDER BY artist ASC, year ASC

this is precisely the sort of thing that usort excels at. let's look at how we might do that:

$sortByArtistThenYear = function($a, $b) {
    if(strcasecmp($a->artist, $b->artist) == 0) {
        return $a->year <=> $b->year;
    }

    return strcasecmp($a->artist, $b->artist);
};

usort($albums, $sortByArtistThenYear);

here, we're just taking the things we've already covered -- string comparison with strcasecmp and integer comparison with the spaceship operator -- and combining them.

first, we see if the two artist fields are the same by testing them with strcasecmp. if they are, we know that we need to sort the albums for that artist by year. we do that with the spaceship operator.

if the artist fields are not the same, we do our standard sorting on the artist field.

the result is that our array is sorted ascending by artist and, if there is more than one record with the same artist, those records are sorted ascending by year. it looks like this:

Array
(
    [0] => stdClass Object
        (
            [artist] => Bratmobile
            [title] => Pottymouth
            [year] => 1993
        )

    [1] => stdClass Object
        (
            [artist] => Fugazi
            [title] => In on the kill taker
            [year] => 1993
        )

    [2] => stdClass Object
        (
            [artist] => Monk, Thelonious
            [title] => Brilliant Corners
            [year] => 1957
        )

    [3] => stdClass Object
        (
            [artist] => Monk, Thelonious
            [title] => 5 by Monk by 5
            [year] => 1959
        )

)

conclusion

usort is an incredibly powerful tool and once we understand how to write sort rules for it we can order arrays of complex data any way we want to.

🔎 this post originally appeared in the grant horwood technical blog

php: writing command-line applications with macrame. pt 1

grant horwood — Wed, 29 Jan 2025 19:18:09 +0000

php doesn't get a lot of attention as a command line scripting language. which is a shame, since php has a lot of features that make it a good choice for writing terminal apps.

in this series, we'll be going over writing interactive command line scripts using the macrame library. we'll be working through building an example project, a script that fetches a list of a mastodon user's followers, from start to end, and will cover topics such as getting and validating user input, building interactive menus, handling command line arguments, accessing files safely, styling output text, and running functions in the background while showing our users an animated spinner.

further information on macrame can be found on the documentation site.

the sample project

the project we will be working through is a simple command-line script that returns a list of a mastodon user's followers. running it looks like this:

the sample script in action

the user selects the desired mastodon instance from a dynamic menu, enters the username as free text, and the script shows an animated spinner while fetching the data. the output is a nice, ascii-style table.

the complete source code for this project is available as a gist for anyone who wants to skip ahead.

the flyover

in this installment, we will go over how to:

install macrame
scaffold a blank script
read command line arguments
create a dynamic menu
read a line of user input (with optional validation)
styling output text

install macrame

macrame is installed via composer

composer require gbhorwood/macrame

scaffolding your script

once we have macrame installed, we can set up a basic 'hello world' script and use that as our starting boilerplate. although this scaffolding is technically not required, using it will make our script a little safer and more compliant. let's look at the code:

#!/usr/bin/env php 
<?php
require __DIR__ . '/vendor/autoload.php';

use Gbhorwood\Macrame\Macrame;

// Instantiate a Macrame object.
// The argument is the name of the script as seen by ps(1)
$macrame = new Macrame("Example Macrame script");

// Enforce that the script only runs if executed on the command line
if($macrame->running()) {

    // Validate that the host system can run Macrame scripts. Exit on failure
    $macrame->preflight();

    // output text to STDOUT
    $macrame->text("hello world")->write();

    // exit cleanly
    $macrame->exit();
}

although this isn't a whole lot of lines, there's a lot happening here. let's go over it.

#!/usr/bin/env php

this line is the 'shebang'. basically, it tells our linux-like operating system which interpreter to use to run this script. this allows us to run our script without having to type php first. the shebang must be the first line in the file, even before <?php.

$macrame = new Macrame("Example Macrame script");

here, we create a Macrame object that we will use throughout the rest of our script. pretty standard stuff. the only interesting part is the argument. this is the name that the operating system will give to our script. for instance, if we run ps to show a list of running processes, our script will show up with this name.

if($macrame->running())

this statement ensures that all the code inside the block will only execute if the script has been run on the command line.

$macrame->preflight();

when we write php for the command line, we do not have as much control over the environment as we do on a webserver that we own and manage. the preflight() call here tests the local php environment and, if it does not meet the minimum requirements, terminates the script with an error message. the minimum requirements are:

php 7.4
posix extension
mbstring extension

note: although macrame does run on php 7.4 and 8.0, due to changes in how php handles multibyte strings in 8.1, strings containing emojis may not be aligned properly in output on php pre-8.1.

$macrame->exit();

this exits the script cleanly, returning a success code of 0. additionally, any temporary files created during execution will be automatically deleted. using macrame's exit() function is preferable to php's die();

running hello world

once we have our basic 'hello world' script written, we can set its permissions to allow execution and run it on the command line.

chmod 755 ./examplescript.php
./examplescript.php

reading arguments

macrame provides a set of tools for parsing and reading command line arguments. let's start with something straightforward: getting the version number when the script is called with:

./examplescript.php --version
# or 
./examplescript.php -v

for this case, all we need to do is check whether either of those arguments exist.

we can do this by calling the args() method on our macrame object, which returns an object containing all of our script's arguments and a suite of methods we can use to inspect them. to test if an argument exists, we can use the exists() method like so:

if ($macrame->args('v')->exists() || $macrame->args('version')->exists()) {
    $macrame->text('1.0')->write(); // output the version number
    $macrame->exit();
}

the exists() method returns a boolean.

🔎 macrame calls are designed to be chained.

command line arguments can also be used to assign values to variables. for instance, to set the username value, we will probably want our users to be able to call the script like this:

./examplescript.php --username=ghorwood

to get the value of this argument, in our script we can use the first() method provided by args() like so:

$username = $macrame->args('username')->first();

as the name implies, the first() method returns the value of the first occurrence of our argument. if we called our script like this:

./examplescript.php --username=firstuser --username=seconduser

then first() would return the value 'firstuser'. if we want the last value, we can call last(). if we want all the values as an array, we would use all().

putting this all together, our script now looks like this:

#!/usr/bin/env php 
<?php
require __DIR__ . '/vendor/autoload.php';

use Gbhorwood\Macrame\Macrame;

$macrame = new Macrame("Example Macrame script");

if($macrame->running()) {

    $macrame->preflight();

    /**
     * Handle the -v or --version arguments
     */
    if ($macrame->args('v')->exists() || $macrame->args('version')->exists()) {
        $macrame->text('1.0')->write();
        $macrame->exit();
    }

    /**
     * Accept first value of --instance=
     */
    $instance = $macrame->args('instance')->first();

    /**
     * Accept first value of --username=
     */
    $username = $macrame->args('username')->first();

    $macrame->exit();
}

the full list of methods for handling command line args is covered in the macrame documentation on arguments

creating a dynamic menu

we're also going to want to give our users the option to use our script interactively. if they don't pass an argument on the command line, we will prompt them to input the data. for the mastodon instance value, we're going to use a menu.

macrame menus are dynamic; our users naivgate them by using the arrow keys to move up or down the list and then hit <RETURN> to make their selection. let's write a function that displays a menu to the user and returns the selected value:

function menuInstance(Macrame $macrame): string
{
    $header = "Select your instance";

    // list of options in the menu
    $instances = [
        'phpc.social',
        'mastodon.social',
        'mstdn.ca',
    ];

    // display the menu, return the selected text
    return $macrame->menu()
                   ->erase() // erase the menu after selection
                   ->interactive($instances, $header);
}

the core functionality here is a call to:

$macrame->menu()->interactive(<array of options>, <header>);

we provide an array of strings to display as the menu options and an optional header text for the menu to menu()->interactive() and the menu is automatically displayed to the user. the user's selection is returned as a string.

there is also the option to erase the menu from the screen after the user has made their selection by adding a call to erase() to our chain. this method is optional, but does keep things clean.

once we have our menu function, we can modify how we get the mastodon instance. we will try reading it from the command line arguments and, if no value has be

as the name implies, the first() method returns the value of the first occurrence of our argument. if we called our script like this:

en passed, call our menuInstance() function.

$instance = $macrame->args('instance')->first();
if (!$instance) {
    $instance = menuInstance($macrame);
}

a side note on styling menus

out of the box, macrame uses the terminal's default style and colour for menus and sets the highlighted item in reverse. we can alter this if we want by adding a few extra methods to our chain. for instance, if we would prefer that our highlighted item was shown as bold, red text, we could write:

$macrame->menu()
        ->styleSelected('bold')
        ->colourSelected('red') // colorSelected() also works
        ->interactive(<array of options>, <header>);

there is a complete overview on the menu documentation page of all the methods available to customize the colour, style and alignment of menus.

reading a line of user input

next, we're going to modify how we get the username to also accept interactive input. in this case, we're going to read a sting of user input text using input()->readline(). here's the function:

function inputUsername(string $instance, Macrame $macrame): string
{
    // the prompt for the text input, with bold styling using tags
    $prompt = $macrame->text("<!BOLD!>Username<!CLOSE!> (for $instance): ")
                      ->get();

    /* alternate method to apply bold styling:
    $prompt = $macrame->text('Username ')
                      ->style('bold')
                      ->get()."(for $instance): ";
     */

    // read one line of user input, return the text
    return $macrame->input()
                   ->readline($prompt);
}

the last line of this function is where we poll the user for input. the readline() method accepts an optional $prompt argument; the text we display to the user telling them what they should enter. the return value is the user input as a string.

a side note on input validation

users make mistakes. that's why input validation is important.

macrame comes with a number of pre-set methods to validate input. we can add as many of them as we want to our chain and if any of the validators fail, the user will be prompted to input again. the input()->readline() function will not return a value until all validators pass.

let's look at an example:

return $macrame->input()
               ->isLengthMin(4, 'must be at least 4 characters long')
               ->doesNotContain('@', 'do not use the @ symbol')
               ->readline($prompt);

here, we apply two validation test: the text must be four or more characters and must not contain the '@' symbol. for both those validation methods, the second argument is the error message we will show to the user if the validation fails.

a full list of the pre-built validation functions is available on the macrame input documentation page. if we want to write our own custom validators, that is covered, too.

what about 'dots echo' output?

if our users are inputting sensitive data, like passwords, we'll probably not want to echo their keystrokes back to the terminal where nosey shoulder-surfers can read it.

to address this, macrame provides a 'dots echo' version of readline() called readPassword().

$macrame->input()->readPassword("enter sensitive data: ");

each keystroke read by readPassword() is echoed back as an asterisk.

styling text

in the example of how to read a line of user text, we saw a bunch of code for styling the prompt text. let's look at that in more detail.

macrame allows for styling text output to the terminal using ansi codes which allow us to apply both styles such as bold and italic, and colours to our text.

we can do this in our script one of two ways. there are methods such as style(), and colour() (or color()), or we can use a basic tagging system with text.

let's look at the method approach first.

$prompt = $macrame->text('Username ')
                  ->style('bold')
                  ->colour('blue')
                  ->get()."(for $instance): ";

here, we created a 'text' object using macrame's text() method, then applied a style and colour before returning it as a string using get().

note that the style and colour methods are applied to all the text in the string. if we want to mix in styled and coloured text with plain text, we will have to create a number of substrings and concatenate them together. this can be cumbersome, especially if we're dealing with large amounts of text.

alternately, we can use macrame's tagging system to make styling our text a little easier. here's an example:

$prompt = $macrame->text("<!BOLD!>Username<!CLOSE!> (for $instance): ")
                  ->get();

the text between the <!BOLD!> and <!CLOSE!> tags will, unsurprisingly, be bolded. there is a full list of all the tags in the documentation.

an important thing to note is that the <!CLOSE!> tag closes all of the preceding tags. this is due to the behaviour of ANSI escape codes.

this means that nesting tags does not work the way we might expect. for instance, in this example, the first <!CLOSE!> tag closes both the <!BOLD!> and <!RED!> tags:

"<!RED!>this is red <!BOLD!>this is red and bold<!CLOSE!> this is plain<!CLOSE!>";

the script so far

our example script so far looks like:

#!/usr/bin/env php
<?php

require __DIR__ . '/vendor/autoload.php';

use Gbhorwood\Macrame\Macrame;

// Instantiate a Macrame object.
// The argument is the name of the script as seen by ps(1)
$macrame = new Macrame("Example Macrame script");

// Enforce that the script only runs if executed on the command line
if ($macrame->running()) {

    // Validate that the host system can run Macrame scripts. Exit on failure
    $macrame->preflight();

    /**
     * Handle the -v or --version arguments
     */
    if ($macrame->args('v')->exists() || $macrame->args('version')->exists()) {
        $macrame->text('1.0')->write();
        $macrame->exit();
    }

    /**
     * Handle the --instance= argument if present, or poll user for instance
     * with dynamic menu if not.
     */
    $instance = $macrame->args('instance')->first();
    if (!$instance) {
        $instance = menuInstance($macrame);
    }

    /**
     * Handle the --username= argument if present, or poll user for username
     * with text input if not
     */
    $username = $macrame->args('username')->first();
    if (!$username) {
        $username = inputUsername($instance, $macrame);
    }


    // exit cleanly
    $macrame->exit();
}

/**
 * Display a dynamic menu of instances to the user, return
 * the selected text.
 *
 * @param  Macrame $macrame
 * @return string
 */
function menuInstance(Macrame $macrame): string
{
    $header = "Select your instance";

    // list of options in the menu
    $instances = [
        'phpc.social',
        'mastodon.social',
        'mstdn.ca',
    ];

    // display the menu, return the selected text
    return $macrame->menu()
                   ->erase() // erase the menu after selection
                   ->interactive($instances, $header);
}

/**
 * Display a text input to the user, return the input text.
 *
 * @param  string $instance
 * @param  Macrame $macrame
 * @return string
 */
function inputUsername(string $instance, Macrame $macrame): string
{
    // the prompt for the text input, with bold styling using tags
    $prompt = $macrame->text("<!BOLD!>Username<!CLOSE!> (for $instance): ")
                      ->get();

    /* alternate method to apply bold styling:
    $prompt = $macrame->text('Username ')
                      ->style('bold')
                      ->get()."(for $instance): ";
     */

    // read one line of user input, return the text
    return $macrame->input()
                   ->doesNotContain('@', 'do not use the @ symbol')
                   ->readline($prompt);
}

what's next

so far, we've covered reading command line arguments, getting user input from menus and as text, and doing some basic text styling for output. in the next post, we will go over:

running a function in the background while we show users an animated spinner
writing safely to files
outputting array data as a nicely-formatted ascii table
paging long output
basic notice level output

🔎 this post originally appeared in the grant horwood technical blog

vim: five tips to become a more mediocre vim user

grant horwood — Mon, 30 Sep 2024 16:35:15 +0000

i am a mediocre vim user. sure, i've been using vim for over twenty years, have handcrafted a custom vimrc, and even written a syntax file, but for my day-to-day usage, my skills are resoundingly and undeniably mediocre.

this is a good place to be. people who are 'power users' spend so much time fiddling with plugins and configurations to build their perfect homerolled ide that they never get any real work done. vim becomes their personality, not their tool. by contrast, novices don't have the skill or knowledge to leverage vim's power. they thrash around, treating vim like a cumbersome and difficult nano; they google how to quit. that's not good for productivity!

the sweet spot is in the middle: mediocrity.

this post is going to go over five vim features that will get you on the path from novice to mediocre.

nobody cares if you're mediocre at vim

doing bookmarks with `m` and `'`

vim allows you to set a bookmark on a given line and jump back to that line at any time. this is a useful feature if you don't want to be like my former co-worker wes who used to keep a scratchpad of Important Line Numbers beside his keyboard.

bookmarks are set by using m plus a one-letter identifier in command mode. so, for instance, to set a bookmark labelled 'a', we would do

<ESC>ma

since bookmarks have a one-letter label, we can have a whole bunch of them. we can have an 'a' bookmark and a 'b' one and so on, until we run out of alphabet.

to jump back to our bookmark, in command mode, we use ' (the single quote) plus the identifier of the bookmark we want to jump to. so, ie.

<ESC>'a

would bring us back to bookmark 'a'.

dangerous 'power user' territory for bookmarks

if we want to venture outside the comfortable zone of mediocrity, bookmarks have plenty of other features we can obsess over. we can jump to not just the line, but the column of our bookmark using the backtick instead of the single quote. we can list and manage our bookmarks with :marks. we can delete them with :delm. lots of stuff, which is all covered on the vim fandom page on bookmarks.

doing shell escapes with `!!`

we can dump the output of any shell command straight into our vim buffer by using a 'shell escape'.

shell escapes are called by typing !! in command mode, followed by the shell command we want to run. for example, if we wanted to insert the current date as output by bash's date command, we would:

<ESC>!!date

or, if we wanted to dump the contents of another file into the file we're editing, we could use cat like so:

<ESC>!!cat /path/to/file

even better, when we shell escape with !!, the content of the current line is used as input for our shell command. so, if we wanted to, say, replace all the letter 'e's in the current line with the number 3, we could use tr like so:

<ESC>!!tr 'e' '3'

this would take the contents of the current line and pipe it into our tr command. the output of tr would then overwrite our line in vim.

doing better shell escapes with `!}`

the !! shell escape is great, but it only works on the current line. truly mediocre vim users can use !} to pipe an entire paragraph into a shell escape.

for example, if we have a paragraph in vim that is a list of rock bands, we can sort that entire list alphabetically by moving our cursor to the first line of the paragraph and applying bash's sort command like so:

<ESC>!}sort

the entire paragraph is sent as input to sort, and sort's output replaces the entire paragraph.

if we have a decent understanding of shell tools like awk and sed, this can allow us to do extremely mediocre things. for instance, if we want to take that entire list of bands and enclose each line in quotes and number them, we could do:

!}!awk '{print NR". \""$0"\""}'

dangerous 'power user' territory for shell escaping

there's lots of shell escaping power outside the comforting boundaries of mediocrity, and we can go there if we feel brave. there's stuff like using :call system() or suppressing output with :silent!! and other rabbit holes we can venture down instead of getting stuff done. there's a fair overview of these in this post on executing shell commands in vim.

doing autocomplete with `^p` and `^n`

autocompletion is a handy tool. we all use very descriptive and, consequently, very long names for variables and functions like good programmers, but the punishment for that is all the typing and the risk of errors. autocompletion takes that pain away.

vim allows us to do autocompletion with ^p (that's the control key and 'p' as a 'chord') and ^n in insert mode.

using ^p will search the file we're editing from the current line up until it finds the first word that matches the text we've typed. the 'p' stands for 'previous', as 'up the file'. if we like that suggestion, we can hit <SPACE> and accept it. if we don't like it, we can just hit ^p again (and again and again) until we find what we're looking for. if our search gets to the top of the file, vim wraps around and starts searching from the bottom.

the ^n combination works the exact same as ^p except is searches down file. the 'n' stands for 'next'.

a note on the dangers of `^p` and `^n`

if we're using a vim keybinding emulator in another program like an ide, ^p probably won't work. instead of doing something useful like autocompletion, ^p will probably be bound to a little-used feature called 'print', which is a way to make a copy of your file onto a piece of paper so you can recycle it. likewise, ^n will probably create for you a 'new' file, as if editing the current file wasn't already enough.

doing block indentation with visual mode

vim does a pretty good job of autoindenting, but there are times when we make a mess of our file and need to add some manual spaces to keep things neat and tidy.

vim allows us to indent entire blocks by selecting one or more lines in the visual mode and then applying > to indent or < to unindent.

doing this is a two-step process: the selecting-lines-in-visual-mode step, and the indenting-the-selected-lines step. let's look at an example of how we would indent one block of four lines one tab stop:

<ESC><SHIFT>vjjj>

let's pick this apart.

to select our block of four lines, we first enter command mode with <ESC>. standard stuff. from there, we enter visual mode with <SHIFT>v. this highlights our current line. that line is our block. we then use j to move our cursor down three lines. this adds those lines to our block. we now have four lines highlighted.

we then hit the > key. this moves the entire block of highlighted lines one tabstop to the right. indentation achieved.

if we want to indent that block another tabstop, we can then type . (a period). the period key tells vim to redo the previous command. we can hammer on that period as much as we want to keep indenting until our block is where we want it.

to unindent, all we have to do is hit < after selecting our block.

dangerous 'power user' territory for visual blocks

it's easy to get carried away with visual blocks; we can do a lot of stuff. in addition to using it for a block of lines, there's a character mode and a mode that allows us to create a block in the middle of a file. we can also use selected blocks as input for other vim commands with the output going back into that block. this even includes shell escapes. the 'linux handbook' has a good overview of visual mode for people who want to flirt with being a power user.

conclusion

vim has a reputation for being difficult. some of that reputation is earned, to be sure, especially in this modern point-and-click world, but a lot of it has been fostered by power users looking to show off and novices loudly expressing their frustration at the entry barrier.

but vim doesn't need to be difficult. we don't need to be those power users with forty plugins and a five hundred-line .vimrc. we can edit text effectively and quickly in vim with only a limited -- mediocre -- skillset.

🔎 this post originally appeared in the grant horwood technical blog

mysql: “thai food near me”, or: doing geo distance calculations in your database.

grant horwood — Fri, 20 Sep 2024 14:46:38 +0000

we're all familiar with the whole "thai food near me" thing. you type that phrase into your phone and it responds with a list of thai restaurants that are, well, near you. and we have a kind-of understanding of how that works under the hood: google or whoever has a database of thai restaurants with their latitudes and longitudes and knows our location from our phone and then does 'some process' to figure out which thai places are nearby.

in this post,we'll be going over that 'some process' part, looking at how to use mysql to do some standard location stuff. we'll cover mysql's POINT and POLYGON types, finding the distance between two points on a sphere (which the earth, contrary to what you may have read on the internet, is), determining if a point is inside of a polygon defined by points, and look at things like 'spatial reference systems' which define how coordinates are plotted on the surface of the earth.

a restaurant attempts an sql injection attack.

making a `POINT` in mysql

mysql has a whole suite of functions and data types devoted to spatial data. the number of them is dizzying and the official documentation is almost criminally dense. fortunately, we can accomplish what we want to do using only a small subset. we'll start with POINT.

POINT is both a datatype and a function that returns that data type. if we wanted to define a point on a good, old-fashioned x/y graph, we can do it like so:

SELECT POINT(3, 7);

the result of that query is our x/y point in a value of type POINT. mysql stores POINT in a binary format, so the result of our select is not particularly useful:

SELECT POINT(3, 7);
+------------------------------------------------------+
| POINT(3, 7)                                          |
+------------------------------------------------------+
| 0x00000000010100000000000000000008400000000000001C40 |
+------------------------------------------------------+

mysql addresses this by providing two convenience functions to extract the x and y values from a point:

ST_X()
ST_Y()

they both accept a POINT value as an argument. for instance:

SELECT ST_X(POINT(3,7)) AS x, ST_Y(POINT(3,7)) AS y;
+------+------+
| x    | y    |
+------+------+
|    3 |    7 |
+------+------+

because POINT is a data type, we can use it in table definitions, just like we would INT or VARCHAR.

CREATE TABLE `some_coords` (
  `coords` POINT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

if we have a column of type POINT, only POINT data can go in there. we'll cover this more later.

a short digression on x, y, maps and a lack of standards

we all learned in school how to plot points on blue-lined graph paper using the x-axis, which runs horizontally, and the y-axis, which is vertical. points were defined as x/y; horizontal first, vertical second. this is the way it has been forever, and everyone agrees on it.

except the people who make maps.

the people who make maps define points as latitude/longitude. latitude, of course, runs north-south, which is vertical on a map. longitude, the east-west axis, is horizontal. the map people, in essence, decided to use y/x.

obviously, this creates problems. let's look at what happens when we create a POINT representing the location of the ship & anchor pub in central calgary, alberta (where i have been known, on occasion, to blog from)

SELECT ST_X(POINT(51.037913, -114.073277)) as longitude, ST_Y(POINT(51.037913, -114.073277)) as latitude;
+-----------+-------------+
| longitude | latitude    |
+-----------+-------------+
| 51.037913 | -114.073277 |
+-----------+-------------+

latitude and longitude are mixed up; our pub is in the wrong place. what's worse, since the maximum value for latitude is 90, we've put the ship & anchor somewhere out in space. not good.

mysql addresses this issue by providing two functions to replace ST_X() and ST_Y() when using points on a map or globe:

ST_Latitude()
ST_Longitude()

this is good stuff, except, if we try to use them in our above query, we get this error message:

ERROR 3726 (22S00): Function st_latitude is only defined for geographic spatial reference systems, but one of its arguments is in SRID 0, which is not geographic.

this error looks daunting (what the hell is SRID 0?), but all mysql is telling us here is that the POINTs we're using haven't been defined as being map points. they're just regular, old bags of x's and y's.

we'll go over SRIDs and SRSs later on.

latitude and longitude has always been y/x.

a better way to make a `POINT`: well-known text

so far, we've selected a value of type POINT by using the function POINT(). this works fine for now, but there is a better, more-flexible way to do this that will make working with POINTs and POLYGONs easier when things start getting more complicated.

the ST_GeomFromText() function takes as an argument a text expression (a string) of the geometric object we want to create (a POINT in this case), and returns a value of the correct type.

these text expressions are formatted using a syntax called "well-known text". the format is, basically, the name of the geometric object you want to create (ie. POINT) and the coordinates that define it. let's look:

SELECT ST_GeomFromText('POINT(51.037913 -114.073277)');

this looks very straightforward, but there's a glaring question: where is the comma separating the arguments in our POINT call?

the answer is that the well-known text here isn't a call to the function POINT(), it's a definition of the data type POINT.

back at the beginning of this discussion, we went over how POINT is both a function and a datatype. when we use POINT() as a function, the coordinates are arguments that are separated by a comma. when we define a value using POINT as a type, the coordinates do not take a comma.

we can use ST_GeomFromText() to create any sort of geometric object that's defined in the well-known text. there aren't many of these, and we'll be sticking in this post to POINTs and POLYGONs (which include things like squares and triangles).

spatial reference systems: not all points are the same

on my desk i have a small chess board where i occasionally work through annotated games. it's my idea of "fun". that chess board is a coordinates system. i also have a large, widescreen computer monitor on my desk. it's a coordinate system as well.

however, just because my chess board and monitor are both coordinate systems doesn't mean that the coordinates from one can be transferred to the other. the x/y position of my white bishop is meaningless on my monitor; that x/y point only has meaning in the context of the chess board.

a context defines things like the origin points, axes, units of measurement and the like. useful stuff that helps us make sense of what a coordinate actually means.

when it comes to plotting points and lines and polygons on the surface of the earth, that context is called a 'spatial reference system', or SRS.

there are a lot of different SRSs. a lot. some of them treat the earth as a sphere, others as a projected flat map. some cover the entire planet, many more only deal with a sub region, like a country. some include a z axis from the center of the earth, most don't.

if we want to peruse all the different SRSs that mysql has, we can run this select:

SELECT * FROM INFORMATION_SCHEMA.ST_SPATIAL_REFERENCE_SYSTEMS;

there are about five thousand of them.

fortunately, we don't need to read through all of these to choose one. we're just going to use 4326, a global, unprojected coordinate system that (just about) everybody uses.

that 4326 number is the id of the SRS. it's called, unsurprisingly, an SRID. if we remember back to when we tried to call the ST_Latitude() function on the POINT we made, we got the error:

ERROR 3726 (22S00): Function st_latitude is only defined for geographic spatial reference systems, but one of its arguments is in SRID 0, which is not geographic.

now that we have more of an understanding about SRSs, we can see that here mysql is complaining that we are asking for the latitude, but the SRS of our POINT isn't one that uses latitude and longitude. the SRS we are using, according to the error message, is SRID 0.

SRID 0 is just a 'flat, cartesian plane' with no units. think of it as a sheet of that blue-lined graph paper from math class stretching off into infinity in all directions. this is a great SRS for some applications, but is not very meaningful for using latitude and longitude to map places on a spherical earth. SRID 0 is the default SRS that mysql assigns to POINTs (and other shapes) when one is not specified.

by comparison, the 4326 SRS is specifically designed for global mapping. it treats the surface of the earth as an ellipsoid, uses degrees for measurement and defines the axes as the equator and prime meridian. exactly what we want. 4326 is, in turn, based on a big set of data about the earth called the world geodetic system 1984, or WSG84, that was compiled in that year in an effort to unify and standardize the mishmash of national mapping data. if you're one of those 'further reading' types, you can read over a detailed explainer on SRID 4326 here or peruse the surprisingly-entertaining wikipedia entry on WSG84.

a developer accidentally uses SRID 0 for their geolocation select.

actually using `SRID` 4326

using SRID 4326 as our SRS when creating a POINT is pretty straightforward; we just add the SRID as a second argument to ST_GeomFromText().

SELECT ST_GeomFromText('POINT(51.037913 -114.073277)', 4326);

and, just like that, our x/y values are now treated as longitude and latitude coordinates on earth. let's try ST_Latitude() again:

SELECT ST_Latitude(ST_GeomFromText('POINT(51.037913 -114.073277)', 4326)) AS latitude;
+-----------+
| latitude  |
+-----------+
| 51.037913 |
+-----------+

exactly what we wanted.

creating a table for our `POINT`s

selecting geometric data like POINTs (or POLYGONs or LINESTRINGs) created using literal data is fine, but what we probably want to do is persist that data in a table so we can use it later. let's do that. we'll start with creating our table.

CREATE TABLE `calgary` (
  `id` int unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
  `coords` POINT NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

here, we've defined a pretty standard-looking table of notable locations in the city of calgary, alberta. the interesting column here is coords, which is defined as a POINT.

that POINT doesn't have an SRS associated with it. this means that on every insert, we will have to define the SRID we are using for our point. this is very flexible, but if we want to we can add the SRS to the column definition.

CREATE TABLE `calgary` (
  `id` int unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(200) COLLATE utf8mb4_unicode_ci NOT NULL,
  `coords` POINT SRID 4326 NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

by defining our coords column as POINT SRID 4326 we are enforcing that any POINT in that column must be of SRID 4326. if we try to insert a point that has a different SRID, mysql will complain with an error like:

ERROR 3643 (HY000): The SRID of the geometry does not match the SRID of the column 'coords'. The SRID of the geometry is 0, but the SRID of the column is 4326. Consider changing the SRID of the geometry or the SRID property of the column.

for all the examples going forward, we will be using a table with a coords column that does not define the SRID.

now that we have a table, we can insert some rows. we'll add a list of calgary landmarks.

INSERT INTO calgary VALUES (null, 'calgary tower', ST_GeomFromText('POINT(51.044270 -114.062019)', 4326));
INSERT INTO calgary VALUES (null, 'peace bridge', ST_GeomFromText('POINT(51.0542 -114.0793)', 4326));
INSERT INTO calgary VALUES (null, 'saddledome', ST_GeomFromText('POINT(51.0374 -114.0519)', 4326));
INSERT INTO calgary VALUES (null, 'national music centre', ST_GeomFromText('POINT(51.04250 -114.06083)', 4326));
INSERT INTO calgary VALUES (null, 'baitun nur mosque', ST_GeomFromText('POINT(51.101743 -113.972039)', 4326));
INSERT INTO calgary VALUES (null, 'olympic oval', ST_GeomFromText('POINT(51.07694 -114.13556)', 4326));
INSERT INTO calgary VALUES (null, 'heritage park', ST_GeomFromText('POINT(50.98528 -114.10833)', 4326));
INSERT INTO calgary VALUES (null, 'international avenue', ST_GeomFromText('POINT(51.03778 -113.98167)', 4326));
INSERT INTO calgary VALUES (null, 'fort calgary', ST_GeomFromText('POINT(51.045139 -114.045778)', 4326));

there's a lot of things to see in calgary!

in these insert statements, we create our point using ST_GeomFromText() and set the SRID as 4326 like so:

ST_GeomFromText('POINT(51.0542 -114.0793)', 4326)

we can then select this data back, getting the latitude and longitude of each location with ST_latitude() and ST_longitude().

SELECT  id,
        name,
        ST_Latitude(coords) AS latitude,
        ST_Longitude(coords) AS longitude
FROM    calgary;
+----+-----------------------+-----------+-------------+
| id | name                  | latitude  | longitude   |
+----+-----------------------+-----------+-------------+
|  1 | calgary tower         |  51.04427 | -114.062019 |
|  2 | peace bridge          |   51.0542 |   -114.0793 |
|  3 | saddledome            |   51.0374 |   -114.0519 |
|  4 | national music centre |   51.0425 |  -114.06083 |
|  5 | baitun nur mosque     | 51.101743 | -113.972039 |
|  6 | olympic oval          |  51.07694 |  -114.13556 |
|  7 | heritage park         |  50.98528 |  -114.10833 |
|  8 | international avenue  |  51.03778 |  -113.98167 |
|  9 | fort calgary          | 51.045139 | -114.045778 |
+----+-----------------------+-----------+-------------+

at last, calculating distance

so far, we've made some spatial POINTs and assigned them to SRID 4326 so we can actually make sense of them as latitude and longitude. it's finally time to focus on what we really want to do: getting the distance between two points.

to do this, we're going to use mysql's ST_Distance_Sphere() function.

as one would expect, ST_Distance_Sphere() calculates the distance between two points, provided as arguments to the function, on a sphere. the distance returned will always be the shortest one (since, on a sphere, we can always go the opposite direction and travel further to get to the same place). the unit of measurement is meters.

ST_Distance_Sphere() takes an optional third argument: the radius of the sphere. if we do not set this argument, the value 6,370,986 meters is used. that's the radius of the earth, and is the value we almost certainly want to use.

knowing all that, an example select would look like:

SELECT  name,
        ST_Distance_Sphere(ST_GeomFromText('POINT(51.037913 -114.073277)', 4326), coords) AS distance_meters
FROM calgary;
+-----------------------+--------------------+
| name                  | distance_meters    |
+-----------------------+--------------------+
| calgary tower         | 1057.9217149476015 |
| peace bridge          |  1859.336539883446 |
| saddledome            | 1495.7790780297603 |
| national music centre | 1008.7085120625501 |
| baitun nur mosque     |  10020.62038333001 |
| olympic oval          | 6146.6116509785015 |
| heritage park         |  6345.541637300453 |
| international avenue  |  6405.199613693066 |
| fort calgary          |  2083.730747912871 |
+-----------------------+--------------------+

here we can see that we passed two POINT arguments to ST_Distance_Sphere(). The first is one we constructed from literal values using ST_GeomFromText(). it's the location of the ship & anchor pub in central calgary, where i promise i am not writing this post. the second argument is our coords column.

the result is the distance from our starting POINT, the ship & anchor, to all the POINTs in our table, in meters.

from here, building 'near me' functionality is just a matter of applying a WHERE or ORDER BY clause.

going regional: finding points inside a square (or any shape)

perhaps, instead of a basic 'near me' feature, we want our users to be able to draw a square on a map and say "show me all the calgary landmarks in here."

to do this, the fist step we need to take is defining a square.

creating a square

a square is a type of polygon, and mysql provides a POLYGON data type that we can use to describe a square (or any shape). POLYGONs are defined by a set of coordinates that identify the corners of the shape. this means, to create a square, we provide POLYGON with five coordinate sets.

wait, five? don't we mean four? a square has four corners, after all.

the important thing to note here is that a polygon must be closed. this means that the first coordinate set and the last coordinate set must be the same. it completes the shape by going back to the beginning. the result is that a square is defined has having five sets of coordinates. to illustrate, let's look at this glorious ascii diagram that shows the five coordinates that create a square.

1/5 ---- 4
  |      |    
  |      |    
  2 ---- 3

with that in mind, we can create a square of latitude and longitude values. the example we'll be using is this square covering most of downtown calgary.

a square covering most of downtown calgary.

to select this as a POLYGON in mysql, we would do:

SELECT ST_GeomFromText('POLYGON( (  51.053913 -114.094391, 51.028008 -114.094391, 51.028008 -114.037743, 51.053913 -114.037743, 51.053913 -114.094391) )', 4326);

given our experience creating a POINT, this should be fairly straightforward. the only difference is that instead of passing one coordinate set to POINT, we pass five to POLYGON. the result is a geometric shape, stored in a binary format, that we can use for comparisons against POINTS or, even, other POLYGONs.

finding `POINT`s 'within' a square

we now have a POLYGON defined from some literal values, and a table full of POINTs, all that's left is to find out which POINTs in our table are inside our POLYGON. we can do this with the mysql function ST_Within(). here's an example:

SELECT  name,
        ST_Latitude(coords) AS latitude,
        ST_Longitude(coords) AS longitude
FROM    calgary
WHERE   ST_Within(
            coords,
            ST_GeomFromText('POLYGON( (  51.053913 -114.094391, 51.028008 -114.094391, 51.028008 -114.037743, 51.053913 -114.037743, 51.053913 -114.094391) )', 4326)
         )

we can see that ST_Within() takes two arguments: a POINT, and a POLYGON. if the POINT is 'within' the POLYGON, ST_Within() returns 1. if it isn't, we get a 0.

conclusion

once we have an understanding of how to create POINTs and POLYGONs and use ST_Distance_Sphere() and ST_Within() we can combine and extrapolate them to get more complex data, like "the closest daycare in a given school district" or "all the burrito busses on this side of the river" or, even, answer the question that has driven so many of the great minds in computer science: "where is a thai restaurant near me"?

🔎 this post originally appeared in the grant horwood technical blog

nginx: putting your site in ‘downtime’ for everyone except you

grant horwood — Fri, 06 Sep 2024 18:09:47 +0000

we've all been in that less-than-ideal situation of something going horribly awry in production and having to put the site into downtime while we fix it. that "scheduled maintenance"[sic.] page is important because it keeps users from seeing our glaring error, but it makes investigating or fixing production more difficult because, well, the site is in downtime.

in this post, we're going to go over a couple of ways we can use nginx to show different content to different users based on their ip address; configuring our web server so that everyone in the world gets our downtime message, except us. we get to see site as normal, allowing us to engage in the not-quite-best-practice of debugging in production.

two users (left) are served the well-crafted downtime page, while the developer (right) sees the real site.

the flyover

we're going to go over four nginx configurations. some of them are very similar, and are presented to show how we can combine and build on these strategies to tailor them to our needs. they configurations are:

using allow/deny to show unapproved ip addresses a 403 with an optional custom error page
using if to return a downtime html string to everyone except an approved ip address
using if to return a downtime html file
leveraging map and if to allow multiple ip addresses access to the site; downtime for everyone else.

tl;dr: if you're looking for the solution you can just copy-paste and modify, then the last one, "leveraging map and if", is probably what you want.

serving 403s with `allow` and `deny`

nginx has a lot of features built in to restrict and permit access. we can use it to throttle bandwidth or limit the number of connections per address to mitigate ddos attacks, but here we're going to look at the allow and deny directives.

in an nginx configuration we can, in a location block, set an arbitrary number of ip addresses to be either allowed or denied. requests from allowed addresses proceed to be handled as normal. denied addresses are served an HTTP 403. let's look at a complete, if somewhat terse, config:

server {

    server_name gbhorwood.test;
    root "/var/www/html/gbhorwood/test";

    index index.html index.htm index.php;

    charset utf-8;

    location / {
       # allow local ip
       allow 127.0.0.1;

       # allow some other ip
       allow 151.101.2.217;

       # 403 is default behaviour
       deny all;
    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    access_log off;
    error_log  off;
    sendfile off;

    client_max_body_size 100m;

    listen 80;
}

here we see that in our location / block we allow two addresses. all other addresses are handled by the deny all directive.

the important thing to note here is that order is important. nginx tests the ip address starting at the top and handles the first directive that matches. for instance, if we did this:

allow 127.0.0.1;
deny 127.0.0.1;

a visitor from localhost would be allowed because nginx stops testing the ip at the first entry for 127.0.0.1.

likewise, if we configured our location block with the reverse:

deny 127.0.0.1;
allow 127.0.0.1;

localhost would be blocked.

the allow and deny directives are not limited to single ip addresses; we can also use cidr blocks. if we wanted to allow only the ips on our local network, for instance, we could do so like this:

allow 192.168.1.1/24;
deny all;

serving a custom downtime file instead of just 403

of course, throwing an HTTP 403 to all our users probably isn't what we want. the point of a downtime page, after all, is to prevent people from seeing our terrible errors, not to serve them different ones.

we can fix this by setting a custom html error page for 403 and use that for our downtime message.

we'll start by creating our custom downtime page. we'll call it downtime.html and, for this example, we'll put it in the /tmp directory. you might (probably!) want to choose a different location. the contents of our downtime file are:

<h1>downtime</h1>
this site is currently down for "scheduled" maintenance.

once we have our downtime html file, we can configure nginx so that instead of showing the standard 403 error page, we show our downtime.html file instead. here's the full config:

server {

    server_name gbhorwood.test;
    root "/var/www/html/gbhorwood/test";

    index index.html index.htm index.php;

    charset utf-8;

    # new stuff
    error_page 403 /downtime.html;

    location = /downtime.html {
        root /tmp;
    }
    # end new stuff

    location / {
       # allow local ip
       allow 127.0.0.1;

       # allow some other ip
       allow 151.101.2.217;

       # 403 is default behaviour
       deny all;
    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    access_log off;
    error_log  off;
    sendfile off;

    client_max_body_size 100m;

    listen 80;
}

looking in the 'new stuff' block, we can see that we are using two directives to serve our custom downtime html file.

the first directive is a call to error_page. this call takes two arguments: the HTTP code we want to assign a custom html file to, and the name of the custom html file. very straightforward.

the next directive is a location block for our downtime html file. we do this so that we can set the root directory that nginx will look in for our downtime.html. this allows us to keep our page out of our main repository and ensures that it will always be served even if we accidentally nuke every file in web page's root (these things do happen).

of course, hijacking HTTP 403 for our own purposes is a bit of a kludge. there are more elegant ways for us to serve our downtime file. let's look at those.

standard http error handling looks askance at a developer thinking of hijacking 403 for a downtime page

serving a string of 'downtime' html using `if()`

if we want to show a string of html for our downtime message to every ip address except our own, we can do that using nginx's if statement.

let's look at a configuration that does that:

server {

    server_name gbhorwood.test;
    root "/var/www/html/gbhorwood/test";

    index index.html index.htm index.php;

    charset utf-8;

    location / {

        # show downtime html string if not whitelisted ip
        if ($remote_addr != "127.0.0.1") {

            add_header Content-Type text/html;
            return 200 '<html><body>Temporarily offline for scheduled maintenance and upgrades</body></html>';

        }


        try_files $uri $uri/ /index.php?$query_string;
    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    access_log off;
    error_log  off;
    sendfile off;

    client_max_body_size 100m;

    listen 80;
}

the interesting stuff here is the if statement inside the location directive. nginx keeps the ip address of the visitor in a variable called $remote_addr. we can test that against our allowed ip using a very familiar-looking if.

if the visitor's ip address is not allowed, we return a string of html. doing this is a two-step process: sending the Content-Type header with the add_header directive, and returning a string with return.

note that return takes two arguments. the first is the http status code, 200 is probably what we want here. the second argument is our string of html.

serving a file of 'downtime' html using `if()`

maybe returning just one string of html for our downtime page isn't enough. maybe we want a tonne of css and some animated gifs; y'know, "rich content".

we can do that by modifying the configuration above so that our if statement serves a file instead of just returning a string. it looks like this:

server {

    server_name gbhorwood.test;
    root "/var/www/html/gbhorwood/test";

    index index.html index.htm index.php;

    charset utf-8;

    location / {

        # show downtime html file if not whitelisted ip
        if ($remote_addr != "127.0.0.1") {
            rewrite ^ /static/down.html break;
        }

    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    access_log off;
    error_log  off;
    sendfile off;

    client_max_body_size 100m;

    listen 80;
}

again, here, we're testing the visitor's ip address in $remote_addr against our allowed ip in an if statement. the difference is that, instead of returning a header and a string, we are using nginx's rewrite directive to serve a file. since the rewrite directive ends with break, nginx jumps to the end of our location block, and our downtime content is served.

using `map` to handle multiple ip addresses more sanely

so far we've seen how to use if to test for one allowed ip address but, ideally, we would like to have a solution that allowed us to whitelist multiple ips. we could certainly attempt to do this with a regular expression, but regexes, as powerful as they are, are difficult to write and harder to read and if you suddenly have to add a seventh address on short notice it's probably not going to be a great experience.

we can avoid all that by using nginx's map function to, well, map ip addresses to boolean values. allowed ips get set to true, all other ip addresses are set to false, and then we use our if to check that boolean. it's a pretty slick solution:

everybody stand back, i know how to use nginx’s map function

map $remote_addr $allow_ip {
  default       0;
  127.0.0.1     1;
  151.101.2.217 1;
}

server {

    server_name gbhorwood.test;
    root "/var/www/html/gbhorwood/test";

    index index.html index.htm index.php;

    charset utf-8;

    location / {

        #  show downtime html file if not whitelisted ip
        if ($allow_ip = "0") {
            rewrite ^ /static/down.html break;
        }

    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    access_log off;
    error_log  off;
    sendfile off;

    client_max_body_size 100m;

    listen 80;
}

at the very top of this configuration, outside of the server block, we are calling map.

the map function has three arguments:

the value to test: this is $remote_addr, nginx's internal variable that holds the ip address of the visitor. this is the value that we will be testing in the list of test cases below.
the variable to set: the variable that will hold the result of our tests. in this example, a boolean.
the list of test cases: a list of tuples, essentially. if the left value matches the value to test, $remot_addr in our example, then the right value is set in the variable $allow_ip

in essence, what this call to map does is look at the user's ip address held in $remote_addr and then set the value of $allowed_ip. it's essentially a switch/case.

once map has run, we can use the $allowed_ip variable in our if statement to serve our downtime html file to everyone in the world, except us.

🔎 this post originally appeared in the grant horwood technical blog

php: concurrency with processes. pt. 2: interprocess communication with shmop

grant horwood — Mon, 19 Aug 2024 16:32:12 +0000

php isn't the sort of language where developers usually think about things like memory. we just sort of sling around variables and functions and let the internals figure out all that 'ram stuff' for us. let's change that.

in the first part of this series, we built a php script that was able to run a number of tasks concurrently by forking child processes. it worked pretty well, but there was a glaring, unaddressed problem: there was no way for those child processes to send data back to the parent process.

in this installment, we're going to solve that issue by using shmop, php's "shared memory operations".

shmop!

a very short and extremely optional explanation of shared memory
a basic flyover of shmop
an actual implementation
opening a memory block with shmop_open
using shmop_write to... write
reading memory with shmop_read
using shmop_delete to clean up
handling errors

a very short and extremely optional explanation of shared memory

when a new process starts, the operating system assigns it a chunk of memory to use. processes can't read or write to memory that isn't their own because, well, that would be a security nightmare. perfectly reasonable.

this creates an issue for us when dealing with the processes we created with pcntl_fork in part one of this series, however, because it means there's no easy way for the child processes to communicate with each other or their parent. child processes will get a copy of their parent's memory when they're created so all the variables assigned before the fork are accessible, but any changes to these variables will be limited to the child process. different memory and all that. if we want the child to be able to write to a variable that the parent process can read, we have a problem.

there are a number of solutions for this, all grouped under the general category of 'inter process communications' or ipc. the one we're going to use for our php script is 'shared memory'.

as the name implies, shared memory is a block of memory that an arbitrary number of processes can access. shared memory blocks are identified by a (hopefully) unique key. any process that knows what the key is can access that memory block. this makes it possible for a child processes to report back to their parent process; the child will write data to a shared memory block and, after it quits, the parent will read the shared data. it's a borderline elegant solution.

of course, there are a few footguns we will need to avoid when doing this: we will need to ensure that the key that identifies a shared memory block is unique, and we will need to enforce that shared memory communication only goes one way to avoid multiple processes all trying to write to the same block at the same time and causing a mess. we'll cover all this in the implementation.

a basic flyover of `shmop`

php has a rich and robust api for dealing with shared memory. the manual states "Shmop is an easy to use set of functions that allows PHP to read, write, create and delete Unix shared memory segments", and... it's not wrong.

let's look at the core steps for using shared memory:

create a unique key: all shared memory is identified by a key. any process that knows what the key of a shared memory block is can access it. traditionally, this key is created by generating data from the filesystem (ie. a value built from the inode of an existing file) because the filesystem is something all processes have in common. we will use ftok for this.

assign a memory block using the key: a process can use the key of a shared memory block to 'open' it using shmop_open. if the shared memory block does not exist, this creates it. the return value from the open function is pointer that can be used for reading and writing. if you've ever used fopen and fwrite before, this process should be familiar.

write data to the memory block: writing to shared memory has a very similar interface as fwrite. the pointer is used, and the string to write to memory is passed as an argument. the function to do this is called shmop_write.

read data from the memory block: reading data from shared memory is done with shmop_read, again using the pointer from shmop_open. the return value is a string.

delete the memory block using the key: deleting shared memory after it's no longer needed is important. this is done with shmop_delete.

the actual implementation

let's start with an example. the code below works and, if you are sufficiently un-curious or a tl;dr-type, you can just copy-paste-modify this, but for everyone else, we'll go over all the shmop steps and explain what they do and how they work.

// the file used by ftok. can be any file.
$shmop_file = "/usr/bin/php8.3";

for($i = 0; $i < 4; $i++) {
    // create the fork
    $pid = pcntl_fork();

    // an error has ocurred
    if($pid === -1) {
        echo "error".PHP_EOL;
    }

    // child process
    else if(!$pid) {

        // create a random 'word' for this child to write to shared memory 
        $random_word = join(array_map(fn($n) => range('a', 'z')[rand(0, 25)], range(1,5)));

        // write to shmop
        $shm_key = ftok($shmop_file, $i);
        $shm_id = shmop_open($shm_key, 'n', 0755, 1024);
        shmop_write($shm_id, $random_word, 0);

        print "child $i wrote '$random_word' to shmop".PHP_EOL;

        // terminate the child process
        exit(0);
    }
}

// wait for all child processes to finish
while(($pid = pcntl_waitpid(0, $status)) != -1) {
    echo "pid $pid finished".PHP_EOL;
}

// read all our shared memories
for($i = 0; $i < 4; $i++) {

    // recreate the shm key
    $shm_key = ftok($shmop_file, $i);

    // read from the shared memory
    $shm_id = shmop_open($shm_key, 'a', 0755, 1024);
    $shmop_contents = shmop_read($shm_id, 0, 1024);

    print "reading '$shmop_contents' from child $i".PHP_EOL;

    // delete the shared memory so the shm key can be reused in future runs of this script
    shmop_delete($shm_id);
}

creating a shared memory key with `ftok`

as we covered above, all shared memory blocks are identified by a unique integer key and before we can get down to the task of assigning memory we have create that key.

in all honesty, we can use any integer we want to, so long as it is unique, however the generally accepted, canonical way to do this is by using ftok to create an integer using an existing file in the filesystem as a reference point.

the rationale for doing this is pretty straightforward. processes don't know anything about each other, which makes it difficult for them to share a mutually agreed-upon value. one of the few things all processes on a system do have in common, though, is the filesystem. hence, ftok.

in addition to the path to an existing file, ftok also takes a project_id argument. this is, according to the docs, a 'one character string', what people in every other programming language would call a 'char'. the purpose of the project id is to prevent collisions when creating shared memory. if two projects by two separate vendors both decided to use /etc/passwd as their argument to ftok, chaos would ensue.

let's look at a fairly straightforward example:

$shm_key = ftok('/usr/bin/php8.3', 'j');
print "shm_key = $shm_key";

here we are passing the full path to a file that we know exists on the system and providing a one character project_id, 'j'. if we run this, the print statement will output something like:

shm_key = 855706266

that's a good integer to use for creating our shared memory!

if you run this code on your system, you will almost certainly get a different return value, even though you have used the same arguments. this is because, under the hood, ftok uses the inode of the file, and that is different from system to system.

if, for some reason, we pass to ftok a file that doesn't exist, we get a warning.

PHP Warning:  ftok(): ftok() failed - No such file or directory in <our script> on line <the line>
shm_key = -1

note that this is only a warning and ftok will charge ahead and give us a value of -1, which will result in problems down the road. be careful.

now, let's revist our call to ftok on line 20:

$shm_key = ftok($shmop_file, $i);

here we have passed ftok the path of the file we have set in $shm_key, in this case /usr/bin/php8.3, a file we know exists on the system.

for our project_id we are using $i, the index of the array we are looping over. we are doing this so that each of our child processes has it's own shared memory block to store its results. remember that if more than one process tries to write to shared memory, Bad Things happen. using the index here helps us avoid that.

opening a memory block with `shmop_open`

if you've ever done file access with tools like php's fopen and fwrite, then using shmop will be very familiar.

let's start with opening a shared memory block with shmop_open:

$shm_id = shmop_open($shm_key, 'n', 0755, 1024);

this function takes four arguments:

key: the unique key we created using ftok.
mode: the type of access we want. when opening a file with fopen we use modes like r for 'read' or w for write. shmop_open's mode is similar to that, but there are differences. we'll go over all those below.
permissions: the read/write/execute permissions of the shared memory block in octal notation. the interface for dealing with shared memory is strongly analogous to file access, and that includes permissions. if you're not confident with octal notation, there are file permissions calculators you can use. we're using 0755 in this example, but you may want to tighten that up.
size: the size of the memory block in bytes. in the example, we are assigning one megabyte, which is clearly overkill. however, note that if we overwrite our shared memory block, the value will be truncated. if we try to read more bytes from a memory block than its size, a fatal error occurs. if we are unsure of the exact size of the data we will be writing to memory, it is better to overestimate how big we need our memory block to be.

an important note here is that calling shmop_open will create a new memory block if one at that key doesn't already exist. this is similar to how fopen behaves, but with shmop_open this behaviour is dependent on the 'mode' argument we pass.

as shown in the example, shmop_open returns a pointer that can be used for access: reading or writing, depending on the mode used to open the memory block.

007 are bad permissions for a spy

a little bit more about that 'mode' argument

the mode argument that we pass to shmop_open determines how we can access our shared memory block. there are four options, all covered in the official documentation, but for the sake of simplicity, we'll only look at the two we need for our purposes.

n : the 'n' stands for 'new' and is used when we want to create a new shared memory block. there does exist a mode c for 'create', but we are choosing to use n here because this mode will fail if we try to open a memory block that already exists. that's a safety feature! in fact, the docs state that using n to create a new shared memory block is 'useful' for 'security purposes'. the pointer returned from shmop_open using mode n is writeable.
a : this is for 'access'; ie. reading. do not confuse this with fopen's a mode, which is for 'append'. memory blocks opened with mode a are read-only.

if we look at the example, we can see that when we open the shared memory block in the child process to write our data, we use the n mode. this creates the new memory block in a safe way and returns a pointer that we can write to.

using `shmop_write` to... write.

once our child process has created a new shared memory block and received a pointer to it, it can write whatever it wants there using shmop_write.

in our example, doing this looks like:

shmop_write($shm_id, $random_word, 0);

the shmop_write function takes three arguments:

the pointer: the pointer returned from shmop_open. note that shmop_open must be called with a mode that allows writing (n in our example), otherwise attempts to use shmop_write will fail.
the value to write: the string to write to the shared memory block.
the offset: the number of bytes in memory to offset the start point of the write by. using the offset can allow us to append to a value already in the shared memory block, but doing this means keeping track of bytes written and can become unmanageable pretty quickly. in our example, we use the offset 0; we start writing at the beginning of the memory block.

shmop_write returns, as an integer, the number of bytes written.

a short note about `shmop_close`

if you've done file access using fopen, you're probably (hopefully!) in the habit of calling fclose when you're done writing.

we do not do that with shmop.

there is a shmop_close function, but it has been deprecated since php 8.0 and does nothing (other than throw a deprecation warning, that is). the standard practice with shmop is to just leave the pointer 'open' after we're done writing. we'll delete it later.

reading from shared memory

once all the child processes have written their data to their respective shared memory blocks an exited, all that remains is for the parent process to read that data. the strategy for this is:

recreate the key of the shared memory block
use the key to open the shared memory in 'access only' mode
read the data into a variable

let's look again at the example we have for reading shared memory.

// read all our shared memories
for($i = 0; $i < 4; $i++) {

    // recreate the shm key
    $shm_key = ftok($shmop_file, $i);

    // read from the shared memory
    $shm_id = shmop_open($shm_key, 'a', 0755, 1024);
    $shmop_contents = shmop_read($shm_id, 0, 1024);

    print "reading '$shmop_contents' from child $i".PHP_EOL;

    // delete the shared memory so the shm key can be reused in future runs of this script
    shmop_delete($shm_id);
}

recreating the shmop key

when we made the key to create our shared memory blocks, we used ftok with two arguments: the path an existing file in the filesystem, and a 'project id'. for the project id, we used the index of the array we looped over to fork multiple children.

we can use the exact same strategy to recreate the keys for reading. as long as we input the same two arguments into ftok, we get the same value back.

opening the shared memory

we open the shared memory block for reading almost exactly the same way as we did above for writing. the only difference is the mode.

for reading, we use the a mode. this stands for 'access', and gives us a read-only pointer to our shared memory block.

reading from the shared memory block

once we have a pointer to our shared memory block, we can read from it using shmop_read.

shmop_read takes three arguments:

the pointer we got from shmop_open.
the offset, in bytes. since we are reading the entirety of the memory block, starting at the beginning, this is 0 in our example (and will probably be for most real-life uses, as well)
the number of bytes to read. in most cases, the smart thing here is to just read the entire size of the block, in our example 1024 bytes.

the return type is a string. if there are errors reading, we get a boolean false.

deleting shared memory blocks

once we are done reading our shared memory, we can delete it.

this is an important step. unlike variables in our script, the memory we assigned with shmop will persist after our program has exited, hogging resources. we do not want to litter our system with blocks of unused, reserved memory, piling up higher and higher with each successive run of our script!

freeing up shared memory blocks is done with shmop_delete. this function takes one argument: the pointer we created with shmop_open, and returns a boolean true on success.

note that shmop_delete destroys the memory block and frees up the space for other applications to use. we should only call it when we're completely done with using the memory.

handling errors

the example we've been going over doesn't really do any error handling. this is a decision borne out of a desire for brevity, not delusional optimism. in real applications we should certainly do some error testing!

we used a path to a file as an argument for ftok; we should test that it exists. shmop_write will throw a value error if our memory block is opened read-only or we overwrite its size. that should be handled. if there's a problem reading data, shmop_read will return false. test for that.

i am asking you to do some error handling

fixing 'already exists' errors with `shmop_open`

if we open a shared memory block and then the script terminates before we call shmop_delete, the memory block still exists. if we then try to open that memory block again with shmop_open using the n mode, we will get the error:

PHP Warning:  shmop_open(): Unable to attach or create shared memory segment "File exists" in /path/to/my/script on line <line number>

if our script is well-designed, this shouldn't happen. but, while developing and testing we may create these orphaned memory blocks. let's go over how to delete them.

the first step is to get the key of the memory block as a hex number. we do this by calling ftok as normal, and then converting the returned integer from base ten to base-16 like so:

$shm_key = ftok($shmop_file, $i);
$shm_key_hex = "0x".base_convert($shm_key, 10, 16);

we do this because linux comes with a number of 'interprocess communication' tools that we can use to manage shared memory blocks, and they all use hexadecimal numbers for their keys.

the first command line tool we'll use is ipcs. we're going to use this to confirm that the shared memory block we want to delete does, in fact, exist.

the ipcs command, when run without arguments, will output all interprocess communication channels, including all shared memory blocks. we'll narrow down that output by using grep with the hexadecimal key we created above. for instance, if our shared memory block's key in hexadecimal is 0x33010024, we could do this:

ipcs | grep "0x33010024"

if we get a line of output, the memory block exists. if nothing is returned, it does not.

once we've confirm that a shared memory block exists, we can remove it with ipcrm

ipcrm --shmem-key 0x33010024

knowing how to inspect and clean up (without resorting to a restart) shared memory allows us to develop and experiment without turning our ram into a ghost town of abandoned blocks.

wrapping up

achieving concurrency in php using fork and shared memory does take some effort and knowledge (and the official manual is scant help). but it does work and, if you've made it through this article and the first installment on pcntl_fork, you should have a good base from which to start.

🔎 this post originally appeared in the grant horwood technical blog

DEV Community: grant horwood

curl: getting performance data with curl

getting total time using --write-out

more timing data

speeds and sizes

response codes and headers

getting the response code

getting headers

getting all the headers as json

using a format configuration file

conclusion

php: a curl cheatsheet

flyover

an overview of a curl request

a basic GET request

reusing the curl pointer for multiple requests

building and sending a query string

getting HTTP code with curl_getinfo

connecting to a different port

error handling

treating http errors as curl errors

sending headers

getting headers

functions to extract response headers

basic auth

sending cookies

getting cookies

cookie jars

getting cookies into file

sending cookies from a file

sending and getting cookies using a file

sending POST requests

sending json via POST

sending form data via POST

sending PUT, PATCH and DELETE

downloading a file

uploading a file

using ftp

running ftp commands

uploading a file via ftp

downloading a file via ftp

nginx: useful basic auth

the flyover

creating accounts with htpasswd

installing apache2-utils to get htpasswd

creating a new .htpasswd file

adding accounts to htpasswd

adding htpasswd accounts non-interactively

deleting htpasswd accounts

choosing a better hashing algorithm

configuring nginx with basic auth

protecting the entire site

excluding locations from basic auth

requiring auth for only certain locations

adding further restrictions by username

restricting by user role

getting the username into our code

conclusion

php: manipulating file pointers with fseek and ftell

the sample file

a short overview of the file pointer

rewinding the file pointer

setting the file pointer with fseek

doing something useful with fseek

inspecting our pointer with ftell

a short note about file bounds

putting it all together: writing tail -f in php

wrapping up

linux: looking under the hood of neofetch

inspecting our os with uname

files with more useful os information

the os-release file

the lsb-release file

learning about memory with free

inspecting our hardware with lshw

getting info about hard disks with df

finding our screen resolution

deducing our desktop environment

getting the current shell

finding up time with uptime

getting total time using `--write-out`

a basic `GET` request

getting HTTP code with `curl_getinfo`

sending `POST` requests

sending json via `POST`

sending form data via `POST`

sending `PUT`, `PATCH` and `DELETE`

setting the file pointer with `fseek`

doing something useful with `fseek`

inspecting our pointer with `ftell`

putting it all together: writing `tail -f` in php

inspecting our os with `uname`

the `os-release` file

the `lsb-release` file

learning about memory with `free`

inspecting our hardware with `lshw`

getting info about hard disks with `df`

finding up time with `uptime`

the basics of `usort`

doing bookmarks with `m` and `'`

doing shell escapes with `!!`

doing better shell escapes with `!}`

doing autocomplete with `^p` and `^n`

a note on the dangers of `^p` and `^n`

making a `POINT` in mysql

a better way to make a `POINT`: well-known text

actually using `SRID` 4326

creating a table for our `POINT`s

finding `POINT`s 'within' a square

serving 403s with `allow` and `deny`

serving a string of 'downtime' html using `if()`

serving a file of 'downtime' html using `if()`

using `map` to handle multiple ip addresses more sanely

a basic flyover of `shmop`