DEV Community: Rob Porter

A cautionary note about use of Javascript's Set and Map objects

Rob Porter — Wed, 12 Jun 2019 14:50:51 +0000

A few weeks ago I was working on a really odd bug. An object was being assembled that contained some user data, and when we went to send that object up to the server as JSON, it was always incomplete -- it was missing data in one of the properties.

I tried all kinds of things to track down what was causing this: was an object merge acting unexpectedly? Was there an event somewhere that removed this data that I didn't know about?

It was actually much simpler than this. The problem, was JSON.stringify() and a Set contained within the object being stringified.

For an example, I'll create a user with some "lucky numbers", which I'm just making up as an example where the entries should be unique, so a Set would (in theory) be better than an Array.

const userData = {
    name: 'Haleema Greer',
    id: 7252,
    luckyNumbers: new Set([55, 45, 62, 21])
};

Looks good enough. Now, let's convert to JSON so we can send this to a server.

JSON.stringify(userData);

// > "{\"name\":\"Haleema Greer\",\"id\":7252,\"luckyNumbers\":{}}"

Wait, what just ate our luckyNumbers?

Apparently Set data just isn't in a format that can convert to JSON without some extra work.

One good solution is presented on Stack Overflow.

function Set_toJSON(key, value) {
  if (typeof value === 'object' && value instanceof Set) {
    return [...value];
  }
  return value;
}


JSON.stringify(userData, Set_toJSON);

// > "{\"name\":\"Haleema Greer\",\"id\":7252,\"luckyNumbers\":[55,45,62,21]}"

Much better!

Many guides will either state or imply Set can be used as a drop-in for Array, but they do not act the same and can have some unexpected consequences if you treat them the same.

Be aware these two types are different, and what works for one, may not work for the other.

Map also shares this issue as well.

const userDataMap = {
    name: 'Haleema Greer',
    id: 7252,
    luckyNumbers: new Map([
        [1, 55],
        [2, 45],
        [3, 62],
        [4, 21]
    ])
};

JSON.stringify(userDataMap);

// > "{\"name\":\"Haleema Greer\",\"id\":7252,\"luckyNumbers\":{}}"

Fixing this isn't as straight-forward, because Map's structure doesn't directly translate to JSON. It will be a matter of how you prefer to solve the problem.

For various solutions to converting Map to JSON, see this Stack Overflow discussion.

Using shell/bash commands in code? Use "long" options

Rob Porter — Mon, 10 Jun 2019 19:48:27 +0000

Many developers at one point or another need to use a bash or shell script inside code.

Maybe it's a command being issued in Docker, or an exec function in Node.js, or perhaps a shell command in a Jenkins pipeline.

One short and simple tip though I've picked up after having to do this a few times is, when documenting commands in code, always use the long options when available.

Long (--) and short (-) options are there for good reason -- one for clarity in documentation, the other for brevity when typing out commands in the terminal.

For example, here's a sample docker command below. Imagine you found this in code:

docker build --no-cache -t myimage:latest -f path/Dockerfile -q .

Let's say you were new to docker and encountered this in a Jenkinsfile or bash script.

The --no-cache seems quite clear about what it does. But what about the rest?

Wouldn't it have been better if this had been how it was written:

docker build --no-cache --tag myimage:latest --file path/Dockerfile --quiet .

Even better yet, what if it had been broken into a longer statement, using line breaks for each option?

docker build \
    --no-cache \
    --tag myimage:latest \
    --file path/Dockerfile \
    --quiet \
    .

Now the next person who reaches this code is less likely to have the urge to look up what -q, -f, and -t is for.

Not all terminal applications are consistent with what their short options mean, it can be easy to mix them up. It is often the case that things like -q will mean --quiet in one application, and --query in another.

Do this, and your code reviewers and inheritors may very well thank you for it.

Creating a ZFS Image of a MySQL Data Directory

Rob Porter — Fri, 07 Jun 2019 13:18:27 +0000

Notes: I wrote this documentation around 2013 as internal documentation at Weever Apps for setting up RAID-Z as a method of making very quick backups of MySQL. Since we no longer use this setup and the documentation will never be used again, I felt it might be a good idea to post it out there for those who might find some use in it. Using RAID-Z for MySQL was amazing, and never failed us, and allowed hourly snapshot-based backups with no downtime. We've since moved to AWS RDS instances though.

This will be part of multiple posts that combined will allow one to build a RAID-Z setup for MySQL. Or, at least it used to be able to! While this is geared towards Azure, I'm sure it would work anywhere.

As I add more parts over the next few days I'll add links here.

Article series

Installing ZFS on CentOS
Creating a RAID-Z Drive for MySQL with ZFS on CentOS
Setting up RAID-Z for use in a MySQL Master-Master Replicator Pattern
Creating a ZFS Image of a MySQL Data Directory

Instructions

Creating a ZFS image for backup is pretty easy.

First, a ZFS snapshot must exist. If one does not already, create one.

If this is for MySQL and the tables in MySQL are using MyISAM, make sure to either turn off the server briefly, or, lock the tables momentarily. In my experience these locks just delay writes until unlocked, and snapshotting is surprisingly quick in ZFS.

FLUSH TABLES WITH READ LOCK;

This example will assume you're grabbing the MySQL data directory, which would be stored in a tank called sqlstorage, with filesystem data. Adjust if your setup is differently named.

zfs snapshot sqlstorage/data@now

Now we can unlock the tables or start the MySQL server up again.

UNLOCK TABLES;

All we need is one command to send this snapshot to a file. Depending on the size of the file system, this could take a while.

zfs send sqlstorage/data@now >/tmp/sqlstorage-backup.zfs

Is something badly out of date? Did this work for you? Please post a comment to help others who might use this for guidance.

Building Expressive Monads in Javascript: Introduction

Rob Porter — Fri, 07 Jun 2019 00:47:03 +0000

The monad is a powerful design pattern that, when used correctly, can completely change how you think about handling values in Javascript (JS). This introductory tutorial is for any level of JS familiarity, even (and perhaps, especially) beginners.

For those already familiar with monads, this introduction only covers the essentials for using monads effectively, and will only touch upon the origins and more technical language when necessary to give context. No attempts will be made to explain category theory or deep concepts of functional programming.

What does "monad" mean?

For the purposes of this introduction, I'd like to refer to the dictionary definition, which pre-dates its use in mathematics and programming: a single unit.

This definition is akin to dyad and triad - meaning two, or three units respectively.

The term "monad" is used differently in mathematics, and in category theory. For programming, the monad was made popular by Haskell, and has been transposed into various languages, including JS. It is used as a way of containing values and controlling mutations.

I think though that the definition of "a single unit" is good to keep in mind. It helps me, at least.

What problems do monads solve?

Any time you've had to deal with trying to keep track of value mutation, any compatible type of monad would have helped wrangle how the values are changing.

If you've struggled with null and undefined causing havoc in functions that can't handle them, a Maybe monad would solve that problem.

For myself, it helps break down value-altering processes into small steps, allowing me to think about one piece at a time, not worrying about values mutating in unexpected ways. One can focus better on individual functions more easily. The results are so much more predictable, and the steps in the process more testable.

Monads can even handle processes that are asyncronous, but for the purposes of this introduction, we're only going to focus on syncronous cases.

How is it used in Javascript?

A monad is best thought of as a container of a value: much like how the container-like types Array and Object can hold a collection of values, a monad does the same.

Each monad you build is like building a new kind of container-like type. As Array has methods like forEach, and as Object has methods like keys, a monad will have standard methods, and methods you can add on a case-by-case basis.

If you've used Array and Object, you've already got some experience that will be useful with monads.

The most basic monad: `Identity`

We'll start our first example with the most basic possible monad, an Identity monad.

First though, a quick note on monad naming and style conventions...

Before we begin to build an Identity monad, I'd like to make clear the naming and styles you'll see in this introduction. When first learning monads, I was quickly hung up on the names and styles. It was enough to slow me down quite a bit.

You'll see monads named with capital letters, often with rather abstract names.

Do not get too concerned with the naming; if an abstract name is confusing you, remember that it is said that "naming things is one of the hardest things in computer science". Often these names point to a particular established design pattern that may have multiple possible awkward-sounding names.

The monad names will be capitalized -- this is an established convention, my assumption is this is to demonstrate they are a special type, much like a class name.

The same will go with monad method names, the most common ones have many established names. When one is introduced, I'll mention other names you may find being used for the same method. I will aim to focus on whichever method name I feel is most expressive for someone new to monads, though opinions will likely differ.

Second, a quick note on the term "identity"...

One more brief side-note: the monad name Identity is based on a term "identity" used for a function which simply returns the value given to it. The Identity monad will effectively do the same. This may seem like a near-useless function (it doesn't do anything!) but it is great for giving the most basic example, and there are some actual use cases in functional programming.

For instance, if you are required pass a function as an argument to potentially alter a value, but wanted to ensure that function didn't actually alter the value in certain circumstances, an identity is a great way to do that.

`Identity`: the code

const Identity = x => ({
    emit: () => x,
    chain: f => f(x),
    map: f => Identity(f(x))
});

// example use:
const one = Identity(1);

Yes, that is all. Write something like the above, and you have written a monad. These are the three required methods.

Many tutorials will hand you a library and just show how to use monads, but I feel a hands-on approach is going to actually make it easier to understand the concept.

That all stated, this Identity monad, at 5 lines, has a lot going on. Let's break that down.

`const Identity = x => ({ ... });`

The simplest part: we'll be using const as we don't want our defintion to ever change. You might know or have heard that const isn't perfect at locking down mutations: if you use const to define an Array or Object, and those can subsequenly mutate.

Thankfully, we're assigning a function expression to our const, which I like to refer to as a Constant Function Expression (CFE). I prefer these over the standard function defintion as they prevent anyone ever meddling with the function prototypes.

If you look up monad libraries in JS often you will find them based upon function or class, which makes them susceptible to meddling.

Our value we're going to pass into the Identity monad is x, and the beauty of a CFE is that the arguments passed into it cannot ever be altered or changed: it is absolutely immutable without having to use any special APIs.

This is why I love this pattern for monads: in just a few lines with no advanced syntaxes it creates an absolutely immutible value!

Once we pass 1 as a value in, nothing can ever change that 1 was the value passed in. If we had used a class and stored the value in an accessor, without some special API usage we'd be able to something like myIdentity.__value = 2 and just change the value.

And while I have not tested this hypothesis, I would think this is the JS monad pattern with the least-possible memory footprint.

Let's start looking at the core methods.

Method: `emit`

Alternative names: join, value, valueOf

Code

emit: () => x,

Example use

console.log(one.emit());
// > 1

This is the simplest method, that just returns the value contained within. Most commonly known as join, however I find that quite unexpressive when it comes to learning in Javascript. I like emit for explaining what it does as a verb: emit the value contained within.

A quick note of warning though, emit isn't something you should necessarily depend on in anything but debugging. In fact, you won't see me use it at all in the primary examples.

Method: `chain`

Alternative names: flatMap, bind

Code

chain: f => f(x),

Example use

console.log(one.chain(a => a + 1));
// > 2

The next simplest method is chain, which is intended to chain various monads together, but can operate as demonstrated above.

f => f(x) indicates a function f is taken, and value x is passed to said function. In this example, a => a + 1 takes the value, returns it plus one.

A more typical usage may be:

one.chain(a => SomeMonad(a + 1));

Where SomeMonad is a monad. In this chain, we transform Identity(1) into SomeMonad(2). When you are using chain, typically you're indicating that the function you are passing in either will itself return a monad (preventing recursive monad-inside-monad-inside-monad...) or that you intend for the result to be non-monadic.

Don't worry too much about why right now, as this I find is less commonly useful as compared to the next method, map. But it is important to understand first before we look at map.

Method: `map`

Alternative name: fmap ("functional map")

Code

map: f => Identity(f(x))

Example use

console.log(one.map(a => a + 1));
// > [not pretty: outputs monad defintion... at least until we implement .inspect() below]

map is the most important method. This is what makes monads so useful: we can take an established monad Identity(1) and through a function, generate Identity(2) without any mutation of our example constant one.

Put simply, it is the chain function with a built-in rewrapping of the resulting value into a new Identity, which itself can be subject to map, chain, and emit on and on for as many functions you'd like to apply to it.

This is the method I most use in a monad.

I sometimes like to think of it like a bank account ledger. All values must be accounted for: where they started (.of), and how they changed over time (map & chain methods). The initial value of a monad is like a new bank account being opened with an initial deposit, each map or chain is a transaction atop it. Nothing will ever change the value of the initial deposit, but we have methods to figure out how much remains in the account today.

One more method: `inspect`

You may have noticed doing a console output of the value after a map isn't going to look pretty. While not strictly required to make a monad work correctly, inspect can help inform us via the console what exactly is in the monad, and what type of monad it is.

const Identity = (x) => ({
    chain: f => f(x),
    emit: () => x,
    map: f => Identity(f(x)),
    inspect: () => `Identity(${x})`
});

const one = Identity(1);
const two = one.map(a => a + 1);

console.log(two.inspect());
// > Identity(2)

This method is important in debugging as a simple emit would not give you the type Identity; just the contained value 2. This is very important when working with multiple monad types.

Lastly, adding a constructor

In all the examples above I have been calling directly Identity. Typically, however, there is a constructor method. In JS, the convention is to add an of constructor. This looks like:

const one = Identity.of(1);

This helps in a couple ways. One, of() is a very strong hint we're dealing with a monad, as there's probably nowhere else you'll see it.

Secondly, it'll allow you to do type-checking behaviour, should your monad have restrictions on what is passed into it.

Typically I handle this using import/export, as such:

const Identity = x => ({
    emit: () => x,
    chain: f => f(x),
    map: f => IdentityOf(f(x)),
    inspect: () => `Identity(${x})`
});

// you might do type-checking here
const IdentityOf = x => Identity(x);

const exportIdentity = {
    of: IdentityOf
}

// or module.exports
export {
    exportIdentity as Identity
}

// or require()
import { Identity } from './Identity.js`;

Enough examples from me, though. Time for you to give it a try.

Try it out: Identity Monad Example REPL

Let's make another monad: `List`

List is the typical name of an Array-like monad.

We'll start with our Identity, but rename it.

const List = x => ({
    emit: () => x,
    chain: f => f(x),
    map: f => List.of(f(x)),
    inspect: () => `List(${x})`
});

For the purposes of this example, we'll assume of constructor has been added to this. When actually making one, we'd also type-check in the of constructor to ensure the passed value is an Array.

Adding more methods

As you can see with the addition of inspect, adding new methods is very easy. And if you write your own monad, why not add methods if you have a particular function you use with map or chain a lot?

In my experience, there are two kinds of methods you might add:

map-like: methods that return back the same type of Monad
chain-like: methods that either return a different kind of monad, or a non-monadic value; it may or may not "exit" the monad pattern, which I like to refer to as "unwrapping" the monad value

Method: `concat`

Concatenation is a fairly simple concept from Array: take one array, and add it onto the end of another. This seems like a method that would be very useful to have available.

concat: a => List.of(x.concat(a)),

// e.g.

const myNumbers = List.of([1, 3, 4, 7, 10]);

myNumbers.concat([12]).inspect();
// > List(1,3,4,7,10,12);

The function is simple: make a new List from using Array.concat on the contained value and the incoming value.

Note that this is map-like; it returns a new List.

Method: `head`

Let's say we wanted to just know what the first item in the List is. It's not an Array so using an index accessor like [0] isn't going to work.

head: () => x[0],

// e.g.

const myNumbers = List.of([1, 3, 4, 7, 10]);

myNumbers.head()
// > 1

This method is chain-like, as it returns a non-monadic value -- in this case, unwrapping part of the value. This one exits the monad pattern, so be aware when using these kinds of methods that continuing to chain map, emit, inspect, etc will not work.

const myNumbers = List.of([1, 3, 4, 7, 10]);

myNumbers.head().inspect();
// > ERROR! We unwrapped from the monad at `.head()`!

More methods

If you know Array well, you know it has an awful lot of methods. You can build a List with all kinds of things.

So here's a good excercise -- take this basic List monad and write some methods of your own!

Try it out: List Monad Example REPL

`Maybe`: the most powerful monad

It's possible you've heard of Maybe (also known as Option): the oddly named, but incredibly useful and powerful monad pattern.

The name "maybe" refers to the idea of "maybe there is a value... but maybe there is not".

In JS, having values that are undefined and null can cause havoc in the wrong place. What if, in every case where we currently have to place an awkward if (x === undefined || x === null) statement, we just could handle those cases right inside the value's container and never expose those unsightly and troublesome null values?

The code

Here comes a lot of code. Don't worry, we'll go through it all.

const Just = (x) => ({
  chain: f => f(x),
  emit: () => x,
  map: f => MaybeOf(f(x)),
  fork: (_, g) => g(x),
  isJust: true,
  isNothing: false,
  inspect: () => `Just(${x})`,
});

const Nothing = (x) => ({
  chain: _ => Nothing(),
  emit: () => Nothing(),
  map: _ => Nothing(),
  fork: (f, _) => f(),
  isJust: false,
  isNothing: true,
  inspect: () => `Nothing`,
});

const MaybeOf = x => x === null || x === undefined || x.isNothing ? Nothing() : Just(x);

const exportMaybe = {
  of: MaybeOf
};

export { 
    exportMaybe as Maybe
}

Use case

To give an example where this would be useful, let's have system that reads a temperature in Fahrenheit and gives it out in Celsius.

const fahrenheitToCelsius = a => (a - 32) * 0.5556;

const reading1 = 15;
const reading2 = null;

const temp1C = Maybe.of(reading1)
                    .map(fahrenheitToCelsius);

console.log(temp1C.inspect());
// > Just(-9.4444)

const temp2C = Maybe.of(reading2)
                    .map(fahrenheitToCelsius);

console.log(temp2C.inspect());
// > Nothing()

Right away we have a problem: for function fahrenheitToCelsius to work, we need a to be a number. Since reading2 is null (maybe a dead thermometer?), Javascript will cast null to 0, giving a constant false reading of -17.7792.

However, since we have encapsulated in a Maybe monad we only have two possibilities: a real number (Just, as in "just a value"), and no value at all (Nothing).

Explanation

How did this happen?

Our Maybe.of constructor did it:

const MaybeOf = x =>
    x === null ||
    x === undefined ||
    x.isNothing ? Nothing() : Just(x);

If the value being encapsulated in the monad was not undefined, null, or already a Nothing, then it is kept in Just. While the name Just may look new to you, its concept is almost identical with Identity! So you pretty much know already how Just works.

Nothing is a rather different monad than most: it doesn't take a value, and every method you use will result in Nothing(). After a Maybe has cast a value to Nothing there's no going back -- all attempts to map or chain just result in Nothing, so you need not worry about functions having unexpected behaviours since they never actually run.

Even emit here returns Nothing(), not null or undefined. This is because I lied slightly earlier, our emit method sometimes can't actually emit a value, especially if it has Nothing()!

But we need to handle what we do with Nothing, eventually...

Method: `fork`

Here enters the prized method of the Maybe monad given above: fork.

One quick side note: not all Maybe monad implementations will have a fork, but handle Nothing in other ways. For this tutorial however, we'll be using it, because we can!

fork is a method in two places here: in Just and Nothing

// Just
fork: (_, g) => g(x),

// Nothing
fork: (f, _) => f(x),

Right away you might see something odd. _ is a style choice often used in functional programming to indicate where we know there will be a value passed, but we plan not to use it. It is like the opposite of a placeholder.

Now let's use it for temperature display:

// assume a `display` function to display the temperature reading, and act like console.log

const fahrenheitToCelsius = a => (a - 32) * 0.5556;

const reading1 = 15;
const reading2 = null;

Maybe.of(reading1)
    .map(fahrenheitToCelsius)
    .fork(
        _ => display('ERR!'),
        t => display(`${t}°C`) // will read `-9.4452°C`
    );

Maybe.of(reading2)
    .map(fahrenheitToCelsius)
    .fork(
        _ => display('ERR!'), // will read `ERR!`
        t => display(`${t}°C`)
    );

Note in this use case we're not even assigning the results of the Maybe into a const value, as in this example we just need to display it.

But if we did need that string value to do something else...

const display = a => {
    console.log(a);
    return a;
};

const fahrenheitToCelsius = a => (a - 32) * 0.5556;

const reading1 = 15;

const temp3C = Maybe.of(reading1)
    .map(fahrenheitToCelsius)
    .fork(
        _ => display('ERR!'),
        t => display(`${t}°C`)
    );

console.log(temp3C)
// > "-9.4452°C"

This should be enough to get you started on using Maybe. It's a very different way of thinking about values than one is usually taught in JS, and will likely take some time to fully grasp.

What helps a lot with understanding the use of monads is practice! On your next small project, try adding in a Maybe module (suggestion below), or write your own. Given enough time, you might not be able to imagine writing code in JS without it!

For now, you can give Maybe a spin in the following REPL.

Try it out: Maybe Monad Example REPL

A note about `emit`

You may have noticed there's not really any examples of using emit (aka join) here. This is because "unwrapping" a monad is something you should actually completely avoid if possible, except when removing multiple layers of container monads.

It's great for debugging, but when you emit you'll need to realize you're leaving the safe world of monads, and into side-effects and mutability again.

There are many monad libraries that don't even have this method, as it isn't technically necessary -- after all, chain can achieve the same effect if it's fed a simple identity function.

The best thing you can do, if possible, rather than using a monad to compute a value to return, is build the monad and pass it all the functions it needs to do what it needs to do.

Don't worry if you can't figure out how to do that for every situation right now though. Just starting to use monads at all can be an exciting learning experience.

Quick review

What has helped me the most over the years is to think of monads as a container. That may help you, or to fall back to the dictionary defintion of single unit may also be of help.

Much like a bank ledger, monads keep their values immutible but allow methods to apply functions atop them to generate new monads, and thus new values.

But do be aware, doing a web search to find solutions on monad problems you might encounter may be a bit challenging. A lot of documentation out there is deeply saturated with technical language you might not be familiar with. A lot of it even I'm not familiar with. Hopefully that will change as this powerful pattern is adopted more widely.

Monad modules you can use right now

My own module, which is not very different from what has been demonstrated in this introduction is available as simple-maybe on npm.

What next?

Once you have grasped completely the concepts outlined in this introduction, other monads will mostly be just slight variations on the types of monad shown here.

In the near future I'll be posting about some other monad modules I've been building, and how they are used, and constructed.

I hope this article's approach has been accessible, even to those new to Javascript, and the code sufficiently expressive to not be a barrier to learning. Please feel free to leave suggestions for improvements or to share some other methods that have helped you better understand the use of monads.

Setting up RAID-Z for use in a MySQL Master-Master Replicator Pattern

Rob Porter — Thu, 06 Jun 2019 13:05:28 +0000

As I add more parts over the next few days I'll add links here.

Article series

Installing ZFS on CentOS
Creating a RAID-Z Drive for MySQL with ZFS on CentOS
Setting up RAID-Z for use in a MySQL Master-Master Replicator Pattern
Creating a ZFS Image of a MySQL Data Directory

Additional note

This documentation is a few years old and it uses the problematic "master/slave" terminology. Better terms have come up recently (my favourite is "master/minion"), but as this is legacy documentation I'm not sure if one can just drop in different terms without causing technical issues in the setup. Please leave a comment if you can confirm dropping the better terms in will work correctly!

Instructions

For MySQL to work on RAID-Z, we must be mounted on /var/lib/mysql. Also, this mount point needs to be owned by mysql user so we need to do the following:

chown mysql.mysql /var/lib/mysql

Then we'll want to create filesystem containers with this, which act like a directory but can be independently tracked by ZFS.

zfs create sqlstorage/data
zfs create sqlstorage/log

Two more chowns to do now..

chown mysql.mysql /var/lib/mysql/data
chown mysql.mysql /var/lib/mysql/log

You must then edit /etc/my.cnf to the following (change server-id depending on whether this is server 1 or 2 though, and auto_increment_offset):

[mysqld]
max_connections=1000
datadir=/var/lib/mysql/data
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
max_allowed_packet=1024M
log_bin = /var/lib/mysql/log/mysql-bin.log
server-id = 1
# this will prevent master-master auto_increment collisions, set offset to server-id
auto_increment_increment = 10
auto_increment_offset = 1
[mysqld_safe]
log-error=/var/lib/mysql/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Set the server-id to the correct ID.

You will then start up mysql for the first time, using

service mysqld start

Next we'll make sure this boots up every time the machine reboots:

/sbin/chkconfig mysqld --list
sudo /sbin/chkconfig mysqld on

Then follow the instructions provided upon this first boot-up to assign a root password, typically:

/usr/bin/mysqladmin -u root password 'new-password'

Then we'll be logging into mysql's terminal.

mysql -u root -p

Enter the password.

Then we'll create our replicator user, using this mysql command:

create user 'replicator'@'%' identified by 'password';

Generate and replace 'password' with a password.

Now give it permission to replicate:

grant replication slave on *.* to 'replicator'@'%';

Now run this and make note of what information it displays:

show master status;

You will need the 'position' and 'file name' provided, and use this on Server #2.

At this point, log into server #2, and repeat everything in this document up until this point.

On Server 2 only...

Time to set up the replicator.

slave stop; 
CHANGE MASTER TO MASTER_HOST = '1.1.1.1', MASTER_USER = 'replicator', MASTER_PASSWORD = 'password', MASTER_LOG_FILE = 'mysql-bin.000003', MASTER_LOG_POS = 106; 
slave start;

Where 1.1.1.1 is the internal IP address of server #1. Replace it with the correct Azure Internal IP. Also use the password and user created for server #1 'replicator', be sure not to mix it up with the replicator password for server #2. The master log file name and log position you should have noted before, replace it here with those values. Position is likely to be 106 though the filename can vary.

Now once again we'll do:

show master status;

And now on to server 1...

On Server 1 only...

Go back into mysql as root user. Enter this:

slave stop; 
CHANGE MASTER TO MASTER_HOST = '2.2.2.2', MASTER_USER = 'replicator', MASTER_PASSWORD = 'password', MASTER_LOG_FILE = 'mysql-bin.000004', MASTER_LOG_POS = 107; 
slave start;

Now just create the database you put in database_name_here in the my.cnf and replication should be working.

Is something badly out of date? Did this work for you? Please post a comment to help others who might use this for guidance.

How to do Security Updates only with Yum in CentOS

Rob Porter — Wed, 05 Jun 2019 13:57:20 +0000

This is a fairly basic "just the security updates, please!" guide for CentOS 6+.

This assumes you're root or have used sudo su.

First, you may need to ensure you have the yum security plugin:

rpm -qa | grep -i yum-plugin-security

If you don't, you can update it with the following.

yum install yum-plugin-security

Now let's check if there are updates:

yum --security check-update

If there are available updates, there will be results after this command.

To update, run the updater with the option --security to specify security updates only.

yum update --security

That's it!

Is something badly out of date? Did this work for you? Please post a comment to help others who might use this for guidance. I originally wrote these instructions around 2013.

Creating a RAID-Z Drive for MySQL with ZFS on CentOS

Rob Porter — Wed, 05 Jun 2019 13:38:49 +0000

As I add more parts over the next few days I'll add links here.

Article series

Installing ZFS on CentOS
Creating a RAID-Z Drive for MySQL with ZFS on CentOS
Setting up RAID-Z for use in a MySQL Master-Master Replicator Pattern
Creating a ZFS Image of a MySQL Data Directory

Instructions

First, let's make sure we have ZFS installed.

sudo su
lsmod | grep -i zfs

Next, in Azure, shut down the server then attach all the disks you'd like to use for RAID-Z; minimum should be a 2-core machine using the max 4 disks. All disks should be the same size, though this may not be a requirement for RAID-Z.

For this example, I'll be making 4 disks with 300 GB each. Give them names so it is clear that they are RAID-Z disks, by making sure raid-z is somewhere in the name.

Each disk should take about a minute to set up.

Start the machine up when the final disk is added.

Once booted back up, check the status of the disks (be sure to sudo su):

fdisk -l | grep GB

The following command will create a zpool called sqlstorage and mount it to /etc/mysql:

zpool create -m /var/lib/mysql sqlstorage raidz -f sdc sdd sde sdf

Note that the minimum for raidz is two disks. For double-parity raidz known as raidz2, you need a minimum of 3 disks. For triple, 4 disks are required. Use the highest level RAID-Z possible with the setup.

To check the zpool status:

zpool status

To make sure it's mounted:

mount | grep zfs

And to confirm the storage size:

df -h | grep sqlstorage

We also want to make sure the drive will mount on bootup:

echo "zfs mount sqlstorage" >> /etc/rc.local

Reboot, and you can then repeat the above checks to ensure it is working on boot.

Is something badly out of date? Did this work for you? Please post a comment to help others who might use this for guidance.

Installing ZFS on CentOS

Rob Porter — Tue, 04 Jun 2019 13:44:34 +0000

Article series

Installing ZFS on CentOS
Creating a RAID-Z Drive for MySQL with ZFS on CentOS
Setting up RAID-Z for use in a MySQL Master-Master Replicator Pattern
Creating a ZFS Image of a MySQL Data Directory

As I add more parts over the next few days I'll add links here.

Creating the VM

First, create a VM in Azure using the CentOS image. It needs to be a minimum of 2 cores to be set up with RAID-Z.

Make sure it is inside the Cloud Service of the application server VMs you wish to connect it to.

User name for the initial user should be azureadmin, but you should be sure to change the password and document it somewhere safe.

If you wish to make use of Load Balancing, be sure to create an Availability Set when asked, or select one that it will be used with if we're adding another load balanced instance.

After logging in via SSH...

First we need to be sudo.

sudo su

Next we're going to need to get ZFS installed, which will require some set up. First, we need to disable SELinux for this to work, at least until ZFS software gets updated to support it (or visa versa). First command disables it for now, second disables it for good.

echo 0 >/selinux/enforce
vim /etc/selinux/config

And change SELINUX= to permissive.

Next we need some new kernel libraries in yum so we need to edit the config file to remove kernel exceptions.

vim /etc/yum.conf

Find the line with exclude=kernel* and comment it out with a hash (#).

Then we're going to install some stuff.

yum groupinstall "Development Tools"
yum install kernel-devel zlib-devel libuuid-devel libblkid-devel libselinux-devel parted lsscsi nano mdadm bc

Next go to zfsonlinux.org and click on CentOS link, which should have commands like these two:

sudo yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release-1-3.el6.noarch.rpm
sudo yum install zfs

This will take a few minutes.

Now we'll load the module.

modprobe zfs

To make sure it's there and working, we'll check using this command.

lsmod | grep -i zfs

Let's set it up so it'll load on boot.

vim /etc/sysconfig/modules/zfs.modules

Paste into this file:

#!/bin/sh

if [ ! -c /dev/zfs ] ; then
        exec /sbin/modprobe zfs >/dev/null 2>&1
fi

Save the file (:wq) and then we'll do this:

chmod +x /etc/sysconfig/modules/zfs.modules

Let's now test that it works, and reboot.

reboot

Then make sure we get output from:

lsmod | grep -i zfs

If so, ZFS is installed!

Is something badly out of date? Did this work for you? Please post a comment to help others who might use this for guidance.

Extracting Test Results from CucumberJs in Jenkins Pipeline Syntax

Rob Porter — Tue, 04 Jun 2019 00:01:57 +0000

Jenkins can be a powerful system for Continuous Integration (CI) services, allowing automation of complex tasks, such as running an end-to-end (E2E) test suite within a Dockerized environment.

You don't automatically get useful data out of Jenkins jobs, however. Perhaps you want to take results from an E2E run and feed them into Slack, notify a monitor on a service like Datadog when there are test failures. You might also want to use the results to inform the successs/failure report of the Jenkins job itself.

Assumptions

This guide assumes you're using cucumber-js (v5.1.0 as of this writing), and cucumber-html-reporter (v5.0.0) in a Jenkins Pipeline job. If you are using another reporter, as long as you have a resulting file with data to extract, you may be able to use this guide with some adjustments.

This was designed around an E2E solution that runs tests entirely in Docker, and concludes by using docker cp to copy the resulting HTML report to the Jenkins job workspace.

Our Jenkinsfile will be using the Jenkins Scripted Pipeline Syntax, which is built with Groovy. You won't need to know much about Groovy as it's similar enough to JS for what we're doing. Groovy is a (sort of) superset of Java, and we're going to take advantage of that in this method.

Method

As our method for extraction, we're going to parse the HTML file that results, find the two lines that contain the pass and failure results, then use some light regex matching to hone in on those exact integers.

This may seem a bit awkward, but parsing the raw cucumber-report.json file would be far, far more work as we'd have to recalculate the results.

Implementation

At the top of our Jenkinsfile let's declare some variables we'll be using in this script.

#!groovy

int e2ePassed = 0
int e2eFailed = 0
int e2eTotal
int e2ePercent

pipeline {
( ... )

Somewhere down the line, when our main task of running tests is complete, we'll be generating an artifact from the Cucumber JS HTML report.

I've used comments below to document what each line is responsible for.

steps {

    /*
     run test, copy the report file out of docker or where ever it is...
    */

    archiveArtifacts artifacts: "cucumber_report.html", fingerprint: true

    // our Groovy script will be contained in this scope
    script {
        println ('Extracting passed & failed values')

        // get the present working directory (PWD)
        String PWD = pwd()

        // based on this, construct the path to the workspace
        String workspacePath = PWD.substring(PWD.lastIndexOf('/') + 1, PWD.length())

        // read our HTML report into a String
        String report = readFile ("${env.PWD}/workspace/${workspacePath}/cucumber_report.html")

        // split each line break into its own String
        String[] lines = report.split("\n")

        // use regex to find the "passed" line and "failed" line
        def foundPassedLine = lines.find{ line-> line =~ /\<span class="label label-success" title=scenarios\>/ }
        def foundFailedLine = lines.find{ line-> line =~ /\<span class="label label-danger" title=scenarios\>/ }

        // match for the numeric values
        def passedMatch = (foundPassedLine =~ /[0-9]+/)
        def failedMatch = (foundFailedLine =~ /[0-9]+/)

        // cast to Integer so we can work with the number values
        e2ePassed = passedMatch[0] as Integer
        e2eFailed = failedMatch[0] as Integer
        e2eTotal = e2eFailed + e2ePassed
        e2ePercent = e2ePassed / e2eTotal * 100

        if (e2eFailed > 0) {
            // do something extra if there are failures
        }

        // print out our results to the console
        println ("Passed: ${e2ePassed}")
        println ("Failed: ${e2eFailed}")
        println ("Percent passes: ${e2ePercent}%")
    }

    // we can also access these values in an `sh` declaration (or other functions) outside of `script` now
    sh "echo \"Passed: ${e2ePassed}\""
    sh "echo \"Failed: ${e2eFailed}\""
    sh "echo \"Percent passes: ${e2ePercent}%\""

    // you may opt to use the vars in plugins like `slackSend`, etc
}

Note: The security settings for up-to-date Jenkins setups will trigger an error upon using readFile the first time. When this happens, go to Manage Jenkins and In-process Script Approval. You'll see a prompt to allow the readFile for use in the specific Jenkinsfile in all future job runs. You must be an admin to approve this.

Once you have this data extracted from the reporter, you could use it notify external systems when there are failures, or to keep a running track of test pass/failure metrics.

Originally posted at https://robporter.ca/jenkins-cucumber/

DEV Community: Rob Porter

A cautionary note about use of Javascript's Set and Map objects

Using shell/bash commands in code? Use "long" options

Creating a ZFS Image of a MySQL Data Directory

Article series

Instructions

Building Expressive Monads in Javascript: Introduction

What does "monad" mean?

What problems do monads solve?

How is it used in Javascript?

The most basic monad: Identity

First though, a quick note on monad naming and style conventions...

Second, a quick note on the term "identity"...

Identity: the code

const Identity = x => ({ ... });

Method: emit

Code

Example use

Method: chain

Code

Example use

Method: map

Code

Example use

One more method: inspect

Lastly, adding a constructor

Let's make another monad: List

Adding more methods

Method: concat

Method: head

More methods

Maybe: the most powerful monad

The code

Use case

Explanation

Method: fork

A note about emit

Quick review

Monad modules you can use right now

What next?

Setting up RAID-Z for use in a MySQL Master-Master Replicator Pattern

Article series

Additional note

Instructions

On Server 2 only...

On Server 1 only...

How to do Security Updates only with Yum in CentOS

Creating a RAID-Z Drive for MySQL with ZFS on CentOS

Article series

Instructions

Installing ZFS on CentOS

Article series

Creating the VM

After logging in via SSH...

Extracting Test Results from CucumberJs in Jenkins Pipeline Syntax

Assumptions

Method

Implementation

The most basic monad: `Identity`

`Identity`: the code

`const Identity = x => ({ ... });`

Method: `emit`

Method: `chain`

Method: `map`

One more method: `inspect`

Let's make another monad: `List`

Method: `concat`

Method: `head`

`Maybe`: the most powerful monad

Method: `fork`

A note about `emit`