DEV Community: Matthew Lucas

Habot - Your Friendly Habit Coach (Azure Hackathon)

Matthew Lucas — Tue, 08 Mar 2022 21:29:08 +0000

Overview of My Submission

When this hackathon opened, I was ploughing through the excellent book Atomic Habits by James Clear.

Good habits are key for us to build happy lives and, conversely, bad habits can ruin them. But good habits are hard to build and bad habits hard to break - seems like we need a helping hand.

The purpose of this hackathon entry was to build a helpful chatbot to coach you through building or breaking habits. Habot (your friendly habit-bot) will keep you on track, monitor your progress and give you helpful advice when you begin to falter - all a little "tongue in cheek" of course.

https://habot.azurewebsites.net/

Let's take a look at some of the key features of Habot:

1. Single Signon with Outlook Credentials

Using your Outlook OAuth credentials you can easily sign up with Habot, and it'll greet you by name and with a smile.

2. Creating your first habit

The first thing you'll be asked to do once you log in is to create a habit.

Habot is intelligent, and can infer one of a number of habits from what you say. For example, it can understand that "I want to begin going to the gym" is the same as "I want to start exercising more".

Once your habit is confirmed, you'll get a cheery dance until next time.

3. Check-in and Progress

Each day you'll need to check in with Habot (I had hoped to use push notifications here, but alas no spare time). When you check in, Habot will keep track of your metrics.

Habot will also give you charts to more clearly demonstrate your progress. It also has emotions, so will feel a little sad if you aren't doing as well as hoped.

4. Getting Help

If you're having a hard time keeping up with your habits, Habot has some advice for you depending on the problem that you're facing.

Habot will also show you inspirational videos to help you get back on track.

Submission Category:

Wacky Wildcards

Link to Code on GitHub

lucas-matt / habot

habot

View on GitHub

Additional Resources / Info

The project has a number of parts to it, and is built with the following technologies:

The Azure Bot Framework, to drive the chatbot.

A Vue.js application for the UI, with the web-chat control built in.

A Rust with Rocket backend to collect metric information and build charts.

Azure CosmosDB (Mongo flavour) to store metric information.

SignPost - A Geo-Social Network (Atlas Hackathon)

Matthew Lucas — Thu, 13 Jan 2022 21:01:38 +0000

For those of us who have had the good sense and fortune (or possibly misfortune?) to play one of the Dark Souls series, we're intimately familiar with the mystical orange soapstone. This item allows players to drop messages anywhere within their world that will then appear to other players in their own worlds. These are used to hint, warn, amuse or (more often than not troll) other players as they journey through the treacherous land of Lordran.

Overview of My Submission

I took inspiration from the idea of the orange soapstone - specifically being able to drop messages at any location for others to read - and came up with the idea of 'SignPost', a geo-social network. This is a platform that allows you to drop messages (signposts) anywhere you like and to read the messages other people have left, where you're located. This could be great for directions, sports (cross-country hiking), games, etc - but is mostly just a fun idea for a hackathon.

SignPost is an Android app supported exclusively by the MongoDB Atlas platform, from user-authentication and login through to geo-querying and search. The app has four core features:

Signup and authentication using your Google credentials (via MongoDB Realm).
Browsing a map marked with "signposts" - messages dropped by other users near your location.
Dropping a signpost yourself.
Searching for signposts across the world

1. A map filled with signposts

Once logged in (via your Google credentials) you're presented with a map zoomed in on your current location. The app fetches any messages in your local area and plots them on the map for you to interact with. These appear as orange markers, and will display the message when touched.

2. Dropping your own signpost on the map

If you have something to say you can easily post your own message on the map. Just tap the exact point at which you want to post it and type away.

3. Searching for signposts across the world

You can search for messages across the whole world, thanks to the Atlas search integration. Simply click the search icon (top right) pick the message from your search results and your map will be immediately transported around the world.

How does it all hang together?

The SignPost app is backed completely by the MongoDB Atlas platform, and uses a number of its features:

Realm / Google Authentication to seamlessly add google user signon and tracking to the application.
MongoDB Atlas with Geospatial Indexes to allow the plotting of messages with their latitude/longitude co-ordinates, and to query records within the locality of a point.
Atlas Search to make it trivial to map the signpost messages into a free text search.
Realm Functions to provide a serverless set of backend APIs for the three key functions of the app - reading, writing and searching for messages.

In summary, MongoDB Atlas was able to provide a complete back-end stack (with the exception of Google Maps) to develop a fully functional Android application, even though I'd never written a line of Android code before 😲

Submission Category:

Choose Your Own Adventure

Link to Code

lucas-matt / signpost

Additional Resources / Info

Open Tracing on Kubernetes —Get Your Traces for Free

Matthew Lucas — Thu, 30 Dec 2021 14:23:32 +0000

Ever since I first mucked around with Istio — a smart service mesh that runs on top of k8s — I was fascinated by its auto-injection feature. Flick the on-switch and Istio gets sprinkled across your existing deployment, giving you fantastic service-mesh powers, without modifying, repackaging or redeploying any of your existing apps in any way.

Demystifying the process a bit, Istio uses a feature of Kubernetes named “Mutating Admission Webhooks”. These are a lot simpler than they may sound. On deployment of a resource, k8s will send any active webhooks a YAML representation of the action being performed. These services can edit the deployment however they require — adding volumes, tweaking environment variables, checking parameters, etc.

Why care? — Because open tracing, that’s why!

What’s OpenTracing?

OpenTracing is an initiative to enable reusable, open source, vendor neutral instrumentation for distributed tracing. In fact OpenTracing itself has become part of a larger project, OpenTelemetry, but that’s for another time.

What’s a trace? A single trace represents the vapour trail left behind by a request after it has hopped around the services in your application. It gives you details about any HTTP requests, database calls, or other spans you may have set up. It can be a useful tool for understanding the shape of your traffic and to spot / debug any bottlenecks across a suite of microservices.

How does this concern webhooks? Well, there’s some small amount of development required to get tracing hooked up in the first place. As the value is somewhat limited unless you instrument all apps in your platform, if you have 10+ microservices this effort can add up quite quickly. Wouldn’t it be great if you could try before you buy by just flicking that switch as you can with Istio.

The rest of the article explains just this feature — for Java apps at least — and how it all hangs together.

Special agents

Amongst the treasures found within the opentracing-contrib project is a Java special agent. By plugging this into our app via the -javaagent JVM flag we can fully enable tracing across any commonly used 3rd party libraries without changing any code or rebuilding the project. The list of instrumented libraries is pretty comprehensive — Jersey, Cassandra drivers, MongoDB drivers to name but a few.

What we’ll do here is automatically plug-in this agent at deployment time using a combination of webhook, init container and tweaking of environment variables to insert the agent.

Auto tracing webhook

The full source is available here, but at it’s simplest the webhook will:

Check the incoming deployment descriptor for the correct tags (autotrace: enabled). If present it’ll apply steps 2 onward, otherwise it’ll leave everything untouched.
Add a volume mount into which the opentracing special agent jar will be dropped.
Add an init container to copy the jar into the shared mount before the application boots.
Tweak the JAVA_TOOL_OPTIONS environment variable to add the javaagent line for the agent.

Deploying it all

First make sure you have Jaeger running in your default namespace — as simple as:

kubectl apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml

[edit]
If you’re using Kubernetes 1.16+ the api changed enough to break the jaeger deployment descriptor above. If you hit an issue try this fixed version instead:

kubectl apply -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/cc2e03335d8fe88eeef46648cff39151215ca97f/all-in-one/jaeger-all-in-one-template.yml

Next, the webhook itself. The source for which is available here: webhook.yml.

kubectl apply -f https://raw.githubusercontent.com/lucas-matt/auto-tracing-webhook/master/webhook.yml

Lastly make sure you label the target namespace so that the webhook gets activated for any deployment within.

kubectl label namespace default autotrace=enabled

As far as this stage goes, we’re done.

Trying it out

Let’s give the solution a quick run through. We’ll need to demonstrate a request across multiple services to show the tracing working well, end-to-end.

kubectl apply -n <your-namespace> -f https://raw.githubusercontent.com/lucas-matt/pass-the-buck/master/deployment.yml

Deployment.yml creates a chain of services A, B and C. A calls B, B calls C and C calls upstream to the world clock API. Each of the services is, by default, a completely trace unaware Spring Boot application.

Each of these services is tagged with the _autotrace: enabled _ label so that our webhook knows to inject instrumentation into the application at deploy time.

spec:
  replicas: 1
  selector:
    matchLabels:
      app: service-a
  template:
    metadata:
      name: service-a
      labels:
        app: service-a
        autotrace: enabled

After port forwarding service-a, all that remains is to make that request.

So far so good — we got the time — but the main question, was it traced? Time to check Jaeger:

Success! We can see how the request moves along each service and out onto the open web.

Conclusion

Not a solution I’d use in production in its current state — for one, the untuned startup of an app is degraded whilst the agent scans the classpath for the complete set of libraries into which it can slot itself — but an interesting experiment nonetheless.

If you’re interested in checking out OpenTracing, or even in how to create your own Kubernetes webhook, please take a look at the source repositories for a few examples:

Why Small Teams Outperform the Competition

Matthew Lucas — Wed, 22 Dec 2021 15:49:10 +0000

Two pizza teams, scrum teams, tiger teams; there are many names for these close- knit bands of high-achievers who speed past the competition to fame and fortune. And there is one factor common to any well-performing team that greatly contributes to that success - size.

Surely bigger means better? More resource means more hands to do the heavy lifting and more feet to stomp out those blazing fires. Of course, any modern worker with even a little experience understands that this is clearly not the case; but why is that?

First lets look at what principles make up a winning team.

What makes up a winning team?

According to Professors Hirotaka Takeuchi and Ikujiro Nonaku, in their paper The New New Product Development Game, great teams embody three core principles.

Transcendence: a sense of purpose, or self-realised goal that surpasses the ordinary aims of an average team.

Autonomy: having full control over their destiny by being empowered to make key decisions and stick to them.

Cross-Functional: having all skills necessary to see a project through from start to finish without relying on a third party.

What has this got to do with size … ?

These principles can only work if the team is able to hold a shared vision and shared understanding of the state of the world.

Every member of a team needs to know exactly what they’re fighting for and why that is important. Every team member needs to understand the key priorities facing them at any moment. And finally, there has to be a good amount of trust within the team so that discussion can be fair, open and free flowing.

We are, of course, social animals and our ability to achieve great things is deeply rooted in our sociology and biology. But, as we shall see, there are limits to how far this can be pushed.

Brain power

You’ve likely heard the assertion that an average adult can hold only 7 ± 2 chunks of information in their working memory at any one time; a chunk being any atomic idea within our heads. The amount of information contained in a chunk can vary depending on our expertise in a topic —for example, a phrase in your native language many only form one chunk, whilst that same phrase in an unfamiliar language could take up many as you struggle to juggle the grammar in your mind.

This limited ability to pay attention to many things at once is part of our human nature, and it applies not just to generalised thinking but to all aspects of our lives, including communication.

As we increase a team’s size, the number of lines of communication doesn’t increase linearly, as you may expect, but by the addition of a new channel between the new member and every other member of the team.

By plotting the formula that describes this — n(n-1)/2 — you can see that things can get quite of hand very quickly:

Each line of communication is another mental burden that your team members have to struggle with whilst trying to hold everyone else’s trajectory in mind. At some point, between the 5 and 9 mark, their brains just give up and teams tend to splinter into more cognitively manageable subgroups.

It’s biological — large groups just get less done!

Social proof

Think back to one of those situations where your team has to make serious commitment by way of a delivery date or some similar pressure? I’m sure there has been more than one occasion where this has resulted in a lot of poker faced, nervous shuffling as no-one wants to be the first to step up and make a suggestion for fear of putting their reputation at risk, or simply making of fool of themselves. Once someone finally does crack and gives it a go I bet that most, if not all of the rest of the team has followed suit pretty closely with the same or a very similar opinion.

This is an example of social proof.

Let’s take a more commonly occurring, but definitely insidious, example … the abomination of canned laughter.

Everyone hates canned laughter. It’s been polluting our comedy shows and TV screens (or rather speakers) for decades but continues to persist, and there is a good reason for this. It has been scientifically proven that we are easily nudged into a chuckle by hearing a number of other people rolling around the floor in hysterics, despite the fact we know they are phantom and have been cleverly added after the fact by some sound engineer.

All of this comes down to a simple heuristic or biological short-cut — that given some uncertainty (lack of concentration, understanding, or similar), we will will make fewer mistakes by acting in accord with the existing social evidence. In other words, usually when a lot of people do something and we’re unsure, it is probably the right thing to do.

Pluralistic ignorance

Closely related to social proof, is the concept of pluralistic ignorance.

Let’s take the example discussed in Cialdini’s masterpiece. An actor pretending to have a heart attack, calls out for help in a public place. You’d think that any passers-by would come rushing in to save his life, but you’d be wrong. Entire groups of people fail to come to his aid, each certain that someone else will surely have helped and the ambulance must already be on its way. With several potential helpers around, the personal responsibility of each individual is reduced. Again, the uncertainty of the situation causes others to “just do” what everyone is doing.

We can translate this into a less extreme circumstance in the case of some important task that needs to be completed. Everyone agrees that the build system is a nightmare and needs to be fixed, or that waiting an hour for the tests to run is just an utter joke, but does anyone actually do anything — probably not.

So what can we learn from this?

Clearly a team is a very delicate ecosystem and individually we’re plagued by our cognitive biases too, but there are many strategies that we can employ to try and make the best of it all. Here are a few suggestions.

Keep teams small (ideally 5–7 people) to avoid paying too high a mental cost just in communication.
Avoid anchoring or social bias problems by using tools such as planning poker or by avoiding asking for commitments in an open and pressured forum.
Assign actions, and agree on an owner for any kind of important task that may otherwise fall victim to pluralistic ignorance.

Survivorship Bias and Negotiating Tech Hype

Matthew Lucas — Sat, 18 Dec 2021 07:17:40 +0000

Bombers with bullet holes

During the dark days of World War II the American military presented their Statistical Research Group with a problem. They wanted to add additional armour to their bombers, but clearly couldn’t put the armour everywhere because of the additional weight it would add to the planes. The group was tasked with working out how much armour to allocate to the various regions of the aircraft to maximise defence whilst minimising any effect on fuel consumption and agility.

Engineers inspected a number of bombers that had seen some action. These planes had bullet holes that were distributed mainly across the wings and body. Comparatively the engines and cockpit had much less damage. This had lead the commanders to make the obvious, but foolish, conclusion that they should enhance armour on areas that had been hit most frequently, namely the fuselage and wings.

One of the many geniuses of the group, Abraham Wald, realised that they were looking at the problem from completely the wrong angle. It’s not that planes weren’t being hit as frequently on the engines and cockpit, but rather those that had been never returned to tell the tale! These were the parts of the aeroplane that needed enhancement, not the areas that could take a battering and still survive.

Survivorship bias

Countless articles, books and documentaries have been produced about successful people and how to capture the principles of their success to improve your own fortune.

Consider Steve Jobs — frequently heralded as a one of the greatest geniuses of our time — how do we emulate his success? Clearly dropping out of college, spending time at meditation retreats and starting a business from your parent’s garage is the way to go. But what about the hundreds of thousands of budding Apple founders for whom this strategy never quite worked out?

Books aren’t usually written about failed enterprises, just the rare, billion-dollar, success stories.

Choosing tech thoughtfully

As well as being responsible for the latest diet fads and the exaggerated performance of mutual funds, we can see this bias lurking in certain corners of the software development world.

How often do you see a wave of enthusiasm for the next high throughput NoSQL system or a push for complicated elastic scaling technology? Companies such as Twitter and Netflix present their wild successes but we don’t usually see qualifications on the size and scale of the teams implementing these solutions.

It’s worth keeping in mind the potential for a mass of silent teams. Struggling under the weight of overpowered, over-engineered, “web-scale” technologies inspired by the industry front-runners. Most of us mere mortals just don’t have the resources, skills, or (most importantly) even the need for such high class deployments. Most of the time it’s just better to keep things simple and known.

Similarly, businesses push on with “AI” for fear of missing out, but without any real understanding of what they really need. Solutions are commissioned that aspire to the heights of Facebook and Google whilst in reality they fumble for a real business-value providing use-case.

Don’t get me wrong, I’m 100% all for learning new paradigms, languages and frameworks. This is just a reminder, as much to myself as anyone else, to take a moment to think past the biases that may lead us to make some regrettable, albeit well-intentioned and over-excited, choices.

Tell Don’t Ask

Matthew Lucas — Sun, 12 Dec 2021 20:53:05 +0000

Quality software, and most importantly software that maintains that quality, is written in a way that protects itself from future corruption. It makes it easy to build upon without introducing new bugs or breaking existing features.

There are many strategies to help us achieve this quality — unit testing for one — but there is an aspect of object-oriented programming that is designed specifically to tackle this problem. The key is to keep data and the logic that uses that data close together.

The aim is to produce “shy” code. Code that doesn’t expose too much to anyone who interacts with it, and only interacts through well-defined commands and queries. This is the very definition of encapsulation. It gives us the protection we need to build in robustness and it is applicable at all levels — object, module and service.

Unfortunately, we all slip up from time to time and accidentally reveal more than we should’ve. There is, however, a principle that can help us avoid this pitfall.

Tell Don’t Ask

When it comes to combining logic with the data that drives it there are two ways you can approach this, and they closely align with procedural and object-oriented schools of thought.

Asking

Procedural code generally asks for some data and then acts on this data. The behavior and data are kept separate.

Take the following example of a bank account. The account has a fixed overdraft limit and a set of transactions (tuples of payee x amount).

class BankAccount:

    def __init__(self, overdraft):
        self.overdraft = overdraft
        self.transactions = []

Say we want to calculate and print the balance of the bank account — a function print_balance. In the procedural style, the function has to pull everything it needs from the bank account and handle all calculations itself.

def print_balance(account):
    amount = sum([amt for _, amt in account.transactions])
    print("Current Balance: %s" % amount)
    if amount < -account.overdraft:
        print("(Overdrawn by %s)" % (amount + account.overdraft))

Whilst this gives a lot of power to the caller it also hands over much of the responsibility. Due to the loss of encapsulation, we’re now exposed to duplication, coupling and inevitably bugs.

This is an anti-pattern known as the Anemic Domain Model.

Telling / Querying

Rather than asking the data structure for all of its data, if we tell (or query) the object for that information, the responsibilities become clearer and the data model becomes richer.

Instead of asking for the raw data, we ask for facts for which the object is responsible — the account balance and available credit.

Rather than adding any transactions directly, we inform the account about them using an add method where it can make any protective assertions based on its state.

class BankAccount:

    def __init__(self, overdraft):
        self._overdraft = overdraft
        self._transactions = []

    def add(self, payee, amount):
        self._transactions.append((payee, amount))

    def balance(self):
        return sum([amt for _, amt in self._transactions])

    def is_overdrawn(self):
        return self.balance() < -self._overdraft

    def available_credit(self):
        return self.balance() + self._overdraft

def print_balance(account):
    print("Current Balance: %s" % account.balance())
    if account.is_overdrawn():
        print("(Overdrawn by %s)" % account.available_credit())

Beware the paperboy

In “The Art of Enbugging” there is a great analogy for this principle.

A paperboy comes to the door to collect his payment for the week. He snatches the wallet from your back pocket, takes a wad of cash, and puts it back. This doesn’t sound quite right does it, but it’s how a lot of software is written. Some central logic bullies several “data” objects to do its bidding.

More realistically, the paperboy would tell you to pay and after some thought (validation) you should hand over the cash. This is how object-oriented code is supposed to work, with you looking after your affairs, the paperboy his, and communication taking place through messages passed back and forth.

Final words

‘Tell don’t ask’ is a good rule of thumb to help you structure your code well, and it doesn’t just apply to low-level code. The core message — behavior/data co-location — is equally as important for broader architecture too.

It is, however, just a rule of thumb and as with many good design principles should be considered within the context of everything else. You will find that in certain circumstances the wider architecture dictates a more important structure (for example layering), and in that case, data co-location is just one factor of many to bear in mind.

“Finding Stuff” — Building a Great Search Experience

Matthew Lucas — Thu, 02 Dec 2021 18:55:06 +0000

Search is everywhere. How would you survive a day without it!? It’s on every website and part of every product. Whether you’re purchasing underwear, trawling Netflix for a Friday night movie, or even leading a cutting-edge research team, it can be a modern miracle to summon what you need at the cost of a few key presses.

Despite being ubiquitous, why is search so often terrible? It’s frequently seen as second class, a bonus. It is plugged in and expected to work out of the box. It takes time, thought, and effort to craft a well-balanced search experience.

Before We Begin

Talking about concepts without an actual example to stare at can often become a challenge. For this post, we’ll be using a sample of around 10,000 books sourced from the Goodreads website. The data set itself, and scripts to load it into Elasticsearch, can be found here:

https://github.com/lucas-matt/search_playground

Principle 1. Search Engines Are Dumb, You Are Not

Often search engines are seen as just another database. They do look and act similar in many ways, but once you scratch the surface, you reveal a wealth of fancy algorithms and highly optimized data structures that differ quite drastically from a standard data store.

There are a number of parts to any search tool that we need to understand so that we can build upon them:

analysis (zapping a chunk of text into fragments)
the inverted index (a lightning-fast mapping of tokens to the documents in which they appear)
the ranking algorithm (identification and ordering of information based upon its relevance to the query)

Analysis

As we push documents into our search engine, they get sliced and diced into lots of little chunks. These chunks, or tokens (as we should call them from now on), are the core currency of the search engine.

We control the way tokens get generated through an index’s schema definition. How we leverage this power has a huge effect on how the search engine behaves down the line — it’s one of the primary levers we can pull to master it. The choices are wide-ranging and include transformations such as token size (n-grams), removal of troubling stopwords (e.g., the, and, this), and stemming of terms (e.g., books to book). We will take a look at some of these in a moment.

The Index

Pick up any nonfiction textbook and flick to the back pages. Here we have a data structure that maps the key terms of the subject at hand to the passages in which they feature most heavily.

The inverted index of a search engine behaves very similarly. At a simple level, it maps the tokens present in a document to the documents in which it occurs.

Ranking

When you search for something, the engine matches your tokenized query against the index and scores each result using a ranking algorithm. Traditionally an algorithm such as TF-IDF is used. It rates a document higher the more frequently the document matches a term (TF, term frequency) offset by how common that term is (IDF, inverse document frequency).

All records are given a score and pitted against one another to identify the winners: your search results.

So search engines are smart, right?

Not exactly. Although the engineering underneath it all is a computer scientist’s dream, search engines are utter idiots! If the tokens generated from the query do not match any in the index exactly, you are out of luck. For example, if your index has horses but the query analyzer generates horse, too bad.

Garbage in, garbage out! It’s your job to curate.

Don’t just blindly add reams of data to the search engine. It’s your responsibility to turn the incoming mass of text into something useful.

Principle 2. It’s Not Just About Tokens but Features

The tokens generated from a document are more than just items in an index. They represent the core ideas about it.

What is a feature? Let’s take the example of the classic title Dracula from our dataset.

{
    "title": "Dracula",
    "genres": [
        "classics",
        "horror",
        "fiction",
        "gothic",
        "paranormal",
        "vampire"
    ],
    "year": 1897,
    "isbn": "393970124",
    "description": "A Chronology and a Selected Bibliography are included.",
    "publisher": "Norton",
    "average_rating": 3.98,
    "ratings_count": 618973,
    "authors": [
        "Bram Stoker"
    ]
}

Here we see just a few of the features concerning the novel — it’s a gothic classic, published in 1897 by the author Bram Stoker, and has an average rating of 3.98/5.

The source material will usually contain a wealth of features just sitting there ready to be picked, but often the most golden ones need to be mined from just beneath the surface. You need to model features to help communicate the ideas in your document. This technique is named feature extraction. Here you can play the mad scientist, distilling the text down to its bare essentials, stripping away all the confusion and muddiness within.

Feature engineering takes place during the analysis phase of indexing. The search engine has several tools to help us do this effectively. Let’s review a couple of them.

n-grams

The default tokenization strategy of any search engine is usually to (a) remove all punctuation and special characters, followed by (b) splitting on any whitespace. It can work quite well as an initial tactic because searching for a particular word will give you a successful match. For those long words that we have trouble spelling, however, this is not a great approach.

Imagine searching for the book Cryptonomicon. Half of your users are just going to type “Crypton” and hit return. With the standard analyzer, this is not going to get very far. Remember that search engines are dumb, and an out-of-the-box strategy is only going to produce the single token Cryptonomicon, which will not match our search request.

N-grams, and in particular edge n-grams, are great for those fields in which you want to partially match. Take a look at the wealth of tokens generated by taking an n-grammed approach to “Cryptonomicon.”

Using this strategy, because crypto is one of those tokens generated by the n-gram analyzer, we get our match.

Stemming

Stemming involves getting to the root of a word. Many terms have the same core meaning: quick, quickly, and quicker all come from the same place, as do happy, happier, happiest, and happiness.

By reducing a term to its stem, you are distilling it to its core essence. Utilizing this approach, you can translate those thoughts in the head of a user into a successful search result. You’re allowing them to converse at the level of ideas rather than worrying about syntax.

For a more detailed dive into stemming, check out the following section in Elasticsearch’s documentation.

Principle 3. Tuning In to Signals

Feature extraction is paramount to building a well-tuned search experience, but you must always keep the end user in mind while tackling it. What are the use cases for search? What kinds of items are users likely to be tracking down? What properties are most helpful in identifying or suggesting this? How can we manipulate features for maximum effect?

We must understand the user and anticipate their intent. One approach to thinking about this (as described in the book Relevant Search) is the strength of a signal.

Signals

When someone makes a search request, they’re hunting for documents that resonate most strongly with their wants. They are giving us several clues to help us help them: author, genre, title, year, etc. The documents that align most closely with the concepts respond with the strongest signal (as measured by the relevance score).

Not all signals are as clear as others, and some may actively work against you. While one feature may allow for a resounding match, others may generate noise, causing a false positive and detracting from the overall quality of the search results. We have the tools, however, to amplify those signals that are crystal clear and to quieten those that may occasionally be useful but more often than not get in the way.

Boosting

It is commonly the case that a hit on one property (title, for example) is much more valuable than on another (e.g., description). We can influence the search engine to take our preferences into account.

By raising one field above another, it increases the importance of its match over a peer. Those that resonate with your query will do so much more clearly, and those false positives will fade into insignificance. Boosting one field over another is as simple as assigning it a positive number.

Let’s say we fancy a classic Charles Dickens book, and we feed our search algorithm with the term dickens.

The results are a mixed bag. Some are Charles Dickens books, for sure, were but beaten to the podium by a number of others. Your users may think it strange for Moby-Dick to win this competition! We can change that outcome by levelling the playing field and adding a boost to the authors field.

Names are often a more precise match than something in a title or description — especially a full name, but we’ll see that later. By adding a boost to the authors field, we’re emphasising those records to show as more relevant to our search.

A word of warning though. It is a bit of an art weighting and balancing fields. Overly promote and you may end up fixing one problem (e.g., searching for the title) but introduce another (e.g., ability to search by publisher).

Match difficulty and boosting

One solution to the balancing problem is to correlate the size of the weight with the difficulty of obtaining a winning match on the field.

Properties that provide us with a crystal-clear signal include:

ID
Full name
ISBN

Hit upon one of these and you’re on to a winner. Match against the description field, however, and you can never be completely confident because of the wide breadth of the content.

You can sleep easier elevating a feature that’s almost akin to a binary signal: It either matches or it doesn’t.

Same text, different analysis

Consider the author’s name. There are at least two different signals you can see here:

Partial match, e.g., surname: King
Complete match, e.g., full name: Stephen King

The latter signal is clearly of more significance than the former. It is much more valuable for us to know that the user is looking for an author by their full name. However, that doesn’t mean a partial match isn’t valuable at all, even if it could just be coincidental — you don’t always know the exact spelling of an author’s full name.

How can we handle both cases when we have but one name field? We can use feature modelling here to split this into two separate fields. Each feature can be tuned independently, the balance giving higher precedence to the more thorough match but allowing the weaker one to add value when appropriate.

Multi-matching

You do not always look for just one feature, but often mash a ton of different clues into one search string. You might search not only for an author but also specific subject matter from that author — Christmas novels by Charles Dickens, for instance.

By using a best-fields approach, Elasticsearch is going to pick the best match of the bunch: the signal giving us the strongest reaction. This approach can get in a bit of a muddle with the cross-feature query we just saw. What we want is to match primarily against the strongest signal, Christmas, and then augment this with a second field to emphasize its importance.

By setting up a tie-breaker, a value between 0 and 1, the search engine will still proceed to match on the best field, but then it will go one step further and use the other signals to further refine the results. In our example, Christmas could be our primary hit, with Dickens adding enough additional weight to hoist those novels into first place.

Principle 4. Feedback, feedback, feedback

Search isn’t a one-way street. It’s not just about handling a search request and blindly serving the user; it’s critical to have a conversation with them.

Suggestions

Often the user does not quite know what they’re seeking. They bash out a vague notion of what they want on the keyboard and see what bounces back. Without a little assistance, this can be a bit of a shot in the dark. They try, fail, and try again until they either flounder into something usable or give up.

By presenting auto-completing suggestions, you’re giving immediate feedback on the kinds of things available, with the bonus of not needing perfect recall or spelling ability.

Suggestions are supported out of the box by most, if not all, search engines. In Elasticsearch, it is just another (albeit special) field into which you copy your titles, author names, genres, and the like.

Highlighting

Have you ever searched for something and wondered how the heck you got the results you did? If your documents are data-rich, it can sometimes become unclear what you’ve hit when searching.

Providing feedback by highlighting the terms that were matched can turn out to be invaluable, allowing the user to tweak their request to be more specific or to filter out unwanted results.

For more information, read about how to highlight in the Elasticsearch documentation.

Monitoring and iteration

“What gets measured gets managed” — Peter Drucker

Building a truly useful search experience requires learning from your users. Domain expertise is clearly important, but actually seeing what your users come up with can reveal some key insights.

Bouncing around? Search after search after search and still not sticking on any one result. Why is this? What terms are being looked for?

No results? Are a lot of different users looking for something that doesn’t return anything useful? Is your search engine badly tuned, or maybe there’s an opportunity to be pounced upon.

A successful outcome? Did the session have a successful outcome? Was something bought, watched, read?

Gather and analyze this information to continually improve your solution.

The Last Word

We’ve touched upon quite a few different aspects that need to be considered when putting together an engaging search experience. We’ve only just scratched the surface here, and I’d advise checking out the resources below for more information.

Search is a complete subject area in its own right, branching out at the more cutting edges into related topics, including recommendations and machine learning. As with most engineering projects, a successful outcome can be achieved by approaching it in small steps. Make sure you start simple, collect feedback, iterate, and evolve as required.

Thanks for reading!

Resources

Dependency Inversion Principle

Matthew Lucas — Wed, 24 Nov 2021 19:50:33 +0000

“The most flexible systems are those in which dependencies refer only to abstractions and not concretions”

An application is almost always comprised of some core business logic triggering real-world outcomes through a set of utilities. A report generator exporting to a spreadsheet or railway traffic system controlling stop-lights being two such examples.

The flow of control of a program is often from core logic to more specialized utilities, such as I/O.

If the source code dependencies flow in this same direction, however, it can result in tight coupling of business logic with lower-level concerns. As low-level implementations are likely to change often, this will be a source of volatility within an application.

To protect against volatility, we want any high-level concerns to depend on abstractions rather than concretions — that is, interfaces rather than implementations. Interfaces change much less frequently than their instantiations, and so referencing these will shield a client from uncertainty.

Of course, something has to know about the dirty detail of implementation, and that’s where dependency injection helps.

Rather than core logic knowing what to use directly, you inject it with “an implementation” ensuring the source is as ignorant as possible to those low-level concerns. The grubby wiring can be tucked away in a few key places, such as factories or dependency injection frameworks, e.g. Spring.

To conclude this principle, it’s worth reiterating a few points listed by Uncle Bob concerning the DIP:

Don’t refer to volatile concrete classes.
Don’t derive (inherit) from volatile concrete classes.
Don’t override concrete functions.
Never mention the name of anything concrete and volatile.

Originally published at Better Programming

Interface Segregation Principle

Matthew Lucas — Wed, 24 Nov 2021 19:48:16 +0000

“No client should be forced to depend on methods it does not use”

Conceptually the simplest of the principles and one that is closely related to the SRP, the interface segregation principle is about finding the most appropriate abstractions in your code. The reason for this, is again, to protect against one avenue of change affecting other orthogonal ones.

Having to depend on an interface that has muddled concerns can lead to a lot of unwanted baggage. The bloat can leak into your code as you have to cater for cases you don’t care about. Upgrades come thick and fast but for features that you don’t use and with the risk that those changes will break those you do.

Overly complex interfaces are usually grown rather than designed. The slow drip feed of additional requirements, odd tweaks, and little hacks can snowball over the years to create a monster.

Strong interface and API design is an exercise in restraint. Cohesion is your guide here. Keep your interfaces cohesive, collecting together those concerns that change together, and free your clients from an extra burden.

Originally published at Better Programming

Liskov Substitution Principle

Matthew Lucas — Wed, 24 Nov 2021 19:45:42 +0000

“If for each object o1 of type S, there is an object o2 of type T such that for all programs P defined in terms of T, the behavior of P is unchanged when o1 is substituted for o2 then S is a subtype of T”

Ouch! That’s a bit of a mouthful. Let’s forget the computer science-y language for a moment and focus on the core point. The Liskov substitution principle is all about doing inheritance well.

It tells us not to create subtypes that, despite conforming correctly at a contract level, diverge wildly from the true semantics creating some nasty surprises in the behavior of our program in the process.

The usual example used (and I’ll do no different through sheer laziness) is that of the square/rectangle problem. To begin, let’s say we have a type, Rectangle, with two methods #setX and #setY to assign the lengths of the two dimensions appropriately — so far so good.

Now, what if we create a subclass for the special case of a Square in which X and Y must always be equal?

If we have a program that uses Rectangle, and then all of a sudden we sneak in the Square subtype, things could get a bit strange when we call #setX and it changes Y too (or vice versa). The subtype has to make sense in the shadow of its parent.

This isn’t limited to just inheritance but also makes sense in ensuring the robustness of any interface, be it an API, REST call, or something else.

Substitution of any of these for another should be seamless, going unnoticed by any caller. If the client has to hack in a workaround to handle different implementations then we’ve botched the point of having a contract.

Respecting this principle, however, can often be quite difficult. Many interfaces aren’t quite as abstract as they really should be, leaking some implementation detail.

This is especially true when it comes to the things you may not always really consider, like exceptions.

To avoid setting this trap yourself, try and keep your interface declarations as simple as you possibly can — other developers will thank you.

Originally published at Better Programming

Open/Closed Principle

Matthew Lucas — Wed, 24 Nov 2021 19:08:30 +0000

“A software artifact should be open for extension but closed for modification.”

Extension, so this must be about inheritance? Sure enough, in Bertrand Meyer’s original incarnation, that was the case but hold on there!

Over the past 30 years, we’ve learned that inheritance is quite a tricky beast to tame, so we might not want to take that route right away.

What do we mean by preventing modification and allowing for extension? If you have to change reams of existing code every time you make a small alteration then you know there’s something wrong with your architecture — it isn’t closed for modification.

Rather than changing what’s already there, you should be able to build on top of a robust foundation through extension. Adding new features should ideally be easy and low risk.

Extension itself has more than one form. Firstly, you have the traditional mode of inheritance. This, as mentioned, comes with its own set of problems (discussed elsewhere).

Alternatively, you can inject additional behavior into high-level components. Interfaces are the perfect tool for extension through injection. They protect against modification by presenting a well-defined contract that controls change.

They then allow us to extend our software by providing and plugging in new implementations of this contract.

The use of interfaces here ensures a robust hierarchy where those important high-level components are protected from changes in lower-level ones. When that next feature request comes along you’ll be ready to slot it in without touching those precious existing functions.

Originally published at Better Programming

Single Responsibility Principle

Matthew Lucas — Wed, 24 Nov 2021 19:01:55 +0000

“Gather together the things that change for the same reasons. Separate those things that change for different reasons.”

It’s a common misconception to simplify this as “every module should do just one thing”. That’s not quite right because it’s not the “doing” that’s the point.

Boiling your code down to the simplest possible components can lead to horribly disjointed software. It’s the reason to change that’s key rather than what it’s doing, but what defines a reason to change?

The principle is primarily about people. Those users and stakeholders that push for new features are those that drive change in your software, and it’s specifically the scope of that change that we’re looking to limit.

It’s frustrating for both users and developers when a reasonable tweak in one concern leaks into, and breaks, another. You’ll want to separate the responsibilities to avoid exactly this and to reduce the mental burden required by a developer when dealing with just one piece.

The single responsibility principle applies across the whole stack, from minute detail to high-level abstraction. It holds when building granular functions or coarse components and in determining architectural boundaries.

It may be seen under a different guise, for example, the common closure principle which itself advises that components should be deployed according to change and that any change should effect a minimal set of components (ideally one).

Much of the SRP can be seen as an answer to a development scalability problem.

Separate those areas that can change for different reasons and you can scale out your engineers without them treading on each other’s toes or scale-up teams without blocking dependencies and bottlenecks between them.

Originally published at Better Programming

DEV Community: Matthew Lucas

Habot - Your Friendly Habit Coach (Azure Hackathon)

Overview of My Submission

https://habot.azurewebsites.net/

1. Single Signon with Outlook Credentials

2. Creating your first habit

3. Check-in and Progress

4. Getting Help

Submission Category:

Link to Code on GitHub

lucas-matt / habot

habot

Additional Resources / Info

SignPost - A Geo-Social Network (Atlas Hackathon)

Overview of My Submission

1. A map filled with signposts

2. Dropping your own signpost on the map

3. Searching for signposts across the world

How does it all hang together?

Submission Category:

Link to Code

lucas-matt / signpost

Additional Resources / Info

Open Tracing on Kubernetes —Get Your Traces for Free

What’s OpenTracing?

Special agents

Auto tracing webhook

Deploying it all

Trying it out

Conclusion

Why Small Teams Outperform the Competition

What makes up a winning team?

What has this got to do with size … ?

Brain power

Social proof

Pluralistic ignorance

So what can we learn from this?

Further reading

Survivorship Bias and Negotiating Tech Hype

Bombers with bullet holes

Survivorship bias

Choosing tech thoughtfully

Further reading

Tell Don’t Ask

Tell Don’t Ask

Asking

Telling / Querying

Beware the paperboy

Final words

Further reading

“Finding Stuff” — Building a Great Search Experience

Before We Begin

Principle 1. Search Engines Are Dumb, You Are Not

Analysis

The Index

Ranking

So search engines are smart, right?

Principle 2. It’s Not Just About Tokens but Features

n-grams

Stemming

Principle 3. Tuning In to Signals

Signals

Boosting

Match difficulty and boosting

Same text, different analysis

Multi-matching

Principle 4. Feedback, feedback, feedback

Suggestions

Highlighting

Monitoring and iteration

The Last Word

Resources

Dependency Inversion Principle

Interface Segregation Principle

Liskov Substitution Principle

Open/Closed Principle

Single Responsibility Principle