DEV Community: vorsprung

LLMs don't reliably follow instructions

vorsprung — Wed, 02 Jul 2025 16:23:11 +0000

Building your MCP server? Struggling with speed or token window size?

There's even another catch, LLMs don't reliably follow instructions!

There are some best practices for building tools as documented by Anthropic

"Provide extremely detailed descriptions. This is by far the most important factor in tool performance. Your descriptions should explain every detail about the tool"

What they don't document is sometimes that this doesn't matter because the LLM won't follow your instructions

It just won't do what you say!

For more: these ideas (as well as being based on my own experience) are discussed in Episode 51 of the excellent "Vanishing Gradients" podcast

Using the Model Context Protocol (MCP) to use FRED data

vorsprung — Sat, 31 May 2025 13:41:32 +0000

I've recently started using Claude Pro on the desktop. It can produce readable reports from a text prompt. Trouble is it can and will "make up" data. This is called "Halucinating" in the current AI/LLM discussions. This is a big problem for most people that just want accurate information!

One way it can interface with your data in a more reliable way is by using the Model Context Protocol (MCP). To greatly simplify what MCP does, it pushes external data into prompts

To get accurate historical financial markets I am using the Federal Reserve Economic Data (FRED) online database via a MCP Server

How to install FRED MCP server

I am using a MacBook with Claude Pro
First you need the python "uv" system. This allows programs like the FRED MCP server to download all the sub systems they need to work and start up. If you already have homebrew, then installing is

brew install uv

then clone the github repo

git clone https://github.com/Jaldekoa/mcp-fredapi.git

make a note of the path where the cloned repo is

cd mcp-fredapi
pwd

Next, alter the Claude developer settings with the menu Claude->Settings then select developer and hit edit config

Add a json section like this, the "directory" setting will be the repo path from above


{

"mcpServers": {
"mcp-fredapi": {
      "command": "uv",
      "args": [
        "--directory",
        "/Users/jamesandrews/stuff/mcp-fredapi",
        "run",
        "--with",
        "mcp",
        "--with",
        "httpx",
        "mcp",
        "run",
        "server.py"
      ]
    }
    }
    }

restart Claude and you should be good to go

What can it do?

initially I asked it

using the FRED data from the mcp-fredapi mcp server show the S&P 500 for the last 3 months

It made this

Then I asked

if the trend downwards continues, approximately when will a price of 3000 be reached?

It responded

To estimate when the S&P 500 might reach 3,000 if the current downward trend continues, I'll need to calculate the rate of decline from the FRED data and then project it forward

OutputResult

Starting date: 2025-01-23, value: 6118.71
Ending date: 2025-04-22, value: 5287.76
Total days: 89
Total change: -830.95 points
Average daily change: -9.34 points per day
Points to target (3000): 2287.76
Days needed to reach 3000: 245 days
Estimated date to reach 3000: 2025-12-23

Using only the last month of data:
Last month starting date: 2025-03-24, value: 5767.57
Last month days: 29
Last month change: -479.81 points
Last month average daily change: -16.55 points per day
Days needed to reach 3000 (recent trend): 138 days
Estimated date to reach 3000 (recent trend): 2025-09-07

Linear regression model:
Slope: -0.00000013 points per millisecond (-10.94 points per day)
Y-intercept: 226338.02
Estimated date to reach 3000 (regression): 2025-11-14
Days from last data point to reach 3000 (regression): 206 daysBased on my analysis of the S&P 500's downward trend, I've calculated several projections for when the index might reach 3,000:

Using the entire 3-month period (Jan 23 - Apr 22, 2025):

Average daily decline: 9.34 points per day
Estimated date to reach 3,000: December 23, 2025 (245 days from the last data point)


Using only the most recent month (which shows a more accelerated decline):

Average daily decline: 16.55 points per day
Estimated date to reach 3,000: September 7, 2025 (138 days from the last data point)


Using linear regression (most statistically sound approach):

Average daily decline: 10.94 points per day
Estimated date to reach 3,000: November 14, 2025 (206 days from the last data point)



The most recent month shows a much steeper decline (almost double the rate of the entire period), which suggests the downward momentum has been accelerating. If this recent, more aggressive trend continues, we could see the S&P 500 at 3,000 as early as September 2025.
However, the linear regression model, which accounts for the overall trend while smoothing out some volatility, points to mid-November 2025 as the most statistically reliable estimate.
It's important to note that these are purely mathematical projections based on past performance. The stock market is influenced by numerous factors including economic data, Federal Reserve policy, geopolitical events, and investor sentiment that could significantly alter this trajectory.

Which is all quite fun!
The advantage of knowing it is FRED data is that it can be rechecked using another method and that the data is "good" and not hallucinated

Buyer beware...

If you ask it about stuff that isn't in FRED then it may well make something up. But it will say that it has done this. Read what it says carefully...

Using the Model Context Protocol (MCP) to query Obsidian note taking

vorsprung — Tue, 06 May 2025 15:31:03 +0000

I've recently started using Claude Pro on the desktop. It can do a lot of things like simple coding and producing readable reports from data

One way it can interface with your data in a more reliable way is by using the Model Context Protocol (MCP). To greatly simplify what MCP does, it pushes external data into prompts

I've been keeping notes in Obsidian for a couple of years, it's semi structured and so an idea target for the LLM in Claude Pro

You can find out about Obsidian on their site It's free to use and open source

Add the obsidian-mcp server

Adding this project is very simple, see the repo on github
There is a npx one liner

npx @smithery/cli install mcp-obsidian --client claude

This automatically updates the config in Claude desktop and allows it to run the latest version of the mcp-obsidian server. Claude asks for the path to your Obsidian database, gets restarted and.....

Fix a clash with another mcp server

...Trying simple queries against mcp-obsidian server didn't work for me!

It kept failing and saying it couldn't find the files in a list of paths that didn't include the Obsidian database path

I worked out that what was happening is that the previously installed "filesystem" mcp server which Anthropic give as an example was used as the basis of the mcp-obsidian server and this causes clashes

I removed the filesystem mcp server and Obsidan was then available

What it can do

I found a useful prompt was 'using the mcp-obsidian MCP server, summarise activity in the Journal'
Obsidian has a "Journal" feature for a daily record of activities
Part of the output summarised what I have been doing and categorised it.
With day to day work it is easy to loose track of what the themes are so this was helpful

Journal Activity Summary

Based on my analysis of your journal entries from late April to early May 2025, here's a summary of your activity:

Journal Structure

Your journal is well-organized with a consistent format that includes:

A date header and status section listing ongoing tasks and projects

A "Today" section with specific tasks accomplished that day

Time tracking for workday start/finish and lunch breaks

Navigation links to previous and next days

Key Work Areas

AWS & Cloud Infrastructure

You've been working extensively on AWS infrastructure tasks including:

Transit gateway configurations for network connectivity between environments

Troubleshooting network paths between different environments (especially customer c9 qualifications)

Implementing and fixing security configurations (IMDSv2)

Infrastructure qualification (IQ) processes

Monitoring & Observability

A significant focus area appears to be implementing monitoring solutions:

Setting up Prometheus and Grafana for monitoring

Working on a "prometheus plan 2" with exporters and systemd configurations

Configuring paths between exporters, Prometheus, and Grafana

System Administration

Regular administrative tasks include:

Server maintenance and patching schedules

System upgrades (Ubuntu 22.04/24.05)

Authentication issues (SSSD configuration fixes)

Access management for team members

Support & Troubleshooting

You provide regular support on issues like:

Oracle database access problems

Application performance investigations

Infrastructure qualification failures

Authentication and access issues

what next

I suppose I had better have a go at fixing the clash with the filesystem mcp server

Using AWS Model Context Protocol awslabs-cost-analysis-mcp-server

vorsprung — Mon, 07 Apr 2025 16:35:01 +0000

I've recently started using Claude Pro on the desktop. It can do a lot of things like simple coding and producing readable reports from data

I was interested to learn about the Model Context Protocol recently. To greatly simplify what it does, it pushes external data into prompts

With this crude method it seems possible that LLMs can be interoperable across differing corporate software, across different types of interface and could be a rosetta stone for a new dawn of one thing speaking to another

I saw that AWS had a new set of MCP servers available, including one that dealt with billing and cost. I am interested in the costs so I did a quick try out. TDLR; this is not a new dawn

step one, get the MCP server running

The information to get the MCP server running with Claude Desktop was scanty but here is a recipe that worked for me (on a Mac)

Do this in the shell. The uv thing is required. uv does some package management plus there is a uvx command that we need in the next step

brew install uv
uv python install 3.13
uv venv
source .venv/bin/activate
uv init costly
cd costly
uv add "mcp[cli]" httpx
pip install awslabs-cost-analysis-mcp-server

Do this in Claude Desktop -> settings -> developer -> edit config. The edit config for me opened vscode. Remove the empty braces and add the below. I have nothing in the MCP_SETTINGS_PATH. I really do have a AWS profile called Hauptsteuerung

{
    "mcpServers": {
      "awslabs.core-mcp-server": {
        "command": "uvx",
        "args": [
          "awslabs.core-mcp-server@latest"
        ],
        "env": {
          "FASTMCP_LOG_LEVEL": "ERROR",
          "MCP_SETTINGS_PATH": "/Users/jamesandrews/mcp/mcp.json"
        },
        "autoApprove": [],
        "disabled": false
      },
      "awslabs.cost-analysis-mcp-server": {
        "command": "uvx",
        "args": [
          "awslabs.cost-analysis-mcp-server@latest"
        ],
        "env": {
          "AWS_PROFILE": "Hauptsteuerung",
          "FASTMCP_LOG_LEVEL": "ERROR"
        }
      }
  }
}

Now in the Claude window there is an extra icon that looks like a hammer. If I click on it it gives some info on the new MCP things I have installed

prompting - hot or not

Here's where it all goes wrong. I added an export file of our RDS servers.
The phrase get_pricing_from_api service_code=AmazonRDS should get Claude to use the MCP to get the current up to date data from AWS api! I whacked in the prompt below, Claude confirmed that it was allowed to run the MCP commands, it was all looking great...

the rdsapr2.csv file has the name of the RDS in column 2, region in column 4 and the instance size in column 5

using get_pricing_from_api service_code=AmazonRDS and the regions available in the rdsapr2.csv file, generate a report of the relative cost of each RDS instance in the rdsapr2 file

I need to compare the no upfront 1 year RI vs. on demand cost

This is the result it came out with

RDS Instances Pricing Analysis

Regional Distribution

Region	Number of Instances	Percentage
us-east-1	38	55.07%
eu-west-1	13	18.84%
eu-central-1	7	10.14%
eu-west-2	6	8.70%
ap-southeast-1	3	4.35%
us-west-2	2	2.90%
Total	69	100%

Pricing Insights

On-Demand vs Reserved Instance Cost Comparison

Instance Type	On-Demand Hourly Rate	1-Year No Upfront RI Rate	Potential Savings
db.t2.micro (MariaDB)	$0.017	Not significant	Minimal
db.m5d.xlarge (Oracle SE2)	$0.805	$0.576	28.4%
db.t3.2xlarge (MySQL)	$1.088	$0.765	29.7%
db.r5.12xlarge (Oracle EE)	$11.184	$6.469	42.2%
db.z1d.12xlarge (SQL Server SE)	$20.513	$19.385	5.5%
db.x2iedn.16xlarge (MySQL)	$35.533	$27.360	23.0%

Cost Optimization Recommendations

Regional Consolidation

- 55% of instances are in us-east-1
- Consider consolidating or optimizing multi-region deployments

Instance Type Analysis

- High variation in instance sizes and types
- Recommend comprehensive utilization review
- Potential for significant cost savings through right-sizing

Reserved Instance Strategy

- Most instance types offer 20-40% savings with 1-year No Upfront RIs
- Evaluate workload stability for potential RI purchases

Database Engine Considerations

- Mix of engines: Oracle, MySQL, SQL Server, MariaDB
- Licensing costs vary significantly
- Review BYOL vs. included license models

Detailed Observations

Lowest Cost Instance: db.t2.micro (MariaDB) at $0.017/hour
Highest Cost Instance: db.z1d.12xlarge (SQL Server SE) at $20.513/hour
Most Common Region: us-east-1 with 38 instances
Least Utilized Region: us-west-2 with 2 instances

Next Steps

Conduct a comprehensive audit of current RDS instances
Create a detailed utilization report
Develop a rightsizing and cost optimization strategy
Consider Reserved Instance purchases for stable workloads
Implement tagging for better cost tracking and allocation

Which is great except

I have no MariaDB
I have no db.z1d.12xlarge instances, not even one
Most of my servers are postgresql which isn't mentioned

I queried this and

what is the name of the db.z1d.12xlarge server mentioned in the csv file?

Edit

I'll retrieve the name of the db.z1d.12xlarge server from the CSV file.

Analysis

I apologize, but it seems that there might be no db.z1d.12xlarge server in the actual CSV file, despite being mentioned in the pricing analysis. This could be because the pricing information comes from AWS's standard pricing API, while the specific servers in your CSV might differ.

So not quite got it doing any useful work yet but maybe some day soon :)

Interviews - the problems and a solution

vorsprung — Fri, 29 Jul 2022 12:27:44 +0000

What was the point in all that?

Does the process CV + interviews = a suitable happy person working?
Clearly in the imperfect world we live in, no
Here are some random thoughts about it. Enjoy

Problem 1: Sell yourself

It makes the process into a sales process. There is an incentive from the employee side to exaggerate the CV skills. There is an incentive to reduce risk by just walking away from the employer side
Treating potential employees as commodities to be bought is not a great way to get engagement from those employees

Problem 2: First impressions

First impressions are often wrong but everyone uses them
All those studies that show assessments of people are made from literally first impressions are true https://www.psychologytoday.com/gb/blog/am-i-right/201302/the-power-first-impressions#:~:text=First%20impressions%20matter%2C%20for%20good,The%20halo%20effect%20distorts%20reality.
People in HR are not immune. Having a checklist is not a vaccine against it
Making a meeting with first impressions part of the process means that this is going to be a problem

Problem 3: Experience is difficult to assess

Technical skills assessments are not for what you actually know from your experience. They are “can you do the puzzle”. In a sense they assess your willingness to jump through hoops. Famously the creator of the Mac “homebrew” system - used by Google internally and countless others couldn’t get a job at Google as he didn’t pass the merry go round of interviews. Here’s a link to the story
https://www.quora.com/Whats-the-logic-behind-Google-rejecting-Max-Howell-the-author-of-Homebrew-for-not-being-able-to-invert-a-binary-tree

Problem 4: Put on an act

It’s too easy to put on an act of being a nice person
Culture tests are either psychometric multiple choice tests or informal assessments. I won’t discuss psychometric tests and their issues but clearly informal assessments are prone to “give them what they want”
As I discussed in the previous articles in the series the "values" that companies have can be easy to claim for your own in words only. Real values are based on actions

Problem 5: Money vs. work

The fundamental employer employee power relation problem is: they want the most work for the least amount of wage. You want the highest wages for the moderate amount of work.
This doesn’t just apply to time/money. You might want to work on the latest technology but the fact is with most companies the $$$ is in legacy systems. In many areas the employer and the employee are at odds on what the aims are. I won’t go into the wider social / economic reasons why this is the case, but it is

Problem 6: Who are the hiring people?

Another problem is that many of the people you meet on your hiring journey are not the people who will be directly shaping your working environment. The head of Cloud or the HR person will not be your peer or line manager
Those people you clicked with in the interview, they are not your manager or co workers

A Solution:

Many people I know seem to agree that the best way to get a job is:

Someone you know and trust works there, they explain the ups and downs
You agree that on balance it might be fun
Your inside contact helps you get the role, often bypassing the normal cv/interview thing

image credit: Pieter Brueghel the Elder: The Tower of Babel

Interviews ... and how to survive them

vorsprung — Wed, 27 Jul 2022 17:26:47 +0000

Types of interviewers and interviews

So you have been offered an interview, formal or informal. It’s a chance to talk to a person.
Types of people you will be dealing with: hiring managers, team members, recruiters

Recruiters, as discussed before are not technically savvy but know more about the business need for this role than you do.
Usually, there is an “initial chat” with a recruiter to screen out clearly unsuited people. Aim to get information from the recruiter about the next few stages, “what they are looking for” in general terms and convince them that you are not crazy.
Next, there will be a technical interview probing what you know. These vary a lot in format, depth and scope
There might be a “practical” technical interview, like pair programming or a takeaway program. This is often with a team member
Last, there is a “culture” or “behavioural” interview, often with a manager of some sort.

With all of these, you must prepare.

Preparation

The Initial chat is usually with a recruiter. Preparation is - get ready to talk about yourself and your achievements. Don’t complain about your current role. Do have a reason you are leaving. Do have a reason why you think you’d like the role. These types of initial chats are usually quite easy if you have covered all the requirements in your cover letter.
Sometimes the initial chat is not with a recruiter but is a technical conversation or even a cultural assessment. If you think this is the case, prepare accordingly.

For either of these subsequent interviews, technical or cultural gather clues from the interview request email, linkedin search the person who will be interviewing you, look at the job description and make notes in any prior interview. It could be that you have worked at the same company at some point and something like that will help establish rapport

If the interview email says “we will discuss your CV” think of a way to explain recent roles as if they are ideal preparation for this role

For technical interviews: Anything that is mentioned, make it fresh in your mind. For example, they mention terraform, take a moment to review a terraform 101 or a cheat sheet.

If the technical interview includes some kind of quick coding test, then practice coding in the language they mention. I made this exact mistake in a recent interview. I had been writing terraform code maybe two weeks before but mainly using EKS (AWS Kubernetes) providers. The test code (which was supplied up front that I didn't read) was for AWS but with making a VPC from scratch with EC2 in it. I was so clumsy and slow with this that the interviewer said "have you ever used terraform before"?

If the role is suited to you, (for example in my case it is SRE) but it is in an area where you don’t have much experience ie Machine Learning then look for talks or papers from conferences that home in on this exact area - in my example SRE for machine learning. Often talks at conferences have a nice diagram or summary about how it works, get this straight in your mind. I’m not saying this is a shortcut for learning stuff overnight! But the process of explaining a composite topic that you perhaps have not explained for a while is worth going over. There isn’t unlimited time in an interview so there isn’t any need to become an expert overnight

For culture interviews, look for company “values”. This is a thing, apparently. Find them, understand them and prepare answers for value-based questions. For example, if one of the values is “Value customers” a possible question would be “tell me a time when you went the extra mile for a customer”

Most interviews are on line now - I won't cover zoom etiquette apart from: Immediately before the interview set an alarm so you are on time.

Pro tip: Prepare for those "interview" questions

As well as technical knowledge there are “interview questions” like “what is your greatest achievement?”. You only hear these questions in interviews. I can’t answer that cold. So I find a list, do answers to them all and write them down.

Pro tip: Questions you ask them!

The other sort of question is the question you ask them. My two favourites are “why are you hiring for this role?” and “describe the first 30/60/90 days, how quickly do you expect people to be productive”. Whatever questions you ask, you will learn more about the role
You need at least 5 of these questions to ask that suit the role and be ready to ask them when there is an opportunity. Some of the questions will be answered by the rest of the conversation you’ve had. If you get to the 2 minutes at the end of the interview that are “for questions” and you say “er, I have no questions” it isn’t a good look. Here is a list https://arc.dev/developer-blog/questions-to-ask-at-an-interview/

What happens next

If you pass the interview then your reward is usually more interviews. Hopefully, you will get to the point where there isn’t another interview

Photo credit

How to get an interview

vorsprung — Tue, 26 Jul 2022 09:24:22 +0000

How to get an interview in the first place

I won’t cover where to search for suitable roles. I used linkedin, my favourite recruitment agents and hired.com. I’m sure you have a bunch of places to get leads. That’s not what I am going to write about here!

So, step one: you have a job description, you like the look of it and you feel that you are in with a chance. What nearly all of them want is a CV and a cover letter.

This CV and cover letter will go to a recruiter person or an automated process that is used by the recruiter person
So the initial point of contact is this recruiter person or an automation. Most automations look for keywords in the list it has for the job role that goes with the application. Most recruiters are more perceptive than the automated process but essentially are doing the same thing, looking for key skills.

CV tips

So your CV has to mention those keywords and mention them in a fairly obvious way
The CV should also list your “achievements” by which I mean you say I was at company X and I did a thing and it had this positive effect. I was at Voodoo Corp, I took charge of the AWS infrastructure and reduced the bill for infrastructure by 20%. List things that the non-technical recruiter will understand and link them to business requirements. In DevOps speak, shift left :)
Be sure to list all your certifications. Certifications seem fairly pointless but the CV is their raison d’etre. All a certification does is prove that you can do a thing to a certain standard to people who don’t know you, exactly the context of a CV
If you struggle to make the CV look good and keep it up to date then I recommend Kick Resume

Cover letter cheat sheet

The cover letter is probably something I could do better but I have a simple formula.
1/ “Hello, I think we should talk” or “Hi I noticed your job ad and I believe I have something to offer” - make a friendly introduction, try and give something positive
2/ Pitch with 3 quick points (you like the idea, you are suitable and you want to do the job) “I am currently in the SRE field and Goats have always fascinated me, so when I saw you advertising for an SRE at Goats R Us I was immediately interested. I have many years of experience (see below) and I am actively looking for a new challenge”
3/ The job description always has a list of “requirements” or “what we are looking for”. Cut n paste this into the letter and fill in after each requirement what your experience is. If you don’t have experience in one of the areas, just skip it.
4/ End with: Thanks for reading, hope to hear from you soon
This method for doing a cover letter takes a little while as you have to fill in the requirements

what happens next

Send the CV/Cover letter off
If nothing happens don’t worry about it. Sometimes the recruiter person is on holiday. Sometimes they filled the role and didn’t pull the Ad and can’t be bothered to reply. If you never hear from them again, no problem!

If you do get an interview…your troubles have just begun

Photo credit

VSCode, pipenv, python

vorsprung — Thu, 16 Dec 2021 09:32:31 +0000

Python squeezes until the pipenv squeeks

The problem I have

I don't use python a lot and although I have all the VSCode plugins for python installed, when I load a project up, there are lots of "Problems" about missing python libraries. "unable to import"

The project works fine in the shell, I can run tests and all the things. All of the projects use pipenv

What I'd like

I want the missing python library warnings to disappear so I can use the "Problems" pane to find...er...real problems with the code

Problem with some online info

It seems (and I vaguely remember doing this in the past) it used to be the case that setting the pythonpath in the settings would fix this issue to a great extent. But now this does not work and the setting is marked as "deprecated"

VSCode official docs

The official docs for "Using Python environments in VS Code" are pretty good but they don't give an exact method for using pipenv. There are some good clues however. It seems that the key is to make a venv in the file folder .venv in the Project.

pipenv work around

This work around is good for Linux or a Mac.

go to the project directory in a shell. This can be any shell on the host, including the vscode terminal
ensure that no virtual environment is in effect. The deactivate command can stop a virtual environment
PIPENV_VENV_IN_PROJECT=1 pipenv shell
say yes when VSCode asks "add this environment?"
probably at this point you need to do pipenv update For projects with lots of dependencies, this will take a while
do a ⇧ ⌘ P "Developer: Reload Window"

If you still have missing libraries....

I found that the project - as well as having a Pipfile - had a library in a "vendor" folder! To fix this make a file in the project directory called .env with the contents PYTHONPATH=vendor

If the folder has a different name to "vendor" set that in pythonpath

Small Wins for 2021

vorsprung — Wed, 15 Dec 2021 13:46:36 +0000

Small Wins for 2021

My small wins!

Here's a list of the things that I did or learnt last year that were not earth shatteringly amazing points in my career - most of them aren't that important or memorable. But that's a great reason to record them

Why keep this list?

Keeping a list like this helps with self esteem, counters imposter syndrome and helps show a realistic view of the things you've learnt. The next time you have a bad day and think "I can't learn anything! Why am I so behind on the latest tech?" take a look at your small wins list and it will put it in perspective. Maybe you aren't a thought leader or incredible creative- but you have done plenty of stuff!

Make this a New Year's resolution!

Why not start a list like this? I've had mine since 2017! It can help you with your motivation and it's a great resource if you have a review or assessment

How I didn't get a DevOps job for Xmas

vorsprung — Fri, 20 Dec 2019 14:12:32 +0000

How I didn't get a DevOps job for Xmas

At the end of November, I had an awkward call with my line manager at the place I was working. They had decided to lay me off as a remote worker and hire someone at their base

The remote working communications had been not great. They were a small company not used to dealing with remote staff. The culture was based around the one office

It had been a cool place to work but I agreed that it was time to look for something else

Shouldn't be too hard right? I am an experienced DevOps/Systems Admin/Developer specialising in systems automation, software-controlled infrastructure, CI/CD pipes. I've got strong Python, Linux admin and a supporting cast of dozens of practices and technologies from many years of experience

I'd prefer remote but I live in commute distance of Bristol in the UK which is a great tech centre

First few attempts

Pretty much the next day after the call I was looking at all the places I normally look for jobs. My CV was, as always, up to date. I use the excellent kick resume site to manage it

I did have to reinstate the details
on LinkedIn. I'd removed them to lessen the amount of spam I get from recruiters. Now, however, I would be fine to get recruiter emails. More about that later

Initially, I applied to a couple of places. A local consultancy where a friend already worked - giving me glowing reviews about it
And a company I'd not heard of before, UK based, who specialised in secure communications for distributed workers

How these things work is that you apply with a CV and cover letter. Then if you get through screening there is an initial call from someone non-technical (maybe in HR if they have that) just to check that you aren't completely nuts

I got through the initial call phase with both of the above easily

Next, all the companies have a second layer - either an interview, a technical challenge or something else.

The secure comms company came up with something else. They had an automated video interview system. It asked questions, gave you thinking time and then you recorded an answer

The consultancy went for a technical task, which they said should take 6 or 7 hours. When I read it I thought "Well maybe, if everything goes swimmingly that's 7 hours" Of course it didn't and I spent the entire weekend building a CI/CD pipeline

Of the companies that use a technical task as part of their process, it's usual to then use this as the basis for a further meeting.

So I was lined up to be grilled by 4 developers and managers in the team I was going to be hired in. Just before the Google Hangout, both of my Bluetooth headsets refused to pair with the laptop and I noticed a bug in the implementation I'd done. I started the interview flustered and didn't really recover. At one point they asked me what stuff was in the VPC I'd set up - a real 101 question which I failed to answer at all convincingly

So that was a turndown and they were left with the impression that, actually, I don't know much about AWS

The secure comms people didn't get back to me for a while. When I contacted them they'd hired someone cheaper instead

Then I had a weeks holiday booked abroad so I had a week off

Round two

Refreshed by winter sun I really got on the case when I returned

There's a data science company local to me.
There's an opening where I used to work.
There's an interesting sounding startup offering fully remote. Apply to all of them!

An email comes in from a contact saying yes, our company is hiring
fill in the form, please.

I contacted my favourite two recruiters SR2 and Techfolk

Both of them come up with some great ideas, one of which leads to an interview/ tech task/ another interview at a software company in Bristol

Unlike the many recruitment agents who contacted me on Linkedin. Most of them were offering roles in London and/or fintech. I also had a surprising number of direct approaches from companies abroad. But I don't want to work in Sweden, Frankfurt or Dublin.

After an intensive 2 weeks of zoom, tech tasks, phone calls and onsite interviews I have got to the "we will let you know" step with 4 companies.

Another week and I should have an offer! Maybe two!

Bust

However, the two favourite ones both let me know that no, I wasn't for them.

The other two are still pending but let's face it. The longer you wait the less likely a positive outcome is

So no job for me!

What did I learn

A few things of interest became apparent

Tech tasks

I am usually not too bad at these. They are often a great learning opportunity as inevitably whoever you are hiring with will assume that their in house way is just obvious: but it will include at least one thing you never used

Another potential problem is that they all assume that you can do frontend things easily. I am a backend only person so that's always a refreshing change

If they say the task will take x hours then best realistic hope is that it will actually take x + 1

Although one of the above was 2x + 1

Noone offers to pay you for what is essentially work in your own time

Travel expenses

I'm sure that I used to get travel expenses for interviews? Not any more! Not offered by any of them

Feedback

Some of them offer "feedback". This is of little use in refining your offer and its main value is to offer closure. One said "there is a gap between working philosophies, methodologies and communication styles" which kind of means that we didn't like the way you talked.

In a sense, the problem with feedback is the problem with the interview process itself. Interviews cannot, in a limited time, work out if you are any use. So also feedback, in a limited format cannot express the irrational decisions of the interview process

The upside

So, no job to look forward to in the new year. But there is an upside. I have loads of time to learn cool new stuff! At the moment I am reading the papers from The Database Internals Advent Calendar, I've got the Tilt system working with my Kubernetes toy project and I have a book on TLA+ to work through

Using AWS with Ansible but from Google Cloud

vorsprung — Sat, 09 Nov 2019 17:40:49 +0000

What is the problem?

Ansible on AWS can use server roles to very conveniently run playbooks with access to AWS resources
Or if it’s from a “home” account, AWS creds in files work just fine.
But ansible running remotely on non AWS (ie on prem, GCP, Oracle Cloud etc) then getting the credentials loaded can be a problem

How it works

Step one, ensure local creds in file

Some credentials should already exist on your local host in the ~/.aws/credentials file. All you have to do is convert those credentials into a format that ansible can load.

Ansible AWS module is based on boto3 and so will honour environment variables AWS_SECRET_KEY_ID and AWS_ACCESS_KEY_ID. Ansible can load variable files into environment variables: this is path I’ve taken

The file format in ~/.aws/credentials is like this

    [default]
    aws_access_key_id = AKIAXEX5OPTQQLXUAVL5
    aws_secret_access_key = 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

    [admin-account]
    aws_secret_access_key = mzTh1Sj9Btjxucbq//X6Ece1z6U6XE3Cysb+Ntdm
    aws_access_key_id = ASIAXEX3PUBILNYIJHAY
    aws_session_token = sXiACeiyqiPpd5z2wAHvEtmTR7/GGf88yVCJ/epz3StU2cU0y1SJUKo6+gch
    2YSsldTlsZzDEBSDubzGKtYur6rmzYZfq+v+KwrUbGobETwn1gvaNd4vg+kEaCg6eNhICisCMRHjrLfK
    YdaikcP9SbUSqQPHwew/F5rtzyWwU5pEle+8uHfYycwY1sSHWOn4bnBA9IpiiJDonTKdnPjWdwuep2E9
    pf7A3fxaPOCb+jss9CoI9YfzPLQ9CQSztqfdhqN4PxYvpXnLSms3Z6SB7cS

The format needed for ansible is

    ---
    env_vars:
    AWS_SECRET_ACCESS_KEY: mzTh1Sj9Btjxucbq//X6Ece1z6U6XE3Cysb+Ntdm
    AWS_ACCESS_KEY_ID: ASIAXEX3PUBILNYIJHAY
    AWS_SESSION_TOKEN: sXiACeiyqiPpd5z2wAHvEtmTR7/GGf88yVCJ/epz3StU2cU0y1SJUKo6+gc
    h2YSsldTlsZzDEBSDubzGKtYur6rmzYZfq+v+KwrUbGobETwn1gvaNd4vg+kEaCg6eNhICisCMRHjrLf
    KYdaikcP9SbUSqQPHwew/F5rtzyWwU5pEle+8uHfYycwY1sSHWOn4bnBA9IpiiJDonTKdnPjWdwuep2E
    9pf7A3fxaPOCb+jss9CoI9YfzPLQ9CQSztqfdhqN4PxYvpXnLSms3Z6SB7cS

AWScre.py in my github will do this conversion

Step 2, run the AWScre

To use the program call it with 3 parameters
The credentials file path, usually $HOME/.aws/credentials
An output path to be used later by ansible, for example awsvars.yml
The account that is being used, for example ‘admin-account’

The output path should be in the ansible directory so that it is available on the remote host

Step 3, use in Ansible playbooks

Next, add a vars_file directive to load the file you just made. Add this to the playbook that needs to use AWS from GCP, like this

    ---

    - hosts: all
    gather_facts: yes
    become: yes
    become_user: root
    vars_files:
        - ../awsvars.yml
    environment: "{{ env_vars }}"

The python script I’ve written to setup the AWS credentials also makes a tweak to the sudo settings on the remote to ensure that the variables are exported during the “become_user”. So the output file that is used by ansible actually looks like this

    ---
    env_vars:
    ANSIBLE_BECOME_FLAGS: '-H -S -n -E'
    AWS_SECRET_ACCESS_KEY: mzTh1Sj9Btjxucbq//X6Ece1z6U6XE3Cysb+Ntdm
    AWS_ACCESS_KEY_ID: ASIAXEX3PUBILNYIJHAY
    AWS_SESSION_TOKEN: sXiACeiyqiPpd5z2wAHvEtmTR7/GGf88yVCJ/epz3StU2cU0y1SJUKo6+gc
    h2YSsldTlsZzDEBSDubzGKtYur6rmzYZfq+v+KwrUbGobETwn1gvaNd4vg+kEaCg6eNhICisCMRHjrLf
    KYdaikcP9SbUSqQPHwew/F5rtzyWwU5pEle+8uHfYycwY1sSHWOn4bnBA9IpiiJDonTKdnPjWdwuep2E
    9pf7A3fxaPOCb+jss9CoI9YfzPLQ9CQSztqfdhqN4PxYvpXnLSms3Z6SB7cS

How well does it work? Transferring software artifacts held in s3 from AWS to GCP was as fast as you’d expect, a few seconds for hundreds of MB

Why Serverless needs Ops

vorsprung — Sat, 16 Mar 2019 09:28:03 +0000

Dr Werner Voglers famously said, "No server is easier to manage than no server".

As he is the CTO of Amazon and so one of the people pushing the move to Serverless, you'd expect him to hold this view.

But in many ways, he is right. But this isn't the end of the story

What is software for?

Serverless or a server is a platform to run software on. It's worth remembering the things that are the purpose of software

Whatever it is your customers are paying you for
Unique value, differentiation
Making your product automated, scalable, 24x7

And serverless can help with these

Whatever it is your customers are paying you for - that can appear quicker in a more responsive way than with using servers

The unique value, differentiation of the incredible software you are making should shine through - so let SaaS do the mundane stuff, you have more resource for specialization

Making your product automated, scalable, 24x7 - this is one of real "batteries included" features of SaaS

How to manage servers

To manage a server you need to be expert at these things

Scalability
Resilience
Availability
Maintainability
Simplicity in complex systems
Instrumentation and visibility
Graceful degradation

Being an expert at these things is server operations. Without these things your software isn't going to perform on a server

But remember serverless, does away with servers. And part of this is dealing with operational issues

The operations that Serverless includes

A great way of understanding the advantage of serverless is to see how it can usurp people doing server operations.

Scalability - serverless does this automatically
Resilience - the cloud infrastructure has resilience
Availability - cloud is 24x7
Maintainability - cloud infrastructure has good maintenance features
Simplicity in complex systems - cloud products are simple and reliable
Instrumentation and visibility - basic metrics are available
Graceful degradation - can be implemented

Of course, the serverless providers have teams of people doing this for you. And they can afford to hire the best people, due to economies of scale

Serverless isn't magic, it's specialisation.

Instead of you having to hire server operations people, this bit of work is supplied by the serverless provider.

Why serverless isn't the end of the story for operations

However, if we dig in a little deeper, often serverless still needs operations people

Scalability - serverless does this automatically but within limits that must be understood. For example, in AWS S3 there is a limit of 3500 put operations per bucket per second. It's important to know the limits and understand the alternatives
Resilience - the cloud infrastructure has resilience usually as some “provision”. For example with AWS DynamoDB, there is a cost limited option or a "on demand" option that must be understood and set up
Availability - cloud is 24x7, except for very unusual events. For example in 2017, there was an outage on AWS S3. Even AWS itself was caught out by this as the service had previously been so reliable.
Maintainability - cloud infrastructure has good maintenance features, use of which has to be planned. For example, if using AWS API gateway as part of your stack, changes to the API have to be understood as a threat to stability and planned in ahead
Simplicity in complex systems - cloud products are simple and reliable. But emergent behaviour is still there. It's easy to add a microservice, and another, and another until there are many of them. Microservices architectures have the services calling each other. So to do some numbers, if each service has one input and one output and there are 10 of them and if they can be arbitrarily connected that's 9 factorial or 362880 different ways of connecting them. In the real world, most of these combinations would be unsuitable but the possibility is still there for misconnecting things. And when something like this does occur, it's down to you to find it, in production
Instrumentation and visibility - basic metrics are available. But application specific metrics have to be added. Going on from the example above with 10 cross-connected microservices, imagine how much easier it is to fix them if there is application level logging implemented for transactions. Companies like Honeycomb have a business supplying this extra, necessary level of traceability in the applications that are implemented on serverless or PaaS systems - which show emergent behaviour but do not monitor everything out of the box
Graceful degradation - can be implemented. And this can be difficult. The ideal form of graceful degradation is that valued customers got to use the part that is still working while the less important traffic is discarded. Maybe someone better informed than me can suggest a pattern for doing this within serverless

In summary: operational problems don't go away because you are running serverless!