DEV Community: mmiask

Career change to IT - things you wish you knew

mmiask — Mon, 06 Dec 2021 12:48:53 +0000

Hi!

I'm Michał and I'm a self-taught web developer, who started working in IT as a manual QA in December 2019. Before that I worked in biotech industry, including a pharmaceutical company in Poland.

I remember the very beginning of my career change and how much work it required (including some proper motivation). All of the doubt, roadblocks but also the excitement of learning stuff in a completely new (to me) field.

If there are people here switching to IT, or who successfully did that - what is your experience? What questions did you have at the beggining of the road that you wish were answered back then?

I want to create an episode of a podcast on the above topic (a live Q&A) where me and a bunch of other people with a similiar experience answer the questions that you would like answered. Is there something you think should be covered in that kind of content? What confused you the most when switching to IT?

Thanks in advance and (hopefully) see you soon!

Yarn.lock: how it works and what you risk without maintaining yarn dependencies — deep dive

mmiask — Fri, 03 Sep 2021 18:13:20 +0000

Coding is fun, there's no doubt about it.

Other things that are fun: Testing! Code maintenance! Keeping dependencies in sync!

No? Only for me? Well, it might not be fun for many of you, but to keep your library/application working properly you'll need to adapt and at least try some of it.

If you have written a JavaScript library and published it on NPM (Node Package Manager), at some point in its development lifecycle you have experienced inconsistent behavior and you didn’t know the root cause. It was working fine for you and some of your coworkers/contributors, but crashing for everybody else. What is going on here?

There is a high chance it was caused by unhandled dependencies in your project. This article addresses that problem and focuses on a very specific task: lockfile maintenance. I'll show you:

What yarn.lock (lockfile) is
Why do you need to do lockfile maintenance
What is the possible solution
What mistakes we've made and how you can avoid them

Disclaimer: Everything mentioned below is based on true events. No developers were harmed during implementation and/or for the purposes of this article

Problem

For the last few months, I've been working at Stoplight, as a part of 11Sigma, on an open-source library called Elements.

An issue emerged in our library integration that made us challenge our belief that our dependencies are under control. In short, our app crashed because of a bug in a dependency’s dependency. Despite fixing the nested dependency, the issue remained to be unsolved.

Fortunately, it turned out to be a non-issue on a fresh install of Elements.

Unfortunately, that meant that we weren't testing what users were using at a given moment.

In order to understand this issue fully, we first need to learn about the lockfile itself and how dependencies are installed in a project.

What is a lockfile?

If you are familiar with how a lockfile works, feel free to skip to “Problem - A Second Look”

To understand why the topic of this article is important to you, it is necessary to know what a lockfile is and how it works. Although it can have different names depending on whether you use npm or yarn, the premise is pretty much the same. I'm using yarn so I'll use yarn.lock as an example in this article.

When you run yarn in your project, two things can happen:

A lockfile (yarn.lock) is generated (if there isn't any) according to the contents of package.json
Packages are installed according to contents of an existing yarn.lock

Important: When you install dependencies in your application or library, only the top-level yarn.lock file is respected. Lockfiles within your dependencies will be ignored.

In short:

When present in the project, yarn.lock is the main source of information about the current versions of dependencies in a project. Yarn uses that information to check if it needs to update anything - it compares dependency versions currently installed in a project (listed in yarn.lock) to version restrictions in package.json and updates packages if needed. Information from the lockfile can be further used by other users to create a repeatable environment elsewhere.

Generating `yarn.lock`

Whenever you run yarn (which is the equivalent of running yarn install) upon a fresh install, a yarn.lock file is generated. It lists the versions of dependencies that are used at the time of the installation process. That means it looks into your package.json and depending on the versioning syntax, it will install your project dependencies, then their dependencies, then their dependencies, and so on...

For more info about dependency versioning check this link

Let's say your project uses two dependencies: chicken and farm. Both of these are external packages, over which we don't have any control:

// package.json (your project)

 dependencies: {
   "chicken": "^1.2.0",
   "farm": "2.3.0"
 }

and farm package uses a pinned (specific) version of chicken:

// package.json (`farm` package)

 dependencies: {
   "chicken": "1.0.0",
   (...) 
 }

This will result in your project requiring two versions of chicken:

1.0.0 for the farm dependency
^1.2.0 as defined in your project's package.json. This will vary upon a fresh install depending on whatever the latest version after 1.2.0 is - the ^ symbol allows changes that do not modify the left-most non-zero element in the version number. For this particular version range, it means 1.2.0 <= installed version < 2.0.0. To give an example - if v1.2.4 is available at the moment of (fresh) installation of your project, it will be installed. Same for v1.5.8, v1.7.2, v1.9.9, but not for v2.0.0.

Both of these versions will be present in the yarn.lock

The folder structure will look like this:

At this point, you have probably noticed that this isn’t a perfect situation - if a sub-dependency (chicken) has a bug introduced in one of its versions, and dependency that uses it (‘farm’) doesn’t pin the version - it could introduce a bug to your project.

There is a bright side though - if your project requires chicken and uses that same/matching version range, e.g.

 dependencies: {
   "chicken": "^1.2.0",
   (...) 
 }

you won’t install two versions of the chicken package. This decreases the size of your project and prevents potential conflicts between different versions of the same package.

Updating the lockfile

Updating the lockfile is a bit easier to explain. An update can happen in 3 situations: When the dependency is added, removed, or modified.

This can happen in two ways: Automagically or manually. You can trigger an update via yarn CLI (which updates both package.json and yarn.lock) using the following commands:

# adding dependencies

> yarn add PACKAGE-NAME

# removing dependencies

> yarn remove PACKAGE-NAME

# upgrading all dependencies

> yarn upgrade

# upgrading a specific package

> yarn upgrade PACKAGE-NAME

# Adding the `--latest` flag at the end of ‘upgrade’ commands makes yarn ignore the specified version range and install the latest version(s).

If you want to go the manual update route, you modify the contents of package.json and then run yarn install. If yarn doesn't detect any differences between versions in package.json and yarn.lock, it won't install anything new and/or update yarn.lock

Question: You installed project dependencies at some point. Time has passed, maybe a few of your project dependencies released a new version. You have also added some additional dependencies. What will happen if you run yarn?

Answer: Well, yarn will of course install dependencies freshly added to package.json that are not yet installed and/or not present in yarn.lock. But if you have ”dependency_A”:“v1.3.5” already in your yarn.lock, latest version available on NPM is ”dependency_A”:“v.1.4.0” and version range in package.json is ^1.2.0 - will yarn upgrade to the latest version? No, it won’t. v1.3.5 falls into the requirement of being ^1.2.0, so yarn doesn’t see the need of upgrading the package.

Problem - A Second Look

Let’s come back to the original issue for a moment:

An integration of Elements in Storybook (a tool for building an testing UI components) was broken because of a bug present in the Elements dependency, JSV (JSON Schema Viewer)

JSV was using a dependency called JST (JSON Schema Tree), which is effectively a sub-dependency of Elements, with a non-pinned (non-specified) version (^1.1.0) listed in its package.json.

A new version of JST (1.1.2), which falls into the requirement of being ^1.1.0, included a fix that would solve our problem

Even though the JST version could be 1.1.0 or above, Elements itself would still hang for us. Installing dependencies (by running yarn within the Elements directory) didn’t help either. Why?

The answer at this point is actually pretty simple - even though both of the JSV and JST versions are not pinned and should update upon a fresh install, our local yarn.lock file was blocking these updates, having v1.1.0 of JST in itself.

Maybe we should just deploy the lockfile alongside other files?

As explained in the paragraphs above - when present, yarn.lock serves as the main source of information about which versions of packages should be installed. If that’s the case, can we just deploy it with the rest of the package when releasing a new version?

tl;dr(too long, didn’t read) - no you don't (sometimes)

That depends on what your project is:

Is your project an application? Then: Yes
Is your project a library? If so: No

Why should you care about lockfile maintenance for libraries?

There seems to be an agreement as to whether the lockfile should be committed. There's an excellent post on yarnpkg covering this topic (both for applications and libraries) if you want to understand the reasoning behind it.

We'll focus on libraries, such as Elements. Plus, committing the lockfile alongside the application pretty much solves the issue of unwanted updates.

Handling lockfile in libraries

Because only the top-level lockfile is respected (the one form users project root directory), yarn will look into the used library's package.json and install the packages with versions described there. Unless you pin each dependency in your library to an exact version, users' projects might end up having different sub-dependencies depending on the time of installation.

So are we doomed? Kind of. Users will always be the first people to discover a breaking change in a dependency (and hopefully file a bug report). To give you some perspective:

Let’s assume your library has 20 external, sub-dependencies
Each of these sub-dependencies can get a new release anytime
Thousands (potentially) of users install your library each day
Each such installation will fetch the latest sub-dependencies
If any of those sub-dependencies introduce a bug, your users might be affected
The above will happen, unless your development team has a way to regularly test that sub-dependency upgrades don’t break your library

How to keep dependencies up to date?

By now, we’ve established that yarn.lock left alone without any maintenance can introduce confusion about the current state of the project/library, as developers might end up having different versions of dependencies installed locally on their machines.

Let’s take a look at possible solutions for keeping the lockfile up to date.

Dependabot

The first approach we looked at was Dependabot - a well-known tool for bumping dependencies. It checks for possible updates, opens Pull Requests with them, and allow users to review and merge (if you're confident enough with your test suite you can even set auto-merge)

We'd been already using Dependabot for security updates and it served the purpose really well!

Why did we decide not to go with it?

Unfortunately, it misses (at least at the time of writing this article) the ability to have duplicate updates for different allow types. That means you can't have e.g. daily updates for dependencies and weekly updates for devDependencies in the same project. In our case, it was about not being able to daily update to versions that include security-related changes and all of the other changes (features, fixes, major updates) on a weekly basis using the same tool.

Also, as it turned out, later on, having new PR for each dependency update is a bit of a pain.

Renovate

After figuring out that Dependabot does not allow us to do the above, we've decided to look for alternatives. One of the most promising ones (and open-source!) was Renovate.

Even though the basic principle of bumping dependencies is the same, the tool itself seems very powerful and customizable. It has 3 applications (Github, Gitlab, and self-hosted), highly granular settings (you can even set custom rules for auto-merging of PR), and allows opening a PR for a batch of dependencies, instead of for each one.

As we are using GitHub for version control, the supported application for it was an obvious choice. Because our usage was a bit unorthodox - updating only yarn.lock and not package.json in order to have a representation of current users environments and at the same controlling the version ranges - we wanted to test it on the self-hosted version first, to avoid unnecessary PRs created by Renovate, or even worse - unwanted merges.

This is where we hit a wall with Renovate - even though it has a great range of options, we didn't manage to configure it the way we wanted - update ONLY yarn.lock once a week and create a single PR.

Because of that, we decided to not spend more time on publicly available solutions, and handle the lockfile maintenance ourselves.

Your own CI job

You may ask: "Why did you even bother with setting those dependency management systems? Isn't it easier to just run yarn upgrade on everything and call it a day?"

And you would be partially right. The thing is that these systems probably do the exact same thing under the hood but put more attention to the possible failures and corner cases. And just because they are already battle-tested, we decided to check them first. Custom solutions built from scratch, in general, tend to be more fragile than the commercially available ones.

Since neither Dependabot nor Renovate met our needs at a time though, our way out was writing a custom CI job that:

Would bump dependencies for us
Run some basic tests against those changes
Create a PR

Our toolchain was:

CircleCI for CI/CD
git and GitHub for VCS
Yarn as a package manager
Jest for testing
Coffee® for energy

Custom command

### bash

 $ git checkout main
 $ export BRANCH_NAME=feat/lockfile-maintenance-ci-job-$(date +"%m-%d-%Y") && git checkout -b $BRANCH_NAME
 $ yarn upgrade
 $ git add yarn.lock
 $ git commit -m "chore: weekly lockfile maintenance"
 $ git push --set-upstream origin $BRANCH_NAME
 $ BODY='{"head":''"'${BRANCH_NAME}'"'',"base":"main","title":"Weekly lockfile maintenance"}'
     && curl -X POST
     -H "Accept:application/vnd.github.v3+json"
     -u $GIT_AUTHOR_NAME:$GH_TOKEN https://api.github.com/repos/stoplightio/elements/pulls
     -d "$BODY"

The premise of this is:

Get the latest changes from main (no need to git fetch as this is being run in a fresh CI job each time) and create a feature branch with a name corresponding to the lockfile maintenance

 $ git checkout main

 $ export BRANCH_NAME=feat/lockfile-maintenance-ci-job-$(date +"%m-%d-%Y") && git checkout -b $BRANCH_NAME

Upgrade all of the dependencies in yarn.lock according to package.json - this mimics what happens for users upon a fresh install

 $ yarn upgrade

Push changes to remote

 $ git add yarn.lock
 $ git commit -m "chore: weekly lockfile maintenance"
 $ git push --set-upstream origin $BRANCH_NAME

Create a PR using GitHub API (more details in GitHub API Documentation)

 $ BODY='{"head":''"'${BRANCH_NAME}'"'',"base":"main","title":"Weekly lockfile maintenance"}'
     && curl -X POST
       -H "Accept:application/vnd.github.v3+json"
       -u $GIT_AUTHOR_NAME:$GH_TOKEN https://api.github.com/repos/stoplightio/elements/pulls
       -d "$BODY"

Both $GIT_AUTHOR_NAME and $GH_TOKEN are secrets from CircleCI - make sure you don't hard code your credentials in the CI config file and/or the command itself.

CI configuration

workflows:
 version: 2
 test-and-release:
   ...
 perform-lockfile-maintenance:
   triggers:
       - schedule:
           cron: "0 3 * * 1"
           filters:
             branches:
               only:
                 - main
   jobs:
     - lockfile-maintenance

Make sure you define the job as well:

jobs:
lockfile-maintenance:
   docker:
     - image: circleci/node:12
   steps:
     - checkout
     - run:
         command: |
           ### THIS IS A PLACE FOR THE COMMAND FROM PREVIOUS PARAGRAPH

By default, CircleCI runs workflows against all commits from all branches. This is definitely not the behavior we want to have for lockfile maintenance. The desired outcome is that it will run once a week against the main branch. We also don't run any tests at this stage, as the PR created against the main branch will trigger the test-and-release workflow that is being run for each branch and contains a test suite, checks linting, and builds a project to see if there are no crashes.

That's where cron jobs come in handy. We first define that our perform-lockfile-maintenance workflow will be triggered by one (test yours using this online tool) by putting cron job description in the triggers/schedule section. Then we apply an additional filter to it, so it only targets main at any given moment.

As for scheduling, we decided to go with Monday before work (Central European Time), so it is the first thing we look into at the beginning of the week. A contributor opens a PR containing changes made to yarn.lock, approves if it looks right, and merges the change to main.

And that's it! You've just set up your first lockfile maintenance flow!

Possible improvements / aftermath

There are few more things you can do to improve your confidence even more:

If you include examples of usage for your library like us (an integration for GatsbyJS, Angular, CRA) you can bump their dependencies as well. This will assure that your library not only is properly tested internally but doesn't crash when applied to a real-life scenario
Serve an environment containing these integrations for each PR e.g. using Netlify. That will make the whole testing process much quicker as you won't need to check out the changes and run them locally on your own
Strengthen your CI pipeline in general: the more that is covered by your testing suite, the less you will have to check

Summary

So there you go, we have just gone to a dependency hell and came back alive!

I believe that what I have described above will help you encounter fewer issues when developing your library, especially if you don't have a full team dedicated to testing bugs.

But even if I didn't convince you to do a weekly/monthly/whatever dependency bump I hope that this article gave you a strong understanding of the lockfile itself, why it is important when talking about compatibility across different machines, and seeing that lockfile maintenance does not have to be a terrible chore that takes an unreasonable amount of time.

If you feel like this article added some value to your current skill set though, please consider resharing it on your social media and following me on Twitter @m_miaskowski.

If you’d like to learn more about our open-source tool, Elements, which was a basis for this article, visit our website.

Photo by Jude Al-Safadi on Unsplash

7 Approaches I Took While Learning Web Dev From Scratch

mmiask — Tue, 04 May 2021 18:24:20 +0000

This article was originally published on 11sigma.com/blog

What this article is: a starting point for people that seek knowledge in a digestible fashion + my personal summary of the best ways to learn.

What this article is not: a guide on how to plan your self-development path.

Let's code!

Do you want to start programming but are not sure how to do it? Or maybe you already know some things but feel stuck?

A little over a year ago, I switched my career path from biotechnology and became a manual QA. It was a huge unknown for me at that time - a year passed, though, and I'm now programming alongside super-talented engineers as a full-fledged member of a Team Undefined (awesome name, I know) in Stoplight's 11Sigma crew.

During that time, I was (and still am) simultaneously working and learning. It was never easy, I had my ups and downs, but now I can confidently say I improved a lot. If only I knew at the beginning what I know now, it would save me a lot of time and stress.

Along the way, I used various learning materials of different kinds. Not all of them might fit you, but it's good to know about their pros and cons - which is why I prepared this summary.

Theory

This part of learning resources is pretty straightforward - it makes you sit and do a lot of reading. No shortcuts, just learning everything by yourself. It can give you a lot of satisfaction, but sometimes an equal amount of frustration.

Documentation & handbooks - per task, when you need to understand complex stuff quickly: TypeScript, vis.js - network + its examples, Cypress, Yarn, Blueprint, Tailwind CSS.
MDN for the win - a ton of useful information on multiple topics for both beginners and advanced.
Online courses w/o progress verification (Interneting is Hard) - they can give a sense of direction and structured material.
Books: Eloquent Javascript, "Clean Code: A Handbook of Agile Software Craftsmanship" by Robert C. Martin.
Websites / guides - like Getting Git Right by Atlassian or CSS-tricks.

Pros

Reliable sources of knowledge.
Allows us to understand both core concepts and dig deeper into specific details.

Cons

You can get bored very easily if you don't use the gained knowledge parallel to reading.

Real-life problems

Let's say you understand the basics of the topic you have chosen. You are either working in a real team or doing some coding on your own. In both scenarios, at some point, you will face a blocker. That's when you should reach for these:

Dev.to, Medium, StackOverflow - tips & tricks for specific cases.
Read somebody else's code / understand the product you are working on - because it's not necessary to reinvent the wheel, but good to understand how it works.
Write something simple and let people judge it - good criticism is one of the best teachers.

Pros

Quick progress.
Working with code that is used somewhere will give you a real-life experience.
Access to great ideas and solutions that you could potentially spend hours on when doing them by yourself.

Cons

Requires a basic understanding of the topic.
Easy to forget what you've learned if you only copy-paste.

Practice makes perfect

You don't like spending hours reading books or documentation? Then learn by doing!

Interactive tutorials - Flexbox Froggy by Codepip.
Courses that involve practical tasks (Codecademy) - you can use your knowledge/skills in real-life problems. Also Udemy and Coursera.
Review code.

Pros

Difficulty level increases with each lesson (suitable for complete beginners).
It's easier to remember new concepts when you use them.
Good courses provide you with quizzes and pet projects.

Cons

Hard to find any cons, actually.

Verify progress

Because it's good to know how far you have come.

Online tests verifying your skills and knowledge.
Answer interview questions for the desired role (online tests).
Work journal - recap on what you've learned so far. I strongly recommend starting one.

Pros

You can evaluate your progress.
Knowledge and skills that are used repeatedly stick better.
If you don't feel like you make good progress, it's a perfect way to check if that's true. A lot of times, it turns out that you've learned more than you thought you did.

Cons

Focusing too much on quiz points and external expectations can decrease your internal motivation and remove the fun.

Collaborate / teach

Pair programming / mentoring - for when you get stuck or are very unfamiliar with a topic. A good mentor will point you in the right direction, not only in a matter of specific task but also unblock you if you feel unproductive or that your progress is too slow.
Asking coworkers or friends that are programming/programming communities for solutions (async a lot of times).
Do a webinar, teach somebody else - that will force you to learn as well. You will also understand the topic better and more in-depth (Lunch & learn style). I did one on best practices for e2e in Cypress.

Pros

Working with more experienced people will allow you to avoid common mistakes.
A ton of great ideas emerge during discussions.

Cons

Requires a big-time commitment.
Huge impostor syndrome might happen at the beginning when you expose your ideas to other people.

Other

Challenges/problems to solve - HackerRank, Codepen, variety of coding games.
YT videos - for motivation, fun & interesting stuff that stimulate your curiosity (Coding Train).

Pros

It keeps learning fun!

Cons

Getting distracted from the actual learning ;)

Conclusions

Choose the learning method that feels best at the moment. You may have some underlying preferences that are based on your character, but changing context will also add freshness to the entire learning process.
Consolidate your knowledge.
Keep learning. If you don't grow as a programmer, you risk using the same solutions all the time (which will probably work but may not be the best possible).
Don't try to be perfect, nobody is.
Don't over-engineer stuff, do the best you can at the moment.
Don't approach learning as another chore, do what is fun.

If you liked the article, consider following me on Twitter

Photo by Ishan @seefromthesky on Unsplash

DEV Community: mmiask

Career change to IT - things you wish you knew

Yarn.lock: how it works and what you risk without maintaining yarn dependencies — deep dive

Problem

What is a lockfile?

Generating yarn.lock

Updating the lockfile

Problem - A Second Look

Maybe we should just deploy the lockfile alongside other files?

tl;dr(too long, didn’t read) - no you don't (sometimes)

Why should you care about lockfile maintenance for libraries?

Handling lockfile in libraries

How to keep dependencies up to date?

Dependabot

Renovate

Your own CI job

Custom command

CI configuration

Possible improvements / aftermath

Summary

7 Approaches I Took While Learning Web Dev From Scratch

Let's code!

Theory

Real-life problems

Practice makes perfect

Verify progress

Collaborate / teach

Other

Conclusions

Generating `yarn.lock`