DEV Community: Natalia D

What is Quantum Computer: explained in 5 mins

Natalia D — Mon, 21 Aug 2023 17:28:14 +0000

Quantum computers already exist
Quantum Physics 101
Quantum Computer is run on qubits
Quantum Computing
How Quantum Computer can solve RSA?
What are Quantum Computers good for?
Links

Quantum computers already exist

If we think that classical computer is a car, then Quantum Computer is not a faster car, it's a boat. It is meant for other tasks [1].

Several Quantum Computers already exist (e.g. China, Google, IBM). Some quantum computers can work only when cooled down to almost absolute zero, other work given the room temperature, but have huge size. It's not clear yet which implementation will win [2].

Quantum Physics 101

Particles (electrons, etc) are not tiny little balls of stuff [3]. That works for lots of thought experiments, but in fact they are superbizarre little fluctuations in fields that permeate the entire universe.

Here is my own understanding. Particles can be like waves in the water, but when you catch them to perform measurements, for a glimpse of time they turn into a little ball with some labels. When you set them free, they are waves again. They are like some magical shapeshifters.

Quantum Computer is run on qubits

Qubits are like waves. A qubit could really likely to be 0 which means a lower energy wave. A qubit could be more likely to be 1 which means a higher energy wave [1].

Each qubit has a probability of being each. When a quantum computer is working, the probabilities of multiple qubits interact. They add constructively or desctructively.

What is the job of software engineer here? She/he will alter the probabilities while the algorithm is running. Quantum computers don't try all the options. A Quantum Computer more like watches the pond, how the wave interacts, and then helps to find the most likely answer.

Quantum Computing

Mathematically [4], a qubit is a linear combination (a chance being 0 + a chance being a 1):

Similar to classical computers, specialists string together qubits, using construct called gates that can alter the states of qubits into circuits. E.g. we can have a qubit that's at the state of 0. We can use Hadamard gate to put it into superposition between 0 and 1. We can have multiple qubits with multiple gates in a circuit:

For the circuit to be useful, at some point you need to read about its outputs = measure it. When a qubit is measured, it loses its superposition and collapses into just a simple 0 or 1. To make sure the collapsed answer we got is a correct one, quantum gates need to be arranged in a way so that it would amplify the correct answer and cancel all the incorrect ones. This process is called interference.

We can also entangle two qubits so that their states have 50% chance of measuring 00 and 50% of measuring 11, but never a 01 or a 10.

How Quantum Computer can solve RSA?

RSA algorithm depends on the difficulty of finding factors of a very large number. Here is an example of factors: for 15 it's 3 and 5, because they are both primes and 3x5 = 15. It's easy to find the factors of 15 by trial and error, but the difficulty grows exponentially as the size of the number increases.

Peter Shor from Bell labs came up with an algorithm, its main idea is that long strings of numbers contain periodicities - patterns of repetition - which may not be obvious at first sight but may be teased out by mathematical analysis. Periodic patterns can be probed using Fourier analysis, which unpacks complicated wave patterns into their component parts. E.g. the complicated sound wave corresponding to the sound of a group of a musical instruments playing a chord together could be unravelled by Fourier analysis into a wave corresponding to a violin note, or to the cello, etc. A quantum computer using Shor's algorithm could set up a superposition with all the possible periods (waves), and select the one that actually fitted number in question. The periodicity found could then be used by a classical computer to reveal the factors of that number.

What are Quantum Computers good for?

They are good for finding structure in tons of data [1]. As for me, I expect that they will at least help to advance ML (especially not-supervised or semi-supervised learning).

A/B testing: should you use it?

Natalia D — Sat, 19 Aug 2023 17:15:38 +0000

If you decided to read this post, I hope you know what an A/B test is. Probably you either work at a startup or at some Big Company with thousands of users.

If you work at a Big Company, you just have to go with the company flow and do the A/B testing. Here are two things that might be useful for you:

The famous Sample Size Calculator
Look at your KPIs and think about whether you can afford to test something more interesting than the button colours.

If you work for a startup, probably you don't have much traffic. Also the majority of traffic doesn't correspond to real people...So A/B testing doesn't make much sense. In my opinion, the best option in this case would be catch some people in the hall and make videos how they use different versions of your website or app. Try to make the videos that would capture not only what's happening on the screen, but also record user's face and what he/she is saying while working with your product. When you rewatch these videos, there will be a lot of interesting discoveries!

What happens when you type a URL into your browser?

Natalia D — Sat, 22 Apr 2023 09:39:44 +0000

From my experience, if you are a frontend dev being interviewed by a backend dev, which often happens when you plan to join a small company, there is quite high chance that you'll have to answer these two questions:

what happens when you type a URL into your browser
describe security attacks that you know.

Not so long ago I had to refresh my notes about it, and I would like to share them with you. So,

What happens when you type a URL into your browser?

I'll omit the basics that people usually remember and focus only on things that I usually have to re-read

Each device on the internet has a unique IP address. An IP address contains 4 numbered parts, e.g. 203.0.113.0. When you're loading a webpage from a domain your browser has never visited before, your browser may need to make a DNS request to resolve IP address, associated with the domain name. This stack of protocols is used when a DNS request is sent through the Internet:

Then your browser will make an HTTPS request to fetch the webpage. This protocol stack is used when an HTTP request is sent through the Internet:

Nobody wants to do DNS requests too often, so different kinds of cache exist: browser cache, OS cache, router cache, ISP (Internet Service Provider) cache. If the browser cannot find the IP address at any of those cache layers, the DNS server on your corporate network or at your ISP does a recursive DNS lookup. A recursive DNS lookup asks multiple DNS servers around the Internet, which in turn ask more DNS servers for the DNS record until it is found.

User initiates communication (Application layer)

A user on one host sends a message or issues a command that must access a remote host. The application protocol associated with the command or message formats the packet so that it can be handled by the appropriate transport layer protocol, TCP or UDP. Besides HTTPS, there are thousands of other application-layer protocols: SMTP, IMAP, and POP3 for email; XMPP, IRC, ICQ for chat; Telnet, SSH, RDP for remote administration; etc.

When the data contains private information, it needs to be transported securely from the sender to the destination. The Transport Layer Security (TLS) protocol uses algorithms to encrypt the data, while certificate authorities help users trust the encryption.

When a user connects to a webpage, the webpage will send over its SSL certificate which contains the public key necessary to start the secure session. The two computers, the client and the server, then go through a process called a SSL/TLS handshake.

Many website hosting providers and other services will offer TLS/SSL certificates for a fee. These certificates will often be shared amongst many customers. More expensive certificates are available which can be individually registered to particular web properties.

Data encapsulation begins (Transport layer)

Data needs to be broken up into small packets, which are then reassembled at the destination. Two main transport protocols are TCP and UDP.

If TCP determines that IP packets are lost, duplicated, or out of sequence, it’ll request retransmission of missed data, correct out-of-order data. UDP offers no guarantees of delivery, ordering, or duplicate protection. Its lack of retransmission delays makes it suitable for real-time applications such as Voice over IP (VoIP), online games, live video streaming, video conferencing. All of these require transferring data as fast as possible, even if it results in a glitch or two. Knowing that TCP is primarily for functions where accuracy is prioritized over timeliness, this protocol is common for Internet essentials like sending emails, sharing files, text messaging and accessing web pages.

Both TCP and UDP divide the data received from the application layer into segments and attach a header to each segment. Both TCP and UDP headers contain the sending and receiving ports and checksum. The checksum data is used to determine whether data has transferred without error. TCP headers also include segment ordering information (don't try to memorise this table, it's just for illustrative purposes):

When TCP wants to establish connection, it sends a segment called a SYN to the peer TCP protocol running on the receiving host. The receiving TCP returns a segment called an ACK to acknowledge the successful receipt of the segment. The sending TCP sends another ACK segment, then proceeds to send the data. This exchange of control information is referred to as a three-way handshake. UDP does not use the three-way handshake.

Data is sent (Network layer)

In the majority of cases, this is dominated by Internet Protocol (IP). The Internet Protocol (IP) is the protocol that describes how to route messages from one computer to another computer on the network. Each message is split up into packets, and the packets hop from router to router on the way to their destination.

Computers send the first packet to the nearest router. You probably have a router in your home right now, and that's the first stop for your current computer's packets. When the router receives a packet, it looks at its IP header. The most important field is the destination IP address, which tells the router where the packet wants to end up.

The router has multiple paths it could send a packet along, and its goal is to send the packet to a router that's closer to its final destination. How does it decide? The router has a forwarding table that helps it pick the next path based on the destination IP address. That table does not have a row for every possible IP address; there are too many of them. Instead, the table has rows for IP address prefixes:

IP addresses are hierarchical. When two IP addresses start with the same prefix, that often means they're on the same large network. Router forwarding tables take advantage of that fact so that they can store far less information.

Once the router locates the most specific row in the table for the destination IP address, it sends the packet along that path. If all goes well, the packet should eventually arrive at a router that knows exactly where to send it.

Link layer, Physical layer

Ok, but who transmits the bits? To do that work, The network layer logic in a host or router must hand off the packet to the data link layer protocols, which, in turn, ask the physical layer to actually send the data.

Data-link layer protocols are classified as LAN protocols, WAN protocols, or protocols that can be used for both LANs and WANs. LANs connect devices within a limited area, such as an office, while WANs connect devices that are spread across a large area, such as an entire country or even the world.

Data arrives at the server

On the opposite side of the communication channel is the server, which serves the document as requested by the client. A server appears as only a single machine virtually; but it may actually be a collection of servers sharing the load (load balancing), or a complex piece of software interrogating other computers (like cache, a DB server, or e-commerce servers), totally or partially generating the document on demand.

A server is not necessarily a single machine, but several server software instances can be hosted on the same machine.

But how a remote computer can communicate with a program running on your computer? There are not enough IP addresses for that. So people came up with an idea of "ports". Basically, port is a number in a range from 0 to 65535, associated with a specific process or service. An app can open a port - to broadcast a message like if "a message comes for port 5190, it's for me".

Ports are software-based and managed by a computer's operating system. During data transfer, only a transport protocol such as TCP or UDP can indicate which port a packet should go to. TCP and UDP headers have a section for indicating port numbers.

Extra details you might want to know

IP addresses

One way to find out your computer's IP address is by searching Google for "IP address". Google knows your IP address, since your computer sends a message to the Google computers as soon as it loads google.com.

Your IP address might be different tomorrow than it is today. Each ISP has a range of addresses they can assign, and they might give you a different one of those addresses each time they see your computer pop up on the network. That's called a dynamic IP address.

Switching to a different Wi-Fi network will definitely give you a new IP address, since each Wi-Fi provider has its own range of addresses that it can give out. Computers that act as servers, like the computers that power Google.com, often have static IP addresses. That makes it easier for computers to quickly send search requests to the Google servers.

There are actually two versions of the Internet Protocol in use today:

IPv4, the first version ever used on the Internet
IPv6, a backwards-compatible successor.

Back when the Internet protocols were first invented, the creators didn't anticipate how popular it would become and that there would eventually be so many devices wanting to connect to the Internet. When it became obvious in the 1990s that the IPv4 addresses were running out, the IPv6 protocol was proposed with a much longer addressing scheme.

MAC addresses

A MAC address and an IP address each identify network devices, but they do the job at different levels. MAC address is assigned by the manufacturer of the hardware interface, while the IP address is assigned by the network administrator or Internet Service Provider (ISP).

The MAC address identifies the device locally, while the IP address identifies it globally. The MAC address is only relevant to the Local Area Network (LAN) to which it's connected and isn't part of the data stream when the packets leave the device's network.

Proxies

Between the Web browser and the server, numerous computers and machines relay the HTTP messages. Those operating at the application layers are generally called proxies. These can be transparent, forwarding on the requests they receive without altering them in any way, or non-transparent, in which case they will change the request in some way before passing it along to the server. Proxies may perform numerous functions:

caching (the cache can be public or private, like the browser cache)
filtering (like an antivirus scan or parental controls)
load balancing (to allow multiple servers to serve different requests)
authentication (to control access to different resources)
logging (allowing the storage of historical information)

Ports and Firewalls

Some attackers try to send malicious traffic to random ports in the hopes that those ports have been left "open," meaning they are able to receive traffic. This action is somewhat like a car thief walking down the street and trying the doors of parked vehicles, hoping one of them is unlocked.

A firewall is a security system that blocks or allows network traffic based on a set of security rules. Firewalls usually sit between a trusted network and an untrusted network; often the untrusted network is the Internet.

Properly configured firewalls block traffic to all ports by default except for a few predetermined ports known to be in common use. For instance, a corporate firewall could only leave open ports 25 (email), 80 (web traffic), 443 (web traffic), and a few others, allowing internal employees to use these essential services, then block the rest of the 65,000+ ports.

If you are still reading, I'll talk about security and encryption in another post. This one got too lengthy to digest already!

The links I used

cloudflare, oracle, mozilla, hypertecsp, sciencefirect, guru99, simplilearn, wowza, imperva, quora, aws, trueconf, khanacademy - internet protocols, khanacademy - ip-v4-v6-addresses

My thoughts about mentoring junior developers

Natalia D — Fri, 03 Mar 2023 10:17:45 +0000

While I haven't personally mentored anyone into a mid-level developer, I've had the opportunity to work at various companies and observe both successful and unsuccessful mentoring approaches. These experiences have given me valuable insights that I want to share with you.

Ticket size & watching deadlines

In a perfect world, every ticket would be designed in a way that a mid-level developer could complete it within three days. However, in reality, refinement meetings are often necessary to clarify the scope and requirements of a ticket. It's not uncommon for developers to encounter unexpected challenges when they start working on a ticket.

It's important to teach junior dev not to take a ticket for granted and to thoroughly evaluate its worth before starting work on it. Here are some key things to consider:

Is the ticket worth doing at all? Sometimes, there may be a misunderstanding or miscommunication, and the ticket may not actually need to be completed. Sometimes devs just need to update the docs or operation management can change the business process.
Is the ticket a duplicate of an existing issue?
Does the ticket have the correct label? A ticket labeled as "frontend" may actually involve backend work and not have other related to it tickets labeled "backend".
Is the suggested UX good enough? As a developer, you may have ideas for how to improve the user experience or implement the feature more efficiently.
Are there any other clarifications needed? Sometimes, additional information may be required from the client, devops, or another team member.

For junior developers, it's important to know when to seek help and when to try to solve a problem independently. In order to strike this balance effectively, I would recommend a few key strategies.

First, if a junior developer is facing a problem they can't resolve on their own, they should leave a "question comment" on the relevant ticket or bring it up in a common chat. This can help ensure that others on the team are aware of the issue and can offer assistance if needed.

However, it's also important for junior developers to try to solve the problem independently before asking for help. A good rule of thumb is to spend at least two hours working on the problem before seeking assistance. This can help ensure that the developer has thoroughly explored all possible solutions and is not simply relying on others to solve the problem for them.

Daily standups can also be a valuable opportunity for developers to discuss any challenges they may be facing and seek input from their peers. If a senior developer feels that a junior developer is facing a challenge that may be too difficult for them to handle quickly enough, they may suggest a pair programming call with screen sharing. In addition, senior developers may also review the problem ticket and offer their own opinion or suggestions in the comments.

Code Review

In my experience with various activities, including fiction writing, drawing, and coding, one thing remains consistent - after working on something for a period of time, it becomes easy to lose a "fresh glance" on the project. This is simply a biological response in the brain to conserve resources. While taking a break from the work for several days can help regain a fresh perspective, a quicker and more effective solution is to ask someone else to review your work.

This is where code review comes into play. While some may see code review as a way for senior developers to mentor junior developers, I see it as an opportunity for knowledge sharing. Developers can ask questions about the code or project, which can lead to valuable insights and learning opportunities. Even a junior developer who may not feel confident in contributing to the code itself can still test the PR code and provide valuable feedback by identifying and reporting a bug. Comments with attached screenshots can be especially helpful in this case.

In this way, code review becomes a collaborative effort that allows all team members to learn from each other and improve their skills.

When code review for a pull request (PR) starts to take up too much time, it's not uncommon for the developer who is most responsible for meeting sprint goals to step in and have the final say in any ongoing PR discussions.

Meetings

Organizing weekly 15-minute one-on-one meetings with each junior developer can be an effective way to monitor their progress and ensure that they are growing and developing in their role.

During these meetings, it's a good idea to ask the developer about any new knowledge or skills they have acquired over the past week. This can help you understand their strengths and weaknesses, as well as identify areas where they may need additional support or guidance.

Additionally, it's important to ask how the developer felt about the tickets they worked on during the week. This can help identify any areas where they may be struggling, as well as celebrate their successes and acknowledge their hard work.

Conclusion

Mentoring junior developers can be compared to raising children - it requires patience, guidance, and encouragement.

As a mentor, it's important to give your mentees space and time to learn and grow on their own. This may mean stepping back and allowing them to tackle a problem independently, even if it means they make mistakes along the way.

Just like with children, it's important to encourage junior developers not to compare themselves to others. Instead, they should focus on their own progress and growth over time. By tracking their own progress and setting achievable goals, they can stay motivated and build the skills they need to succeed.

The difference is that meeting sprint deadlines is crucial. If you're unsure about a developer's level of expertise, it's best to assign tasks with lower priority. That way, if the deadline is missed, it won't have a significant impact. However, if a ticket with an upcoming deadline needs to be completed, consider pairing up the junior developer with another experienced developer.

The junior devs should learn how to keep other devs in the loop efficiently but you also have to approximately track their time by yourself (based on information you receive at daily standups) and act if the ticket takes too long.

Effective Frontend Test Strategy and some notes about Cypress

Natalia D — Mon, 27 Feb 2023 17:38:46 +0000

As a developer, I frequently find myself working on projects that lack a dedicated testing team or resources. As someone who has spent a noticeable amount of time performing manual testing, fixing frontend unit tests, and writing end-to-end Cypress tests to ensure the highest code quality possible under high business pressure, I have developed some strong opinions about organizing testing strategies in such circumstances.

Time for testing

From my experience, usually there is no time specifically reserved for tech debt or automated testing. Generally, every day the business is like:

There is some time, though, during which developers have to do regression before they show the code to the client/product manager. However, this process can be time-consuming and overwhelming, taking up several hours or even an entire day!

I'll be talking about the case when you are often asked to do manual regression and you don't have much time to make your life easier, only little time slots here and there (e.g. when the board is not so busy, you've made some PR's and have something to do while you wait for other devs to review them).

Test pyramid

Let's talk about tests for React or Angular. To spend less time developing them you will want to have more presentation (dumb) components and less smart components.

If you have a lot of complicated web forms it makes sense to use Storybook and write a storybook test for each such form realised as a dumb component.

When you have written a bunch of tests for smart React/Angular components, you might find out that it's hard to maintain them, because each test has a lot of dependencies and json mocks that you have to update quite often. Given that bringing a test back to life might take 2-3 hours (and I haven't even included in this estimation time for other devs to code review it) my recommendation is to save efforts and skip tests for most of the features. Think of the most critical parts that your app has instead. E.g. if you do payment operations through your app it makes sense to write as much tests as you can to make sure that money goes where it's supposed to! In other words, before you create a test for a smart component/service/module think: "is it worth supporting such test? How often it's used by other modules? How critical is the feature?"

Some developers might argue that you can make make your code more testable: e.g. extract some pure functions into separate modules that can be easily tested by using simple Jest unit tests. The thing is, when you join a large legacy project, in most cases the team won't be willing to do any radical change to existing code unless you really sell the idea that you can make everybody's life better by refactoring this piece of code and you can do it quickly enough.

I've found that writing end-to-end (e2e) "happy scenario" tests for the most critical scenarios is the most effective solution. E.g. you can write tests for cases when user successfully

logs in,
signs up,
restores password.

My preferred tool for e2e testing is Cypress.

UPD. Recently I've tried "@testing-library/react"+"@testing-library/user-event" and it turned out not so bad! Don't have resources to write about it now, but hopefully will make another post about React unit tests in the nearest future. TLDR is I would replace some bits of Cypress with unit tests with these libraries.

Cypress

Cypress has well maintained documentation, they even have a section about best practices which I highly recommend to read. I would just like to mention several things.

Selecting elements for test
You can use data-qa attributes in your HTML and select elements in test like this:

cy.get('[qa="company-name"]').type('test-company');

Long tests issues

Cypress doesn't like tests that take too much time to run. If a test takes too much time, Cypress can throw one of the following errors (there are GitHub issues about that - first, second). I've mentioned some ways to deal with it in this post.

Wait

There is only one situation I can imagine where you might want to write something like

wait(100);

You are probably in a hurry and you plan to run Cypress tests only locally. Such tests will be very flaky if you run them as a part of CI/CD pipeline!

Waiting for a HTTP request to finish is covered well enough in the docs. Let's look at a couple of other useful approaches that I've also been using.

First of all, you can ask Cypress to wait until some thing is visible:

cy.get('[data-qa="sign-out-btn"]').should("be.visible");

Second of all, you can use constructions like this (in tests or cypress commands):

cy.request({ ... }).then((response) => {...});
cy.then(()=> { cy.request({ ... }).then((response) => {...});

Localstorage and setting env variables

cy.request({...}).then((response) => {
  window.localStorage.setItem('token', response.body.token);
  Cypress.env('userId', response.body.data.id);
});

How to reset database after tests

If you have mongo, I have written a post about it.

If you have mysql - this might help. I haven't tried it, so not a legal advice! :)

The why and how to reset the database after each test run (Cypress, Mongo 5.0.11, React example)

Natalia D — Mon, 27 Feb 2023 07:03:16 +0000

As a developer, I frequently find myself working on projects that lack a dedicated testing team or resources. To minimize the amount of manual regression testing I have to do before deployment to staging, I've found that writing end-to-end (e2e) tests for the most critical scenarios is the most effective solution. My preferred tool for e2e testing is Cypress.

However, a question that often arises is: how can we ensure that each of these tests can be

run multiple times;
run independently.

That's why we need a way to reset the database after each test.

My approach is based on Gleb Bahmutov's post. The difference is that this post is for mongo 5.0.11, and his post is for older mongo versions.

Another difference from Gleb's post is that all the tests I'll talk about will be about app entry flows: Sign up, Log in and Logged in tests.

Let's consider a following example (repository is here)

Btw Cypress recommends to clean up database before each test. I wrote "after" in the post header only for SEO purposes.

Sign up Test

You have written a test for sign up scenario. First time test passed. You try to run the test again and get an error "Email address already exists":

You might want to skip what I've written below and just take a look to branch feature-cypress-reset in the repository instead. There you will see a signup test that can be run an indefinite amount of times.

File cypress/support/db.js helps to connect to the database.

If you are a frontend, you don't have to worry much about refactoring this file. Devops that will convert your PR into working version will change uri and client.db("rests") to something that works for your company. They will set up a separate testing environment that includes a test database. This test environment will allow run tests without the fear of damaging or corrupting important data in the production system.

The most interesting file to look at is cypress.config.ts. I spent some time trying to understand why await users.remove({}) doesn't work before I realised that it was deprecated in newer mongo and replaced with deleteMany!

Then you just add a cypress command for cleanup:

And use it in the test like this:

Login Test

When I've started to write tests for logged in users, I've discovered that Cypress doesn't like tests that take too much time to run. If test takes too much time, Cypress can throw one of the following errors (there are GitHub issues about that - first, second):

abort test running with some message about web sockets;
abort typing into form fields, e.g. instead of typing "somebody@gmail.com" type "somebody@gma".

The best thing is you can do for such tests is to use shortcuts. E.g. instead if imitating user login by typing values into forms, send a POST request using Cypress command. Or change a database record directly using Cypress task.
Also, this setting might help:

Cypress suggests to do session switching inside tests but we don't use it. Sadly I don't remember any details about why it wasn't useful for our current project.

Let's say you want to test login flow. You'll need to have a user in your DB before the test!

You might wonder how I got an encrypted version for password that I decided to use. I temporarily put in app.js the following code:

Logged in Test

Let's say we want to test something for a logged in user. We will use a shortcut for logging in, as it is advised in Cypress docs:

Then we'll use it like this:

Conclusion

I would like to tell more about Cypress but this post is too long already :( The only thing that I would like to mention (again) is that if you are a dev who worries about QA but doesn't have much time for it Cypress is definitely the best choice based on the ratio how much time you spend writing/maintaining tests vs how much functionality you get tested.

Uploading Source Maps for Angular

Natalia D — Wed, 08 Feb 2023 22:00:34 +0000

Tired of sifting through confusing Sentry error messages? You're not alone.

I recently struggled with uploading sourcemaps to Sentry, but after some head-banging and persistence, I've gained valuable experience that I want to share with you.

UPD. This post has been updated after a timely follow up call with Sentry team.

The task was to organise uploading sourcemaps process for our Angular application. However, these steps could also be useful for those working with React applications.

To get started, I organized a testing environment by creating a simple app using the ng new command. I then connected my personal Sentry account and added the Sentry.captureException("...") function to the click event of a button. To easily deploy the app, I utilized the user-friendly service Netlify (which I highly recommend).

While I don't want to go into too much detail about setting the release version, I understand that some readers may be interested in exploring different approaches. If that's the case, I encourage you to take a look at this discussion.

Settings in angular.json should be like this:

"sourceMap": {
  "scripts": true,
  "hidden": true,
}

To simplify the process, I have compiled all the necessary commands into a bash script:

version_str=$(grep -Eo '\"version\"\:\s\"[0-9]\.[0-9]\.[0-9]+\"' package.json)
version=$(echo $version_str | grep -Eo '[0-9]\.[0-9]\.[0-9]+')
version="$version-production"
sentry-cli releases new "$version"
sentry-cli releases set-commits --auto "$version"
sentry-cli releases files "$version" upload-sourcemaps docs/
sentry-cli releases new "$version" --finalize

Also I have put all necessary auth credentials to .sentryclirc file like this:

[defaults]
org = <my-organisation-name>
project = <my-project-name>

[auth]
token=<my-token>

After doing all this you might still see *.map files in your build folder, but don't worry, they won't be visible in production.

My machine-learning pet project. Part 5. Exploring epochs and batches

Natalia D — Wed, 10 Aug 2022 21:43:18 +0000

My pet-project is about food recognition. More info here.

The last thing I did was to feed a part of my dataset to Retinanet. This part consisted of 71 images. Retinanet basically complained that my dataset was too small for training.

It said, "Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches".

I looked up epochs and batch sizes. One training epoch means that the learning algorithm has made one pass through the training dataset, where examples were separated into randomly selected “batch size” groups. But I've just remembered that 1 picture is worth a 1000 words, so I've drawn a picture for you, check this out:

steps_per_epoch (or steps) is number of batches to run per training epoch. So if we have batch_size = 32, with 71 images we can break dataset into 2 batches (=2 steps). 64 images will be used for training, and 7 images will be redundant.

I excluded 7 images from my little dataset and ran command:

keras_retinanet/bin/train.py csv dataset/annotations3.csv dataset/labels.csv --epochs 50 --steps 2

but I got a message:

Failed to load the native TensorFlow runtime.

It means I forgot to load the environment where Tensorflow had been installed. I ran command:

conda activate mlp

repeated the train.py command, and ran into a different error message:

train.py: error: unrecognized arguments: --epochs 50 --steps 2

After reading this issue I realised that I needed to change my command to this:

keras_retinanet/bin/train.py --epochs=50 --steps=2 csv dataset/annotations3.csv dataset/labels.csv

After that Retinanet started working and finished its calculations without errors:

Next time I'll try to plan my real dataset size, I need to find a compromise between what I can label and what amount needs to be labelled. I'll research the methods for making the most of small sized datasets.

My machine-learning pet project. Part 4. Feeding the dataset to Retinanet

Natalia D — Sun, 26 Jun 2022 10:02:51 +0000

My pet-project is about food recognition. More info here.

Meme from csteinmetz1 on reddit

Today I want to feed my dataset to Retinanet using a csv format. For now, I am going to use only 2 labels and about 80 images (half labelled Salad, half labelled Pastry) just to try Retinanet, to get an idea how it works.

According to retinanet readme, annotations file should look like this:

pic-037.jpg,80,20,500,120,Salad
pic-025.jpg,520,250,1152,953,Pastry
pic-with-nothing.jpg,,,,,
pic-004.jpg,0,0,1600,1113,Salad
...

But Scrapy made me a csv file like this:

Apple Cake,"Some apple cake description...",https://www.some-recipes-website.ru/binfiles/images/20200109/m12b509e.jpg,"[{'url': 'https://www.some-recipes-website.ru/binfiles/images/20200109/m12b509e.jpg', 'path': 'full/ae00a78059ad08506aa4767ed925bef5dccabf63.jpg', 'checksum': '55088c744a564af5ed8d4e5ea6478d20', 'status': 'downloaded'}]"

So the first thing I did was to write a small Python script which converted Scrapy csv to a Retinanet csv.

I spent some time wondering what are the requirements for the images in the dataset. I found one opinion here:

No need to rescale your image, because RetinaNet resize the image to get the appropriate size by default.
In general I would advise to keep your settings the same during inference and training.

Ok, so I won't do anything about images resolution - for now.

Then I needed to install Retinanet on my computer. I cloned the repo and went to its root.

I have macOS Monterey, arm 64. The biggest challenge was to install tensorflow, which was one of Retinanet dependencies.

I followed this article. Installed conda, then

conda create --name mlp python=3.8
conda activate mlp
conda install -c apple tensorflow-deps
pip install tensorflow-macos

I copied the dataset files (csv and images) to the root of retinanet repo. Made labels.csv:

Salad,0
Pastry,1

Then I followed retinanet Readme:

pip install . --user
python setup.py build_ext --inplace

Tried to run training script

keras_retinanet/bin/train.py csv dataset/annotations.csv dataset/labels.csv

but Retinanet said:

ValueError: invalid CSV annotations file: dataset/annotations.csv: line 1: malformed x1: invalid literal for int() with base 10: '43.0'

I fixed the annotations file so it contained integer coordinates instead of float ones and repeated the attempt.

Retinanet worked for about 3 mins and said:

Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least steps_per_epoch * epochs batches (in this case, 2 batches). You may need to use the repeat() function when building your dataset.

Well, that wasn't unexpected (because I had only about 80 images in my dataset, remember?). I don't know what epochs are at the moment, but I will find it out while writing a next post.

My machine-learning pet project. Part 3. Brushing up and labelling my dataset.

Natalia D — Sun, 12 Jun 2022 18:25:27 +0000

My pet-project is about food recognition. More info here.

Scrapy made me a folder with images and a .csv with rows like:

Apple Cake,"Some apple cake description...",https://www.some-recipes-website.ru/binfiles/images/20200109/m12b509e.jpg,"[{'url': 'https://www.some-recipes-website.ru/binfiles/images/20200109/m12b509e.jpg', 'path': 'full/ae00a78059ad08506aa4767ed925bef5dccabf63.jpg', 'checksum': '55088c744a564af5ed8d4e5ea6478d20', 'status': 'downloaded'}]"

Now, I needed to create .csv files like this:

pic-037.jpg,80,20,500,120,risotto
pic-025.jpg,520,250,1152,953,risotto
pic-with-nothing.jpg,,,,,
pic-004.jpg,0,0,1600,1113,beans
...

To do that, I googled smth like "best machine learning label tools 2022" and found Label-studio. I followed these steps from their docs:

python3 -m venv env
source env/bin/activate
python -m pip install label-studio

But I couldn't launch label-studio until I did some of these things:

pip install wheel
pip install spacy
pip install cymem
brew install postgresql

(link1, link2 might be helpful)

I didn't have any problems importing my data to Label-studio. The only setting I had to do was to add my labeling interface code:

<View>
  <Image name="image" value="$image"/>
  <Choices name="choice" toName="image" showInLine="true">
    <Choice value="Salad" background="blue"/>
    <Choice value="Soup" background="green" />
    <Choice value="Pastry" background="orange" />
    <Choice value="Nothing" background="orange" />
  </Choices>
  <RectangleLabels name="label" toName="image">
    <Label value="Salad" background="green"/>
    <Label value="Soup" background="blue"/>
    <Label value="Pastry" background="orange"/>
    <Label value="Nothing" background="black"/>
  </RectangleLabels>
</View>

After that, labeling interface looked like this:

I had label "Nothing" for confusing images that I decided to exclude from the dataset:

After I finished labeling a portion of images, I chose to export them in .csv format. I got a file with rows like this:

/data/upload/1/83d8ce57-7478f053119ca2a85c4932870ef3e1833eb3eeb5.jpg,14,Pastry,"[{""x"": 13.157894736842104, ""y"": 6.5625, ""width"": 41.578947368421055, ""height"": 74.375, ""rotation"": 0, ""rectanglelabels"": [""Pastry""], ""original_width"": 570, ""original_height"": 320}]",1,20,2022-06-06T07:11:56.377261Z,2022-06-10T21:02:35.758255Z,1209.599

I was surprised when I saw x, y, width and height. Then I've read in the docs that "Image annotations exported in JSON format use percentages of overall image size, not pixels, to describe the size and location of the bounding boxes."

I wrote a small python script to check exported regions:

from PIL import Image
img = Image.open('../../images/full/7478f053119ca2a85c4932870ef3e1833eb3eeb5.jpg')

x = 13.157894736842104
y = 6.5625
width = 41.578947368421055
height = 74.375
original_width = 570
original_height = 320

pixel_x = x / 100.0 * original_width
pixel_y = y / 100.0 * original_height
pixel_width = width / 100.0 * original_width
pixel_height = height / 100.0 * original_height

left = pixel_x
upper = pixel_y
right = pixel_x + pixel_width
lower = pixel_y + pixel_height

box = (left, upper, right, lower)
region = img.crop(box)
region.show()

When I launched the script, it showed me the correct region cropped out of the original image:

So now I understood how to get pixel annotations if I need them.

Next post is going to be about trying to feed the dataset to Retinanet. I am going to use only 48 images scraped so far just to see what format of input it really needs.

My machine-learning pet project. Part 2. Preparing my dataset

Natalia D — Thu, 02 Jun 2022 22:17:36 +0000

My pet-project is about food recognition. More info here.

Data sets in tutorials vs data sets in the wild. From Towards AI on Twitter

First thing that came to my mind was to scrape stuff from some instagram account. Have you seen how many recipes are there??? Millions. And they have descriptions, from which I could extract labels. I thought it would be ez. I managed to scrape about 10 posts using this:

instaloader profile nytcooking --no-videos --no-metadata-json --slide 1 --post-filter='date_utc >= datetime(2012, 5, 21)' --sanitize-paths

So far so good. But then I tried to scrape a bit more, e.g. posts for 3 months, and started getting 429 Too many requests errors. Creating new instagram profiles didn't help. How could I, alone, beat an army of well paid developers? I needed another approach.

I chose one of the recipe websites that I used to visit. It has good photos and descriptions and it's easy to scrape. I picked Scrapy to do the job. It's actively supported (last commit - 5 days ago), has good documentation and a readable code.

I saved one sample webpage to my desktop and launched scrapy shell:

scrapy shell ../Desktop/test.html

It helped me to prepare a bunch of selectors like this:

recipe.xpath('./p[@class="material-anons__des"]//text()').get()

Then I created a project template using command like this:

scrapy startproject myproject [project_dir]

I took a look at their example repo at github.

The only thing I have struggled for some time was item pipelines. At one point I copied some stuff from stackoverflow to settings.py. And scrapy started to complain, smth like file A has an error now. I went to scrapy github and found a nice little comment in file A about what really should be in the settings file now. So that's what I added to settings.py:

FILES_STORE = './csv'
IMAGES_STORE = './images'
ITEM_PIPELINES = {
    'scrapy.pipelines.images.ImagesPipeline': 1
}
FEEDS={
    './csv/items-%(batch_id)d': {
        'format': 'csv'
    }
}

These settings allowed me to download images and make a *.csv file with parsed titles,descriptions and image paths urls without writing any extra lines of code. In my opinion, Scrapy is a very powerful instrument, so I totally recommend to use it when you want to scrape some data (not from instagram).

The next post is going to be about brushing up scraped data and labelling it.

My machine learning pet-project. Part 1

Natalia D — Wed, 01 Jun 2022 20:06:13 +0000

Machine learning has sneaked into our lives. Every night I tell Alexa "turn the lights off". Every day my phone successfully performs face recognition before I pay with it. Given the tremendous improvement of this field I decided to find out how hard nowadays it is for an ordinary frontend engineer to apply machine learning for some task.

A long, long ago my friend and me used to share photos of what we had eaten during the day. We tried to eat less and make healthier food choices. So I decided to try to make something (a website? some bot? to be refined later), which helps the user to tweet about his recent food choices. Show it the food photos and it creates a tweet code with suitable emojis which you can share. Thanks to Wordle for the inspiration :)

Of course, being a former researcher the first thing I did was to google what other people have done about food recognition. The most interesting article I found so far is here

I decided to kind of repeat the author's way, but give it a bit of my own tweak. E.g. most probably I'll use 3 labels: pastry, salad and soup.

Next post is going to be about how I started to gather the dataset for my project.

DEV Community: Natalia D

What is Quantum Computer: explained in 5 mins

Quantum computers already exist

Quantum Physics 101

Quantum Computer is run on qubits

Quantum Computing

How Quantum Computer can solve RSA?

What are Quantum Computers good for?

Links

A/B testing: should you use it?

What happens when you type a URL into your browser?

TOC

Browser looks up IP address for the domain

User initiates communication (Application layer)

Data encapsulation begins (Transport layer)

Data is sent (Network layer)

Link layer, Physical layer

Data arrives at the server

Extra details you might want to know

The links I used

My thoughts about mentoring junior developers

Ticket size & watching deadlines

Code Review

Meetings

Conclusion

Effective Frontend Test Strategy and some notes about Cypress

Time for testing

Test pyramid

Cypress

The why and how to reset the database after each test run (Cypress, Mongo 5.0.11, React example)

Sign up Test

Login Test

Logged in Test

Conclusion

Uploading Source Maps for Angular

My machine-learning pet project. Part 5. Exploring epochs and batches

My machine-learning pet project. Part 4. Feeding the dataset to Retinanet

Meme from csteinmetz1 on reddit

My machine-learning pet project. Part 3. Brushing up and labelling my dataset.

My machine-learning pet project. Part 2. Preparing my dataset

Data sets in tutorials vs data sets in the wild. From Towards AI on Twitter

My machine learning pet-project. Part 1