DEV Community: Saurabh Goyal

[Level-Up-Series] Program in Assembly

Saurabh Goyal — Tue, 04 Jun 2019 10:56:07 +0000

This is one of the experiments to be done under Programmer Level-up-series, checkout this post to know more details.

As per original experiment, the idea was to write a simple Assembly program but while doing that I tried to get a comprehensive grasp of the language and hence started with a bit of history and the very basics. After reading all this, I have much more to share on fundamentals than on Assembly language itself. Let’s start-

Assembler | Compiler | Interpreter

We’ll discuss what are these, why they were needed, how things started, what should I answer when people ask me if Python is interpreted or compiled :p.

Early in the time, Different processors/architectures (I’ll call them environments from now on) required their own set of machine code (0s and 1s) instructions to perform same operations because of the way their hardware was wired. The case is same even today, any type of program you want to run, at the final layer where any non-machine-code program is converted to machine code instructions, that translator is environment specific.
Putting that aside, writing machine code instructions even for one specific environment was cumbersome since they are not easily understandable by humans. Of course people didn’t remember machine codes for each command they wanted to perform, they had something called Opcode tables which was a mapping between human-readable commands such as Load A, 10 to machine code command such as 00111010, they wrote their programs in opcodes and then by looking up the table, they converted that to machine code instructions.
Obviously enough, a need was felt of a program which can automate the above step by taking opcoded (human-readable commands) program and generating machine code program from it, a Translator. One such translator was created and was called Assembler since it translated the first structured version of the opcode instructions which were called Assembly language. A point to note here is that it worked exactly like a mapping , i.e. it took one instruction of opcode and converted to one instruction of machine code, there was no complex translation. Such type of naive translator is called Interpreter. So an Assembler is nothing but an Interpreter for Assembly language.
As it started speeding up the programming, community wanted to have even higher levels of abstraction with which complex set of machine code instructions could be written as much simpler high level instructions without programmer focusing on low level stuff such as where and how of memory allocation, this is when Compilers were born. Compilation was not a direct mapping of instruction in one language to instruction in another, rather a sophisticated and complex process of reading the whole program first then converting that into machine code program with all kinds of optimizations, this extended the possibilities a lot.
The main challenge with this though, was that it was a common and true perception that such a compiled program can not achieve the same level of performance as that of a hand written Assembly program, mainly because of the memory allocation logic and optimizations that humans can think of depending on the situation. Obviously that performance came with the trade-off of huge entry barrier to programming. Any way, one such language A0 was created which was compiled into machine code instead of being translated one line at a time, though it didn’t become much popular because of the earlier mentioned perception. Later IBM created FORTRAN on the same principle which was widely accepted.
Now there were languages which had their compilers and were easy to code in but there was still an issue - different compilers had to be written for different environments. A solution was proposed which included both compiler and interpreter for translation. This was the introduction to Write Once, Run Everywhere concept. There were 2 steps-
- The high level source code would be compiled into some kind of intermediate-code. This was different from normal compilation in which source code is converted to machine code.
- That intermediate-code will be interpreted by an interpreter written specifically for the environment in which program has to be run.
Point to note here is that, this solution also needed writing environment specific interpreters but was still preferred to writing environment specific compilers because of two reasons-
- Writing environment specific compiler for high level source code was much more cumbersome than writing environment specific interpreters for much lower level intermediate-code.
- Bytecode provided an ease of sharing the program without sharing the source code which provided some kind of abstraction.

COBOL was the first language to start with this concept and it made writing and sharing programs extremely simple.

In interpretation process, a major optimization method was introduced, it’s called JIT (Just-In-Time) Complilation. What it does is that when interpreter is going line by line and translating to machine code, it also feeds the lines to a module called JIT-compiler which starts finding duplicate lines, complex set of lines and other types of instructions which can be optimized on and keeps storing their machine code translations for future usage. With the collected insight and data, it saves repeated translation of multiple lines of intermediate-code to machine code by interpreter. All this happens on the fly as and when lines come, hence Just-In-Time. It creates a significant positive impact in performance.

Needless to mention that in reality there are much more intricate details about each point mentioned above, I have just tried to give an overview of all three things.

Popular Implementations for Languages

After going through above, one thing we need to understand is that languages are not compiled or interpreted, it’s the implementation of their translator which takes such approaches. A language may have one or more translators using different approaches. We’ll go through some popular implementations (translators)-

JAVA
- Implemented in C, It has two components - JAVAC, working as compiler and JVM, working as (interpreter + JIT compiler)
- JAVAC compiles the source code (.java) into bytecode (.class)
- Then bytecode is interpreted into machine code line by line by platform specific JVM which acts as interpreter while applying JIT compilation.
Python
- CPython
  - Implemented in C, acts both as compiler and interpreter
  - Implicitly compiles the source code (.py) to intermediate-code (.pyc), no manual action is needed
  - Then intermediate-code is interpreted into machine code line by line.
  - Even the interactive shell that comes with it follows the same process.
- Jython
  - Implemented in Java, acts as compiler only
  - Compiles the source code (.py) into Java bytecode
  - Then bytecode can be fed into any JVM (Java Virtual Machine) which produces the machine code.
  - Provides access to full Java library.
- PyPy
  - Implemented in Python itself, acts as compiler and (interpreter + JIT compiler)
  - Compiles the source code (.py) into intermediate-code (.pyc)
  - Then intermediate-code is interpreted into machine code line by line while applying JIT compilation.
  - In some cases, it’s faster than CPython since it makes use of JIT compilation.
- IronPython
  - Implemented in .Net, acts purely as compiler
  - Compiles the source code (.py) directly to machine code
  - Provides access to full .Net library.
C
- Original - I don’t know the exact name of C’s original compiler.
  - Implemented in Assembly, acts as compiler only.
  - Compiles the source code (.c) into machine code.
- GCC
  - Implemented in C itself, acts as compiler and assembler.
  - Compiles the source code (.c) into Assembly (.s)
  - Then intermediate-code (.s) is converted into machine code by assembler.

Registers and Memory

Before we go into the Assembly language itself, There are couple of things I would like to clarify -

Physical form of Memory - In terms of memory, the smallest data unit is bit which is either 0 or 1 so basically we want to have two states possible in our memory system to identify whether it has 0 or 1. In earlier time, Magnetic tapes were used which are made of tiny magnetic particles and there can be two states of direction of magnetic charge in those particles, thus providing 0 or 1. Later on, logical gates (Physical form of boolean functions) were used which when structured in a particular way such as Gated Latch become stateful , i.e. can store a bit of memory. I highly recommend checking out this video, It explains Memory in detail and amazingly simple manner. After having a mechanism of building 1 bit memory, next requirement was to have more of that, considering 1 bit memory system as abstract memory blocks, such blocks were then combined in forms of grids of grids of grids and so on thus achieving such high amount of memory that we see today. A 1 GB RAM stick actually has ~1 Billion physical memory blocks to store that many bits data, it’s amazing how technology has grown to be able to do that in extremely small spaces.
Registers - For efficient usage, access and writes on memory by processor, processors generally have very small amount of their own memory, called Registers. They are used by processor to hold instructions, data to perform instruction on and multiple other things. The registers have a digital representation also which provide a virtual abstraction between actual memory and their representation in programs, these registers are referenced in programs instead of individual memory addresses.

Assembly Language

Finally we are here! As we already know now, Assembly is a low level language created to provide an abstraction to programmers from machine code. In current time, Assembly is used when program needs to interact with low level components such as OS, Processor and BIOS. Being low level, it’s much faster than other high level languages, hence it’s another major usage is in time critical jobs./saurabhgoyal/

Below, I have given a very high level view of the language, it’s not a tutorial at all, for detailed tutorial, check out the links given in references section.

Addressing data in memory
- There are 3 steps towards execution of an instruction
  - Loading instruction and operands data from memory to register.
  - Identifying the instruction.
  - Executing the instruction.
- Registers are used to hold data required to perform an instruction. To verify that correct data was copied, for each byte of data, registers generally have an extra bit called Parity bit. Using this, correct count of set bits can be ensured, it’s not full proof though since it does a mod on the total count. Read more here.
Syntax
- Sections - An assembly program generally has following 3 sections in it-
  - data - All constants values are declared and initialized here.
  - bss - All variable memory spaces are declared and bound here.
  - text - Instructions for program logic go here.
- Statement format
  - [label] command [operands] [;comment]
  - The parts in square brackets are optional
- Memory segment - Memory is segmented into 3 parts to store specific type of data in it-
  - data - Stores data elements for the program, basically elements represented by .data and .bss section.
  - code - Stores instruction codes for the program, basically elements represented by .text section.
  - stack - Stores values passed to function calls in the program.

Conclusion

I feel that discussing actual syntax and full tutorial of the language here is useless, I have given some very basic idea above. Check out this repo to see more detailed code and instructions. Till now, I have added three basic and popular programmes - hello-world, triangle and fizzbuzz.
The whole experiment along with this post took ~4 days.
My favourite part of this experiment was definitely reading about compilers and interpreters.
/10x-dev
Checkout other posts from level-up series here.

Other Learnings

Makefile - I learned how to use this and what exactly is the benefit. I highly recommend it and there is a nice little tutorial given in references section, check that out.
Linux commands - Some new linux commands I picked up - file (Tells file type of passed file) and tee (Provides a plumbing like T for output stream of a command. Usage ex.- sending output to both stdout and a file)
Linux file descriptors - STDIN is represented by 0, STDOUT by 1, and STDERR by 2.
Linux null device - I knew about /dev/null earlier too but not formally, it’s a null device (a file actually), discards any input given to it, so used mainly when you don’t want the output or error from a command.

References

Repository with implementation - https://github.com/SaurabhGoyal/programmer-achievements/tree/master/assembly
Tutorial source - https://www.tutorialspoint.com/assembly_programming
Registers and Memory Video - https://www.youtube.com/watch?v=fpnE6UAfbtU
Programming Languages Video - https://www.youtube.com/watch?v=RU1u-js7db8
Assembly (using NASM) in 64 bit - https://cs.lmu.edu/~ray/notes/nasmtutorial/
Makefile - https://opensource.com/article/18/8/what-how-makefile
Level-up Series -
- Blog post series
- Achievements Gist

Levelling up as a developer

Saurabh Goyal — Thu, 30 May 2019 02:48:07 +0000

Most programmers/developers, if not all, have a hunch to up their game, to level up in their skills and to be considered among the good programmers out there, unfortunately not everyone is sure what path to take and more importantly how to even measure their skills. Data structures, algorithms, system design, tech stack knowledge and experience are the industry standard right now but they are not quite measurable and do not provide users with trackable objectives to complete.

^{https://ardalis.com/working-through-roadblocks-a-guide-for-new-programmers}

With that said, A quick google search landed me on this blog post by jasonrudolph and this lists some of the objectives which can be considered as achievements for a developer. What I liked about them is that for someone who doesn’t have a path, they provide a nice way to get started, get uncomfortable in their programming routine and be better on the way.

The steps are very simple-

Select a particular experience to pursue.
Pursue that experience to completion. (Achievement unlocked!)
Reflect on that experience. Really soak it in. Maybe a blog post would be in order?
Return to Step 1, this time selecting a new experience.

Step-3 is the most crucial, this is what will ensure that you learn something with each achievement.

I have forked the gist which had the list of these achievements to be completed, check them out, If you like them, fork it and add your own achievements. The current ones in the list are really interesting and I am positive that everyone can learn a lot with each achievement. I’ll be updating this post as and when I complete an achievement. All posts under this series will have the prefix Level-Up-Series.

Achievements

*1. Write a program in assembly language (June 2nd, 2019) -

So the basic idea was to write any small program in Assembly, but while doing that I learned a lot of the other things. Check this post for detailed information on this.

Scheduling a task - for dummies

Saurabh Goyal — Sat, 29 Jul 2017 17:48:07 +0000

What -

Basic concept is that there is a work at hand and there is some related task to be done, but not right away. Some situations when scheduling a task may be needed instead of doing right away are -

When the task is not to be done now because of lower priority, un-availability of required resource, complexity of task, cost of running the task, priority of the main work at hand or various other reasons. Then you schedule that task to be done later.
When the task is to be distributed to one or more other people who’ll do it. You need a common platform to communicate about the task to them. You schedule that task to be queued in their list of work.
When the task is to be repeated at a certain frequency such as daily or weekly. You schedule that task to be run periodically.

Now how do we schedule? Let’s look at the components involved. At it’s core, a task is nothing but a set of instructions to be performed on a particular data set. That is the data you need to pass to schedule a task. The combined data can be called a Message which is passed from the scheduler to whoever will perform that task.

How -

Protocol

We need a protocol (a set of rules or a contract) to do that in efficient and inter-operable way in machines (read- computers). The protocol used for that is called Messaging Protocol. One of the most common messaging protocol that we use or rather did use in our daily lives is post office service, where we have to provide destination, source, add required stamps and enclose the contents in a wrapper for the messaging to be completed. In computers, one such protocol is SMTP (Simple Mail Transfer Protocol), which powers the email service.

If you are wondering why a separate protocol, let's just think this way - a protocol is nothing but a refined version of whole or parts of one or more other protocols customized for one specific purpose.

Followings are the benefits of using a specific messaging protocol over other such as HTTP/IMAP -

HTTP is a synchronous protocol and works on a point to point connection which means whenever a message is sent from the sender, it’ll be received by only one receiver and in synchronous mode, i.e. if there is any interruption before the data has been completely received by the receiver, data will be lost. This may be fine for web applications where client(receiver) will request again, but is not suitable for messaging which requires guarantee of message delivery to one or more clients which may or may not be available at the time of sending the message,
HTTP has high dependency of sender and receiver on each other and each part of communication highly depends on proper understanding and knowledge of both. This prevents the consumers, publishers and other parts from being independent of each other and anywhere and with any level of capacity.
Messaging protocols are created for messaging only and hence they optimize on connection establishment and request-response cycle in multiple ways, making it the solution for this purpose.

As most messaging protocol make use of asynchronous execution of sub-parts such as publishing, routing and consuming, they are named asynchronous messaging protocol. There are various such protocols - AMQP, MQTT etc.

Components & Process -

Now that you have the message and a protocol to send it, what’s missing? Consistency and Reliability. In simpler terms, We need to ensure -

The message does not get damaged/altered in any way before the person gets it.
The message reaches the intended person.
And most importantly, the task in the message gets executed successfully before discarding the message.

For the above, one good approach is having a message storage and a storage-manager to prevent losses and manage routing and message persistence in case of errors. So there are-

Messages to be delivered which consist of the task to be performed and the data on which it’s to be performed.
One or more publishers that produce messages and
One or more consumers which consume messages.
Message storage - Queue is the data structure used mostly for this.
Manager - A program/service/middleman that links all parts and facilitates the messaging.

This may sound simple but the whole benefit of message reliability, interoperability between all kinds of resources because of common standard, scalability in terms of messages, publishers and consumers and the dependency removal between publishers and consumers is all that async messaging protocols bring to the plate.

In general, There are two popular patterns for messaging -

Publisher Subscriber - Publishers publish messages for certain topics. Consumers subscribe for certain topics. Whenever a new message is available for a topic, all subscribed consumers get that message.
Message Queue - Publishers add message to a queue and consumers consume that queue. A message is consumed by only one consumer.

Let’s Talk AMQP-

AMQP (Asynchronous Message Queuing Protocol) is a specific protocol that uses the components mentioned above and adds some more.

^{https://commons.wikimedia.org/wiki/File:The-amqp-model-for-wikipedia.svg}

The store-manager/middleman is called Message Broker.

In a nutshell, Broker maintains one or more queues where all incoming messages are stored until they have been successfully consumed as expected. These queues may be in-memory or persistent. The messages can be added directly to the queues by broker or another layer called exchange be added which controls and manages which messages are to be added to which queue based on rules called binding.

The consumers are called workers (Their work is to consume the queue). They may be increased or decreased based on load on the queue. They can be configured to consume only certain types of queues thus to distribute consumption of different kind of tasks to different workers.

^{https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html}

Popular names -

Celery - Just a client to be used as a producer to assign messages to broker or consume messages from broker. It basically is an interpreter between your application and broker and uses AMQP to interact with broker, so that your application does not have to do all that. Other similar solutions are - pika, kafka etc.
RabbitMQ - A broker that manages queues and interacts with publisher and consumer. Has highly reliable and comprehensive features to ensure message delivery and prevent any loss. Other similar solutions are - ActiveMQ, Amazon SQS, kafka etc.
Redis- A data store that stores key-value pairs, supports certain types of data-structures as values and performs very well for high speed requirements. All data is stored in memory and hence is often used as cache. Other similar solutions are - memcached, mongodb etc. Why this is here - It has recently added support for publisher-consumer pattern through AMQP and hence is also getting popular as a broker.

References:-

Sensitive info in the project!

Saurabh Goyal — Wed, 19 Jul 2017 17:18:07 +0000

I have been more and more involved in tasks such as deployment and server management recently and I am still struggling through the nuts and bolts of the whole process from pushing the changes in the repository to deploying those and making sure that each resource in the infrastructure has successfully started running with the new changes.

When going through our deployment scripts and resources the other day, this question popped up in my head.

How do we manage our sensitive info in the project?

The sensitive info that consists of any of the below and more-

API tokens to third party services. (such as Google APIs, Twilio etc.)
Credentials/access keys to connect to either your own resources or others’. (such as AWS secret and access keys, address of resources such as servers, database instances etc.)
Configurations for various services for your app. (such as Email configuration, Encryption keys, Critical settings or flags etc.)

After some digging, I found we have stored these into the codebase only and is included in the version control system(git). This is a basic mistake that people do in the initial phase of a project without understanding the consequences. I'll talk about it more later in the post.

Before proceeding, we must understand how and why above happens. So you have created your awesome app which uses some third party APIs or have some critical configuration. The data for those is sensitive and must not be made public, so you have setup your project such that whenever someone is initiating your app, he or she will provide all that sensitive data in a configuration file or through environment variables manually and your app will use that.

All your sensitive info is off the code-base, secured to one or more authorized personnels and things work fine UNTILL your product starts to grow and you face following problems.

You have multiple environments of your app - dev, staging, prod, each with a different set of sensitive data/configuration.
The sensitive data/configuration itself starts getting big as number of services and resources you use increase.
You have whole system and separate team for DevOps, and the work is not restricted to a set of personnel anymore. The way they store/manage the sensitive data is a big concern for you and may cause you tons of losses in worst cases.

That is to say, the offline method is not scalable. When above happens and there are multiple versions of configuration files, you need to find a way to store them securely and manage them better, also making sure their availability and integrity. You can’t keep all those offline anymore.

The intuitive and probably the best option is to put them in a private secure location (accessible only be authorized personnels such as devops engineers) and whenever instantiating the app, all data is pulled from there.

The locations can be places like a repository or AWS S3 bucket or any of the multiple available services to store sensitive data (PassPack, Swordfish, CHEF server etc.). In any case, we must always store encrypted data to those places and decrypt before using at our end to ensure security even if data from that location is compromised.
Although, in that case you’ll have to provide keys/credentials to fetch the data from that secure location. But this is not same as before because we have successfully abstracted all sensitive data, which may increase a lot, behind one secure location/service, size of credentials to which is and will always be fixed.

The above is the basic idea on how to automate deployment and configuration of one or more servers while keeping the sensitive data managed at one place securely. Of-course it’ll need some work at our end to modify our deployment script accordingly. To give you a better understanding, I’ll explain what We did in one of my projects-

We stored all the sensitive data in an S3 bucket. The data was encrypted before upload using a password. We used a tool called Ansible-Vault for encryption which is good because we use Ansible for all our deployment scripts and Ansible-Vault is a module of Ansible. The major benefit of using Ansible-Vault is that when running Ansible scripts, if we provide the password of encryption used with Ansible-Vault then Ansible, when processing any variable file, decrypts the file before its use automatically.
Alternatively, we could use something else for encryption and modify our deployment script to decrypt the files manually.
Now whenever deploying, all that is needed by the script are AWS keys for fetching file from S3 and the password/key for decrypting that file.

Note:- A repository is better in terms of modification and history management of sensitive data itself, which is not fully available with S3. The benefit mainly comes when you have lots and lots of sensitive data and multiple DevOps engineers who may be making changes to same key/s simultaneously. Although this is improbable in most projects.

Now let’s discuss what I wrote earlier - Committing any sensitive data to version control system is a big mistake. Because-

The sensitive data is in the history even after moving out. Proper removal will require-
- History manipulation - which although possible is not very clean and smooth.
- Rotation of all your sensitive data - Once a piece of sensitive data is public, it’s never fully safe to use that. Eventually you’ll have to rotate it for a new value which may be quite some work based on it’s purpose and howmany places is it being used.

With all that said, there are multiple tools such as truffleHog which search repos for any sensitive info in full history. They must be used if not confident.