DEV Community: Shuvam Manna

Octo's guide to the Git-verse

Shuvam Manna — Wed, 30 Jun 2021 09:14:23 +0000

Earth to Earth-22. I hope this is a time when things have gone ahead, and we have left the umpteen crises behind. This is the bajillionth article on git floating across the multiverse, and why should we consider this to be any different.

What is git? A life form in a remote planet asks. The human tears up. The excerpts of the conversation that ensued shall be duly paraphrased and translated — because honestly, live translation is an extremely resource-intensive process and we would not want to burden you with the same. If the adjacent xkcd comic is any indication, if there are two words to describe git, one of them is definitely — confusing. The second one might lift you up a little, maybe a lot, when we say that one of git’s many superpowers is that it is absolutely amazing.

Now that brings us neatly back to one question — What is git? And that also ties in with a second question of why we need it in the first place when you also have things like Subversion (duh!) and Mercurial (who is in no way related to Mercury, the Roman brother of Hermes — the Greek God, not the brand)

Git is a version control system. The idea behind git is that you can track changes in your project folder and store snapshots of how the project had looked at certain points in time. It tries to unclutter the ubiquitous problem of creating umpteen folders to maintain snapshots of your code by creating a virtual Checkpoint mechanic you find in games. It is especially helpful when we are collaborating with others, or finding instances of when certain changes were made and reverting them (if they break your existing infrastructure), if necessary. And it is also useful as a documentation tool if used properly with the right set of comments and commit messages. The most powerful aspect of git, and where it ascertains its supremacy over virtually any other version control system is that it was built to be distributed. And because it is distributed, there are certain weird things and quirks that don’t always make perfect sense.
With other version control systems (referred henceforth as vcs) like Subversion or Mercurial, you would notice that they are sequential — your version numbers follow a neat and simple 1,2,3,4,5… In git, you have this uber long commit ID that is a 160-bit long hash string that identifies versions or commits history checkpoints.
Now, while this was done to facilitate the unique naming of commits in a decentralized system, the question naturally pops up is: How likely is it that the hash generated for a future commit will coincide with the hash of some past commit?

The question is closely related to something we normally refer to as the Birthday paradox, from where we see that if we were to randomly select n *from a set of **N* distinct elements, the probability of drawing the same element n more than once will be greater than half if — n ≥ 1.2√N.
Every time a commit is added to a git repository, a hash string that identifies this commit is generated. This hash is computed with the *SHA-1* algorithm and is 160 bits long. Expressed in hexadecimal notation, such hashes are 40 character strings.
To go a bit further into the math of the probability of a hash collision *(without using scary figures) *— *At a high rate where every human in the world (say, 7 billion) makes a commit every second, mankind would need nothing less than 6.66 million years to produce a number of commits large enough to create a hash collision with 50% probability!
Click *here to read more about the Birthday paradox.

The life cycle of a project with git version control initialized starts with the git init command, which initializes the entire set of operations that are to happen on the repository. What git init does is it initializes a directory called .git inside your project folder. Now, for other vcs, it mostly operates on a client-server model where you check-in your code synchronously by coordinating with, often manually, with other clients connected to the same centralized server.
In git, the .git directory is essentially the folder with some metadata where all operations are performed. Since it is decentralized, there is no network communication. All operations are local and file CRUD operations happening inside the .git directory.

The workspace

In the git-verse, as we initialize the git tracker in a folder, the underlying mechanism slaps on these two additional zones on top of your working layer.
This brings us to another command — git add that takes your files and moves them to the Staging area as a rough draft that you can later publish to your repository history. In this context, we can imagine staging being this fantastical inter-dimensional layer that holds your data.
You generally use the command as git add . or git add * that adds all the relevant unstaged files in your working area to staging. You can also do a git add to include specific files into this layer.
To undo this step, i.e. to get back data from staging into the working layer, we use the commands git reset for a generic removal of everything from staging and git reset if we just want to do that for a specific file.

We use another command called git commit to move data from the staging layer to the Repository layer. This command takes the rough snapshot of your work that you have in your staging, where you add and remove files and changes, and saves that snapshot forever in the repository later.

So, let’s say we take a file and run git add, you should be ideally able to track changes as they happen in the .git folder, as they are just file operations. There is no actual file movement involved. The system takes the file, observes it’s contents and forms a BLOB, takes some header information like how long the file is, etc. and passes it on to the *SHA-1 *algorithms that give us our 40-character hash, and stores it in the *objects *subfolder under .git .
Now when we run the git commit -m "Some commit message" in terms of the git system, it creates another structure in the *objects *subfolder called a *tree. *The purpose of a commit is to create a snapshot of your project at that point in time, with the message mentioned in quotes — The *tree *represents what your working directory looked like at that point in time.
To see the components of a *tree *snapshot at any point in time, use the command git ls-tree .
The tree essentially contains a reference of UNIX or OS (depends on your Operating System) permission code (say 644 or 755), a numeric reference to your type of file, a reference to your BLOB, the commit ID and the actual file that was added/changed.
Now the interesting bit is, what happens when you rename a file?
On the terminal, you see an old file is deleted and a new file is created. But under the hood, you essentially create a new tree that has a pointer referencing the same old BLOB because the contents remain the exact same. This is the mechanism employed as a redundancy handler and takes up lesser memory.
Read more on *trees *in git here.

Another cool command/tool that we would explore before moving on to the repository layer is a command called gitk.
This is a free GUI tool that gets installed when you install git itself, and helps you visualize the commit, the commit message and commit ID, etc.

The last thing that we would touch base on in this post, is the idea behind branches. The basic idea that pops into our head when we talk about branches is that there is a divergence — that the trackers go sideways, and a whole new route emerges and something of that sort and that idea is probably reinforced when we mention that git uses trees — but that’s the wrong mental model to have.
A branch is essentially a pointer to the elements that the tracker is looking into. For instance, the default branch tracker when you initialize a git repository points to your first commit. Once you do a second commit, the branch pointer points to your second commit, which in turn points to your first commit and so forth.

Therefore, in a broad sense of abstraction, branches are essentially pointers to a specific snapshot/commit in your git repository. They are not divergent literally like branches, and these branches are in no way related to the *tree *objects that we mentioned earlier.

This is the inner functioning of git in an extremely abstracted manner — an insight as to how git works under the hood. Obviously, this does not cover all aspects, there are far finer nuances to this amazing version tracking system, for instance, the intelligence layer that groups similar BLOBs to optimize memory usage, or the redundancy checks that take place to ensure that the .git folder does not exceed the repository source code itself in size (That would be hilarious though).
There are some pretty nifty sources to learn more about git around the Web. Use this cheat sheet by Atlassian to learn more about git commands.
Some more commands to enhance your git experience can be found here.
Until next time, keep making Peatigraffes 🤘🏼

If you want to talk about Communities, Tech, Design, Web & Star Wars, get in touch with @shuvam360 on Twitter.

Originally Published on Medium in 2020

Democratizing ML: Rise of the Teachable Machines

Shuvam Manna — Tue, 11 May 2021 07:31:05 +0000

Late in 2018, Google Creative Labs came out with the concept of Teachable Machines. A Web-based demo that allowed anyone to train a Neural Net into recognizing and distinguishing between three things and bring up suitable responses. It was a fun example to play around with and served to teach many the fundamentals of how Machine Learning works at a fairly high level of abstraction. Recently, they released Teachable Machines v2, a full-fledged web-based dashboard to play around with models that can be retrained with your data, and the models which can further be exported to work with different projects and frameworks, thus letting it out into the wild.

The models you make with Teachable Machine are real Tensorflow.js models that work anywhere javascript runs, so they play nice with tools like Glitch, P5.js, Node.js & more. And this led me to think about how this tool was making some really powerful ML capabilities available to everyone, in the process, democratizing the idea that everyone — from the noob to the pro can use this for prototyping their vision or even put things into production at a scale. But with the availability of these Teachable Machines, let’s take a peek under the hood.

Holy Grail of Machine Learning

The idea of Machine Learning is pretty simple — a machine that learns on its own, similar to how humans learn. But these machines are governed by a representation of the primal human instinct — *Algorithms. *A voice in your head saying Do this, no don’t jump off a cliff, you’re not Superman, nor do you have a parachute or the very act of learning why an Apple looks like an Apple is governed by these small instincts.

Hundreds of learning algorithms are invented every year, but they’re all based on the same few ideas and the same repeating questions. Far from being eccentric or exotic, and besides their use in building these algorithms, these are questions that matter to all of us: How do we learn? Can this be optimized? Can we trust what we’ve learned? Rival schools of thought within Machine Learning have different answers to these questions.

Symbolists **view learning as the inverse of deduction and take ideas from philosophy, psychology, and logic.
**Connectionists reverse engineer the brain and are inspired by neuroscience and physics.
Evolutionaries simulate the environment on a computer and draw on genetics and evolutionary biology.
Bayesians believe learning is a form of Probabilistic inference and have their roots in statistics.
Analogizers learn by extrapolating from similarity judgments and are influenced by psychology and mathematical optimization.

Each of the five tribes of Machine Learning has its own general-purpose learner that you can in principle use to discover knowledge from data in any domain. For the Symbologist, it's the Inverse Deduction, the Connectionists’ is Backpropagation, the Evolutionaries’ is Genetic programming, and the Analogizers’ is the Support Vector Machine. In practice, however, each of these algorithms is good for some things and not for others. What we ideally want, in these cases — is a single Master Algorithm to combine all of their best benefits.

Enter the Neuron

The buzz around Neural Networks was pioneered by the Connectionists in their quest to reverse engineer the brain. Such systems “learn” to perform tasks by considering examples, generally without being programmed with task-specific rules. For example, in image recognition, they might learn to identify images that contain donuts by analyzing example images that have been manually labeled as “donut” or “not donut” and using the results to identify donuts in other images.

An Artificial Neural Net or ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal to other neurons. An artificial neuron that receives a signal then processes it and can signal neurons connected to it.

courtesy Wikipedia

The most first neural networks only had one neuron but these aren’t very useful for anything so we’ve had to wait for computers to get more powerful before we could do more useful and complex things with them, hence the recent rise of neural networks. Today’s neural nets consist of multiple neurons arranged in multiple layers.

In the figure, the leftmost layer is known as the Input Layer, and by happenstance, the rightmost — Output Layer.

TL;DR: Neural networks consist of neurons arranged in layers where every neuron in a layer is connected to every neuron in the next layer. A neuron multiplies the data that is passed into it by a matrix of numbers called the weights (and then adds a number called a bias) to produce a single number as output. These weights and biases for each neuron are adjusted incrementally to try to decrease the loss (the average amount the network is wrong by across all the training data).
A great website if you wish to learn more is machinelearningmastery.com

Teachable Machine

The Teachable Machine relies on a pre-trained image recognition network called MobileNet. This network has been trained to recognize 1,000 objects (such as cats, dogs, cars, fruit, and birds). During the learning process, the network has developed a semantic representation of each image that is maximally useful in distinguishing among classes. This internal representation can be used to quickly learn how to identify a class (an object) the network has never seen before — this is essentially a form of transfer learning.

The Teachable Machine uses a “headless” MobileNet, in which the last layer (which makes the final decision on the 1,000 training classes) has been removed, exposing the output vector of the layer before. The Teachable Machine treats this output vector as a generic descriptor for a given camera image, called an embedding vector. This approach is based on the idea that semantically similar images also give similar embedding vectors. Therefore, to make a classification, the Teachable Machine can simply find the closest embedding vector of something it’s previously seen, and use that to determine what the image is showing now.

This approach is termed as the k-Nearest Neighbor.
*Let’s say we want to distinguish between images of different kinds of objects we hold up to the camera. Our process will be to collect a number of images for each class, and compare new images to this dataset and find the most similar class.
The particular algorithm we’re going to take to find similar images from our collected dataset is called *k-nearest neighbors. We’ll use the semantic information represented in the logits from MobileNet to do our comparison. In k-nearest neighbors, we look for the most similar k examples to the input we’re making a prediction on and choose the class with the highest representation in that set.

TL;DR: The **k-nearest neighbors** (KNN) algorithm is a simple, supervised machine learning algorithm that can be used to solve both classification and regression problems. It’s easy to implement and understand but has a major drawback of becoming significantly slows as the size of that data in use grows.
Read more here.

What can you do with TM? (Yellow Umbrella, anyone?)

Teachable Machine is flexible — you can use files or capture examples live. The entire pathway of using and building depends on your use case. You can even choose to use it entirely on-device, without any webcam or microphone data leaving your computer.

The subsequent steps to using these for your projects/use cases are pretty simple. You open a project, train the model on your custom data — either by uploading images/audio or capturing data using your webcam or microphone.
This model can be further exported and used on your projects just like you’d use any Tensorflow.js model.

Barron Webster, from the Google Creative Lab, has put together some really amazing walkthroughs to get started with TM. Check out how to build a Bananameter with TM here.
The demo is also out in the wild as a *Glitch *app at https://tm-image-demo.glitch.me/

Happy Questing!

If you want to talk about Communities, Tech, Web & Star Wars, hit me up at @shuvam360 on Twitter.

Originally Published on Medium in 2019

Pablito Planeta, your design astrologist

Shuvam Manna — Sat, 09 Jan 2021 19:53:08 +0000

What I built

Pablito Planeta is a Zodiac Prediction app based on the Pablito Planeta series on Instagram by Pablo Stanley. As soon as I saw those videos, I was smitten, and wanted to web-appify them real quick. Also, I still had no idea of what to build for this Hackathon.
Thankfully, Pablo responded in a jiffy, approving the idea and for me, giving the proverbial green flag.

With this app, you choose your Zodiac, and voila, you have your predictions in front of you.

Category Submission:

Program for the People, because well, you're predicting, for the people, right. Right?

App Link

Find the app right here - Pablito Planeta Web App

Screenshots

Description

The app is a fun place for users to see their Zodiac predictions for their Signs, along with their Design amulet, Google Font, and Color.

Link to Source Code

Pablito Planeta on Github

Permissive License

MIT License(Link to License)

Background

I was super stoked by Pablo's series on Design Astrology and wanted to hack together a Web-based interface for folks to see, learn, and share the predictions for them.
Also, it was really cool to hack with vanilla Javascript after long.

How I built it

This is a static app, and I used DigitalOcean's static site deployment tools to set up an integration with the Github repo. Plus, this is a Design aesthetic I had been planning to use for a long time.

Additional Resources/Info

Pablo Stanley's Instagram