DEV Community: Joana Chavez

99 problems but Python ain't one!

Joana Chavez — Wed, 22 Aug 2018 20:46:14 +0000

Let's start from the very beginning, defining the different approaches of how code is finally executed. In general, when a programming language is used, two approaches are possible: either compilation or interpretation. Oh and there is also a third one which is a hybrid approach and is in between both of those said previously. Let's take a look:

Compilation: When compilation process is done, what is done is a executable code obtained from the whole code, so from all the source code what is obtained is a executable that can be used as it is and that has the same characteristics needed by the machine. (Let's say an example would be Java, which through compilation it can be run over the JVM) The guy in charge here to do this job is the compiler (duh! of course he is!) In the image below, we can see more in deep the whole process...

As you can see, it takes the whole thing in there and then produces the executable to be then processed.

Interpretation: On the other side, with interpretation we obtain let's say a "translation" of each instruction from each of the statements, so it is a job done statement by statement, that is, instruction by instruction is being interpreted by the interpreter (duh again, of course that's his name!) Languages that could be used as an example of interpretation are shell scripts, like bash for example, or Javascript, PHP, and oh! also the most important dude in this article: Python.

This process can be seen in a better explanation here:

As you can see, statement (stm for short) by statement are processed and immediately executed.

Ok but there should be a third one right? there is always something in between... and yes there exists also a third approach that uses a combination of both approaches and it is for intermediate languages, we can see it in the following image:

In this last one, the compilation is done up to an intermediate language and then using the executor which is no more than an interpreter like the one used in the interpretation process, we can say that is an approach in the middle (has some features from compilation and some from interpretation)

Okay, but going back to the idea of why Python is so cool? I mean, it is widely used in the areas like Bioinformatics, Data Science and so on, and the reason is because of many reasons like for instance:

Readability : It is considerably ordered, and this helps a lot for making it easier to people to read code ( even to make your code work you will need to indent everything okay before using it! )
Code reuse: There are many modules available to be used in Python for different purposes, that's the feature that some call "batteries included"
High Performance: It's very close to C, but the use of other libraries that enable an efficient computation like for example NumPy, increases much more the performance.
Simple: Again, and this one is also related to the first one in this list, the readability and the minimalistic characteristic it has, makes it easier to concentrate in the problem you have instead of thinking in other low-level language problems. (Like for example exceeding memory or garbage collection... who thinks of that in Python? )
Easy to learn: Another point for Python, the learning curve is considerably low and you'll learn very fast! As a matter of fact I think this is the one that actually is considered for Bioinformatics areas, I had colleagues in Bioengineering areas who are not used to code all the time, but this language makes it easier to get started!
Free and Open source: Another one yet is that it's free! so you don't have to worry about licenses and things like that, even with so many different IDEs like Spyder and package managers like Anaconda. (Another cool feature from Spyder better than the not free IDE PyCharm is that for Pycharm you don't get the coolest help inside the IDE, while with Spyder you can get to search the documentation of the code inside the IDE, how cool is that?
Interpreted: This one is more related to the previous we talked about the performance but it's about it.. since it's interpreted, the execution is very fast, but of course this will also depend the amount of data you're processing!

Great, so now that you got all these ideas I hope this has been an eye opener on what to expect from the language, and if you are considering on learning about it, I totally recommend it!

If you're having girl problems I feel bad for you son, I got 99 problems but Python ain't one!

Singularity : a "Docker" for HPC environments

Joana Chavez — Tue, 12 Jun 2018 18:04:30 +0000

Companies nowadays are under pressure to digitally transform their applications but are constrained by existing applications and infrastructure while rationalizing in a diverse portfolio of clouds, data centers and application architectures.

Many different container platforms have been released to the market, overall the most well-known is Docker, a very powerful container platform that works perfectly for "almost" all use cases of applications. I said “almost” because Docker is oriented to micro-service virtualization.

However, in the HPC (High Performance Computing) context, scheduling resources is an essential feature that can considerably determine the performance of the system. This type of applications run a wide range of computationally intensive tasks in various fields, some examples including quantum mechanics, weather forecasting, climate research, oil and gas exploration, molecular modeling, biological macromolecules and physical simulations. Centers and companies that use this type of applications cannot run the risk to run Docker on their environment because it simply does not fit their use case. Some of the main reasons of this incompatibility are the following:

Security issues: HPC environments are multi-user systems where users should only have access to their own data. While Docker gives superuser privileges, it does not avoid the ability of accessing other users’ data and with it getting also control of the cluster and the computing resources. Singularity avoids this security problem by allowing different mechanisms for privileges:

Root owned daemon process: This mechanism is the most similar to Docker, because there is a root owned background process that allows the management of containers and spawns the jobs within the container. The access of users to the daemon is allowed through the IPC control socket for communicating with the root owned daemon process.

Limited usage to root: This mechanism allows only the root user to run the containers.

User Namespace: This mechanism allows a user to virtually become another user and run a limited set of privilege system functions. Singularity supports the user namespace natively and can run completely “rootless” or without privileges, of course the features are severely limited.

SetUID: Singularity supports the “old school” UNIX method for running a particular program with escalated permission. This mechanism gives a lot of flexibility in terms of supported features and legacy compliance.

Capability sets: with capability sets you can set up privileges by process and per file basis. This is an alternative to SetUID because it allows much finer grained capability control.

Portability:
Another awesome feature from Singularity is the portability. You can take your container image with your environment which inside has all of your scientific apps and tools that you need and carry it from system to system. Docker uses layers for this matter, making the image more difficult to share.

In the necessity to support all the previous needs, Singularity was born. It was developed by Sylabs and a community who continuously work on this platform. The first public release was on April 2016 and it had a massive uptake from different companies and research groups around the world. As a matter of fact, this powerful tool got the attention of the HPC Wire team and was also named as one of the top technologies to watch on 2017.

We have just started to scratch the surface of deep learning and AI. This is still a new unexplored field and it’s expected to be evolving rapidly for the next 20 years at least. Singularity is already the standard de-facto in the academic world for running HPC workloads related to ML, DL, and AI, and it is deployed at hundreds of supercomputer research centers worldwide. Singularity is the best choice for Enterprises that want to enter these new emerging fields because it creates a bridge between the research world and the enterprise world and allows recent advances in research to freely flow into enterprise offerings. New disruptive technologies require the IT infrastructure of enterprises to adapt quickly, but this conflicts with the need of IT departments to preserve stability, security, and performance. Singularity solves this problem by introducing a layer of abstraction between the applications and the underlying operating system, while providing a safe passthrough to access computing resources required by ML, DL, and AI directly, with no performance penalty and without compromises on security.

Encryption 101 : Introducing the idea of Symmetric Encryption

Joana Chavez — Sun, 16 Apr 2017 22:25:36 +0000