DEV Community: MoRoth

How did I reduce project onboarding from 2H 😫 to 3 MINUTES 🔥🔥🔥 ?

MoRoth — Sat, 04 Nov 2023 21:09:16 +0000

Following along "Getting started" texts for complex projects can be frustrating. It is also hard to write them well and also it requires the people following it to do many many manual actions.

Taking for example Cocmd
For someone to get stared it requires having

git installed
node installed
vscode (or some other ide)
cloning all the repos
setting up git hooks
going one by one and building with npm and cargo

So instead of writing a long readme on how to do all that.
We decided to use cocmd own cli to create super fast onboarding experience for project contributors:

Install cocmd Getting Started
install and Run the Contributors onboarding packages:

cocmd install https://github.com/cocmd/cocmd
cocmd run cocmd.contrib-onboarding

it's going to:

install git
install rust
install node
clone all Cocmd repos
build the website
build cocmd cli
open vscode with all the repos

You can do the same for your own projects

and be the superhero of productivity in your workplace!

onboarding
playbooks
sharing cmd stuff

Use cocmd to make your dev life better cocmd.org

join the open source project

How can you help?

Data teams can deliver 10x better to the rest of us

MoRoth — Sun, 29 Oct 2023 14:01:11 +0000

TLDR

This is a standard DBT project example data project "Getting Started",
long long long list of instructions on how to setup and run everything. familiar right?

with CoCMD you could install all dependancies, setup and run it with one line, for any operating system:
datateam.onboarding

behind the scenes, just one yaml and an awesome hub:
code

Setting the stage

Data teams are instrumental in data-driven organizations.

A significant aspect of their role involves conveying to others how to manage data operations and services.

A key facet of their responsibilities entails effectively imparting knowledge on managing data operations and services to others. This often takes the form of

extensive how-to guides,
documentation,
archived Slack conversations,
and the sharing of internal knowledge. They guide teams on various tasks, such as setting up machines for data operations, conducting queries, establishing connections, and more.

The Challenges Data Teams Face

Complex Toolchains: Data professionals often use a range of tools for data collection, data processing, and analysis. Coordinating these tools and ensuring their proper setup can be a daunting task.
Onboarding: New members joining the data team need to quickly adapt to the toolchains and workflows. A smooth onboarding process is crucial for their productivity.
Repetitive Tasks: Data teams frequently deal with repetitive tasks, such as data preprocessing, model training, and report generation. Automating these tasks can save valuable time.
Collaboration: Effective collaboration within the data team and across departments is essential for sharing insights and results. This requires streamlined processes for sharing code, data, and documentation.

How CoCMD Can Help

CoCMD offers a simple and open command-line interface to address these challenges. Here's how data teams can benefit:

Streamlined Toolchain Setup

CoCMD can automate the setup of the various tools and packages needed by data teams. With a single command, team members can have their development environments configured consistently.

Effortless Onboarding

New team members can be onboarded seamlessly using CoCMD. By following predefined onboarding playbooks, they can quickly become productive, eliminating the time-consuming process of manual setup and configuration.

Automation of Repetitive Tasks

CoCMD allows data teams to create playbooks for automating repetitive data-related tasks. Whether it's data preprocessing, model training, or report generation, these tasks can be turned into repeatable, one-click processes.

Enhanced Collaboration

CoCMD playbooks and routines can be easily shared within the team and with other departments. This ensures that everyone is using the same tools and processes, leading to improved collaboration and consistent results.

Getting Started with CoCMD for Data Teams

To get started with CoCMD for your data team, follow these simple steps:

Install CoCMD: If you haven't already, install CoCMD using the installation command provided on the CoCMD website.
Select Data Packages: Choose the data-related packages you need from the CoCMD-Hub, which includes packages for data analysis tools, libraries, and more.
Find in the hub what you need
https://cocmd.org/docs/packages/from_hub/
or write your own https://cocmd.org/docs/intro
Create Data Playbooks: Develop playbooks and routines specific to your data tasks. These can include data preprocessing, machine learning model training, and report generation.
Share with Your Team: Share your playbooks with your data team, making it easy for everyone to access and use them.

With CoCMD, data teams can deliver 10 times more effectively, providing insights and solutions to the rest of the organization faster and with greater consistency:

great dev experience with cocmd cli
private and secure
re-use community knowledge
automatic Docs generation
automatic notifications system
automatic recommendation system
ready distributions to all operating systems

Visit the CoCMD website to learn more about CoCMD and how it can revolutionize your data team's processes.

Contact the CoCMD team at mzsrtgzr2@gmail.com for any questions or assistance.

New opensource to replace long "Getting Started"s forever

MoRoth — Wed, 18 Oct 2023 11:03:13 +0000

Do you find long readmes/wikis with complicated setup instructions frustrating?

So Do I :)

Instead of writing down long instructions with prerequisites, installation command, configuration changes and all this with operation system variations notes - let's create an amazing experience for the people who needs the software you created. No one cares about the "steps" along the way, just make it happen.

So I created CoCMD. A simple and open cli app to make the onboarding and playbooks experience amazing.

here is the basic concept:

How does it look?

Let's take for example our Kubernetes package.
I created several onboarding and playbooks that can be useful.

Showcase

It's important to talk about specific "stories" where the benefit of using CoCMD is obvious.

Project Onboarding

Onboarding new hires is a very important process in any company. It's the first impression that the new hire gets from the company and it's the first impression that the company gets from the new hire. It's also a very important process for the new hire to get to know the company and the people in it.

Read how to do it with CoCMD Here

Playbooks

Usually, every team has a set of routines and procedures that are executed on a regular basis.
This can also be called "playbooks" or "runbooks".

CoCMD can help you automate these routines and make them available to your team.

Read how to do it with CoCMD Here

Codebase CMDOps

Any project can have CMDOps (Command Line Operations). Lets say for example, your project requires some installation steps, or you want to add some shortcuts for your team or community to use.

Read how to do it with CoCMD Here

Cocmd-Hub

I created several public CoCMD Packages for Programming, Devops, Databases and general Machine setup.
You can visit in https://cocmd.org/#hub
or run

cocmd install

What Packages to install?:
> [ ] zsh
  [ ] vscode
  [ ] jenkins
  [ ] impala
  [ ] git
  [ ] k8s
  [ ] python
  [ ] osx
  [ ] node
  [ ] go
  [ ] rust
  [ ] aws-s3
  [ ] docker
  [ ] prometheus

and get the full list

Create your own

add cocmd.yaml to your project
follow along the guidelines https://cocmd.org/docs/packages/package-specification (still WIP)

Now your team/community can run

cocmd install <your-repo>

and get your onboarding and playbooks.

Visit the project website on https://cocmd.org
Visit the github repo on https://github.com/cocmd

Help wanted

https://github.com/cocmd/cocmd/issues
there are interesting issues to work on, help me :)

contact me on mzsrtgzr2@gmail.com

Use the "right words" in interviews

MoRoth — Wed, 11 Oct 2023 07:17:13 +0000

In one of my recent interviews as a Senior Software Engineer, I realized that no matter how experienced you are, it's essential to understand what interviewers are looking to hear.

Before the interview, I did my research and discovered that they were interested not only in technical skills but also in problem-solving abilities, teamwork, and effective communication.

My train of thought coming to my next interview was that the human brain is a remarkable organ, often likened to a vast library filled with books of words, each representing a key to an entire world of knowledge. It possesses an incredible capacity to correlate words and concepts, allowing us to access these worlds of information effortlessly.
A single word can act as a gateway to a complex web of interconnected ideas.

In the high-stakes world of interviews, where time is limited, and attention is at a premium, the choice of words can make all the difference.

Interviews are intriguing scenarios where:

time and focus are in short supply
Your knowledge and expertise are crucial
but the challenge lies in selecting the right words.

These words serve as bridges, linking your responses to the interviewer's own experiences and understanding.
When this connection occurs, it's like forging a meaningful connection in a limited timeframe, greatly boosting your success rate in interviews

In this context, think of your words as keys to unlock a shared understanding. Let's say you're discussing a complex data migration project. Instead of diving into the intricate technical details, try to use some of the words i wrote down in the following list (or create your own...)

project success metric - daily active something… etc
immutable / mutable
“relational” / nosql
consistency
“blob database” - s3
volume / scale - talk about data storage volume, data access volume
“linear scale”
partitioning
index
sorting
pagination
websocket when needed (streaming, pushes)
cdn, edge
bottleneck
load balancing + lb metrics
io / cpu
data replication, avilablability, downtime
storage data tiers - according to lifetime etc
event bus
“ACID requirements”

and overall good terms to put in your explanations:
good terms to. use

“good question”
“lookout for”
“tradeoffs”
“Horizontally scaled”
“offloading”

I know that for non-native english speaker using these terms is not natural.

How to come up with your own list?
Creating your own list is a simple process. Begin by delving into YouTube videos within your niche of expertise. Pay close attention to words and phrases that instantly resonate with you, clarifying complex explanations. These are the very terms you'll want to weave into your interview responses.

For me, these type of Videos were perfect - https://www.youtube.com/watch?v=M6UZ7pVD-rQ&ab_channel=IGotAnOffer%3AEngineering

Some final words -

I believe the secret sauce of Great interviews is making sure your explanations "click" with the interviewer. It's not just about what you say, but how you say it. By choosing words that resonate with their experience, you're not only showcasing your expertise but also building a connection. So, when your words "click," your interview rocks!

cheapest code signing

MoRoth — Thu, 14 Sep 2023 06:44:58 +0000

nice table to sort them out by pricing and features

https://www.ssldragon.com/ssl-certificates/filters/secures/135/?orderby=price

Causal Profiling and Why

MoRoth — Wed, 28 Sep 2022 12:05:57 +0000

Notes for interesting lecture on Performance and Profiling.

Performance used to be easy (Moore's law) - just wait for hardware to improve and boom, there is a Performance boost

Performance may be changed accidentally by input parameters, environment (stack/heap size), link order, function address, cache sizes etc - this is called "Layout change"
Stabilizer is repeatedly changing all of this things while running and measuring Performance
Explains something really interesting about "Null Hypothesis Significance" testing.

Creating one random Layout and testing is not enough. It's required to do it many times with different Layouts to make sure the improvement is not by chance.
If the speedup occures in low chance like <0.05 we reject the speedup.

Profiler - measures end to end runtime of components

Profilers are not good at finding out important bottlenecks. Very general.

Causal Profiler

This is like an oracle saying where to focus Engineering efforts to get whole Performance improvement for the App.

Virtual speedup - slow other components, this simulates one component being relatively faster.
you need to mark where to measure ("Progress Point").

Lookout for Interviews Lies 🤨

MoRoth — Wed, 17 Nov 2021 13:30:28 +0000

Setting the stage

My last places of work were really really ok, however I found that there was a gap between what I expected of the job and what happened in reality. And it can be pinned down to the very first interview call.

The story goes like this - an "interviewer" tell you things you want to hear, name-dropping technologies, architectures and promises. Many times, these are future plans, wishes etc. I'm not saying these are intentioned lies. It's describing the job in future/partial terms, not talking about things you "don't want to hear" while hoping you will think "hey, this is exactly what I'm looking for, very soon this all be true".

BTW there are many variations to this phenomenon - not just to technologies/architectures considerations... it can be really anything related to your day-to-day - responsibilities, terms of employment, work place conditions, team properties (size etc), product roadmap, people interactions and the list goes on.

I will focus here on Developers interviews, but you can infer it to other domains as well

Spoilers

In reality, months go by (even years) and the promises don't happen.
You mostly do things you didn't think you'd do.
Every deviation of "the plan" is excused with a super important "Business requirement".

Do the thing

Every "interviewer" (recruiter, hr-rep, founder) you talk to - describe a very bright picture - fitting the story to your "ultimate" job description.

Pitfalls

Here are some aspects in the story you need to check before pursuing a job offer from any company:

Lookout for "future plans"

"This is going to be a distributed system...
"The next release will be with XXX technology...
"Eventually, there will be X people on the team...
Many many aspects of your job can fall in these gaps - plans & promises with no definite roadmap or timeline. This is highly risky.

Lookout for % of focus

"The team works on...
"The system uses...

Even if something is talked about during an interview it doesn't mean YOU will do that. This can easily be "name-dropping" for things you want to hear.

Lookout for on-going tasks/ technical debt / legacy

NOT talking about on-going tasks, legacy system - should trigger a red flag.
It can be very much not-intentional that an interviewer won't mention these things as they are not "sexy" to an interviewee. You need to make sure you understand what will be required of you for current systems that run.

Fireproof your decision

Here is a list of questions to ask the interviewer to make sure the job description is what you think it is.

Disclaimer: Not all companies/people will like you asking those questions. They can think you won't be a team player or you have some hidden agenda just doing things you love or interest you. Try to get the answers to these questions even without actually asking out loud - pick up clues and make self notes.

Describe the team's last Sprint / assignments?
What will be my first "serious" task?
How do you do a... b... c... (in present tense)
Look for facts / not promises or opinions -
- last week we did a....
- this sprint we do x,y,z...
- last bonus check was X% of salary....
- we hired 2 new people last month...
- last design review took X hours....
- We deploy X time a week...
Talk to a team mate in your level
- what is your day-to-day like?
- did you feel you have impact on decisions - new tech/framework/features?
- How many "on call"s you receive?
- How did your manager received a suggestion you had?
- how much time you spend coding / discussions?
- How do you solve issues in the team? - ask for a concrete example
- How do you solve issues with your manager? - ask for a concrete example
Talk to your direct manager/supervisor

Note: Some companies hide behind an NDA (non disclosure agreement) and can refuse to answer/show what you ask - This is straight up BS and I would recommend avoiding those companies altogether. NDA is not really a reason not to answer candidate's questions.

Why does this happen?

Human nature

It's the interviewers job to bring good people in.
Once he/she decided you fit - it's human nature avoid telling you things that will disarm you or even change the facts a little bit. It's mostly unconscious decisions (I want to believe that...).
BTW, same goes for you: You're not telling things about yourself the other side doesn't want to hear, right?

Urgent Business requirements vs Advancing / Fast vs Right

Every project has a planned roadmap and things change as we go along. I'm not talking about complete Pivoting the company... I'm talking about what we do in the upcoming Sprint. this is ok of course.

However, you don't want to be in a place of work where there is 0 new features, 0 new technology and 0 personal development.

There is a spectrum of companies that behave differently -

keep a technology/code/etc even it doesn't fit anymore just because it requires "learning".
avoiding refactoring "old/bad things".
not listening to any suggestion for improvement by anyone.

When a company/team is always busy with super urgent Business requirements it can never advance technically - which means you won't too.

Founders and Startup companies

The behaviour I described varies between companies (and teams). Take an extra step of cautious with Startup companies and the Founders that run them.

Founders are extremely optimistic

Founders are the most optimistic people regarding the company/product/future. When it comes to avoid talking about un-pleasant issues, technical debt, decision-making, roadmap - you can hardly trust them to come forward and bring up hard things to talk about.

Unknowns

Mostly, there is no way to verify the any thing they drop along a conversation.

It's mostly business

As I already wrote, there is a spectrum of companies that will prefer Business requirements over "Innovation/New approaches/etc" - Most of times, Startup companies are 100% about Business requirements as they have very specific milestones and drastic changes in requirements.
For example:

hard code something for this client
do that thing manually, you know what? let's do this manually every week - mark it in your calendar, cheers.
we don't need to pay for this service we really need every day, let's all work like cavemen in the 90th...
hey this database is a better fit for this - no time now
refactor this module? - no time now

"It will be ok"

When you spell out the words "it will be ok..." before you consider taking an offer - really think about that. YOU are taking the risk here. will it really be ok??? because it won't.
You need to start your job feeling "It will be awesome"

The 20->80 rule

You start the job interview process - after the first call you are 20% certain that you understand the job description and what your future there will be like.

You need to take those 20% certainty and make it 80% (as it never can be 100% until you really work there for some time).
How?

ask the hard questions
check online - glassdoor, hr groups etc
talk to past employees

Best of luck!

FSX for Lustre

MoRoth — Mon, 19 Oct 2020 18:44:25 +0000

AWS architecture of Sisense with FSX for Lustre for real time hot data queries with several datasources.

https://www.youtube.com/watch?v=SxFag4CMWU8&list=PLhr1KZpdzukdeX8mQ2qO73bg6UKQHYsHb&index=2&ab_channel=AmazonWebServices

https://aws.amazon.com/fsx/lustre/

Spark Journey begins...

MoRoth — Mon, 28 Sep 2020 10:14:44 +0000

As an engineer with several years of experience in Backend and Frontend projects it feels like the next natural step is big data challenges.
In the big data world I expect to find computing, IO and scaling challenges not usually found in ordinary/plain/textbook architectures.

I decided that Spark is the best way to get started. Specifically - the Databricks certification, which is focused on Spark programming and architecture.

My game plan to pass the Databricks spark certification is to:

Read "Learning Spark Lightning fast big data analysis" book and work through all the examples + summarising important insights and lessons so I can repeat those later.
Go over the skeletons of Databricks Developer course that I found on GitHub from 15 months ago. Should be pretty updated - https://github.com/vivek-bombatkar/spark-training + https://github.com/vivek-bombatkar/Spark-with-Python---My-learning-notes-
Going through example questions.

Please, If you can advice on any source of preparation - write in the comments it will help me.

I will update as I go for others (and myself).

Learning Schedule

Theory

Reading throughly the book "Learning Spark Lightning-fast..."
I think it's reasonable to go through 2 chapters per week.
this means: reading, summarizing and running important code snippets on my own.

Week 1
Chapter 3
Chapter 4

Week 2
Chapter 5
Chapter 6

Week 3
Chapter 7
Chapter 8

Week 4
Chapter 9
Chapter 10

Week 5
Chapter 11 - Quick read it's not that important

Hands on coding

Basics (4 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-pyspark

https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-from-pandas-to-spark

https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-weather-df

https://github.com/vivek-bombatkar/spark-training/blob/master/spark-python/jupyter-weather-df/Weather%20Analysis%20Exercise.ipynb

Advanced topics (10 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced

Windows (4 notebook)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-windows

https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-windows

UDF (3 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-udf

Spark execution(1 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-execution

Caching (3 notebooks)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-caching

Pivoting (1 notebook)
https://github.com/vivek-bombatkar/spark-training/tree/master/spark-python/jupyter-advanced-pivoting

total 26 notebooks
I hope to do 3-4 notebooks per week (some will be easy some harder, so taking the average). This will result in 8 weeks of going through the notebooks. Learning what I'm missing etc.

Everything should take 3 months until I'm ready for the exam.

Books PDFs

Learning Spark: Lightning-Fast Big Data Analysis
First Edition
https://b-ok.asia/book/2493162/9b8d4f?dsource=recommend
Second Edition
https://laptrinhx.com/learning-spark-lightning-fast-data-analytics-2nd-edition-436517903/

Spark: The Definitive Guide: Big Data Processing Made Simple
https://b-ok.asia/book/3505368/f04c83?regionChanged

Spark in Action
https://b-ok.asia/book/3502170/d3383b

Don't mess with default Python scoping

MoRoth — Sat, 05 Sep 2020 10:29:02 +0000

Using the global and nonlocal keywords shouldn't happen too often imho. I think it's hard to debug and read. Newcomers may not see the big picture as the original writer of a piece of code. So, when those are used - keep very tight set of tests to make sure it won't be broken in the future and also - more understandable. I tend to read tests to understand how to use.
Why not use them? - I think it makes code less modular. So Use it only in the very same Unit of code.

For example,

https://github.com/aws-samples/aws-concurrent-data-orchestration-pipeline-emr-livy/blob/master/dags/airflowlib/emr_lib.py


def client(region_name):
    global emr  # <-- (1)
    emr = boto3.client ('emr', region_name=region_name)

... other functions...

def get_cluster_dns(cluster_id):
    response = emr.describe_cluster(ClusterId=cluster_id)
    return response['Cluster']['MasterPublicDnsName']

... other functions...

I won't call this a "good" example (I would rather gather those functions to a Class with an emr property) - but at least it's "understandable". The writer intended for the "client" method to be called first for initialization before any other function in this file.

If you can think of other examples to when it's right to use global and nonlocal, please comment in the discussion.

Best Python GIL overview yet

MoRoth — Fri, 04 Sep 2020 17:35:39 +0000

Most of Python experienced programmers know what is the GIL (Global Interpreter Lock) keeping Python code thread safe no matter what operations we make on shared memory.
Most of python programmers know when to use Multiprocessing vs Multithreading and Asyncio for intensive IO operations.

This article really opened my eyes to why GIL chosen as ultimate solution in CPython's interpreter architecture and how it affected the community when extending CPython/Python:

https://realpython.com/python-gil

In this article I found great example to how CPython is not thread-safe - look for the example with "reference counter", beautiful example.

What I really like is how the writer suggests a potential solution: Use locks for all Objects in memory and lock/unlock every operation it goes through.

In a nutshell, this solution is problematic because it will create dead locks and performance degradation (locks require cpu) - which can turn away many developers.
So the GIL is THE ONE LOCK for all threading operation that can be released by CPYTHON code (mostly happens on IO calls and Thread exit).

Do your own Python's @property

MoRoth — Mon, 18 May 2020 13:06:02 +0000

After reading this chapter in "Expert Python Programming" -
Advanced Attrbiute Access patterns - https://subscription.packtpub.com/book/application_development/9781789808896/4/ch04lvl1sec39/advanced-attribute-access-patterns
decided to implement @property partially.
Here is the final result - https://gist.github.com/mzsrtgzr2/46975900b8d7a8e6f8cc51d1a40ca940

class myproperty:
    def __init__(self, func):
        self.f = func

    def setter(self, func):
        self.setter_func = func
        return self # (1) why do we need this? read below on 

    def __get__(self, instance, klass):
        return self.f(instance or klass)

    def __set__(self, instance, value):
        self.setter_func(instance, value)


class Foo:
    def __init__(self):
        self._val=0

    @myproperty
    def temperature(self):
        return self._val

    @temperature.setter
    def temperature(self, value):  # (2) why need same name as getter?
        print('setting to', value)
        self._val = value

ins = Foo()

print(int.temperature)  # 0
ins.temperature = 6
print(int.temperature)  # 6

It's important to understand how decorators work. This is the key to understand this code. The decorated function is REPLACED with an instance of your decorator (or a function in case you use a decorator function and not class). That being said, you can add functionality to your decorator class like __get__ and __set__, like in the code above.
__get__ - doing a "read" operation on the decorator class.
__set__ - doing a "write" operation on the decorator class.

This is another way to look at decorators - it can be used for "data descriptors" (when you access a field, as in ins.temperature) and not just as function wrappers (implemented with __call__, as in ins.temperature()).

Important - why is the decorated set method should also be named temperature?

A decorator works like this:

@dec 
def func1:
   pass

actually converts to

func1 = dec(func1)

so the name of the function is important.

What happens if we don't use the same name in (2) or drop the return self statement at the end of our decorator: what used to be temperature in our class is overridden to None - losing all the functionality we thought we built. We really happens:

class Foo:
    def __init__(self):
        self._val=0

    def temperature(self):
        return self._val
    temperature = myproperty(temperature)  # instantiating myproperty class

    def temperature(self, value):  # (2) why need same name as getter?
        print('setting to', value)
        self._val = value

    temperature = temperature.setter(temperature)

In this example, the property field temperature is always the reference name and can't change it.