DEV Community: Bisvarup Mukherjee

Running apache flink locally mac m1

Bisvarup Mukherjee — Sat, 30 Jul 2022 18:17:00 +0000

I have an intel mac m1 air and I want to install and run apache flink locally. Instead of installing correcrt jdk version and flink version, etc. I used docker.

Step 1. Create docker network if it doesn't exist
todo: check if it exists
docker network create flink-network

Step 2. Set env variable in terminal 1
FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"

Step 3. Launch JobManager in terminal 1

 docker run \
    --rm \
    --name=jobmanager \
    --network flink-network \
    --publish 8081:8081 \
    --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
    flink:latest jobmanager

Step 4. Set env variable in terminal 2
FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"

Step 5. Launch TaskManager in terminal 2

docker run \
    --rm \
    --name=taskmanager \
    --network flink-network \
    --env FLINK_PROPERTIES="${FLINK_PROPERTIES}" \
    flink:latest taskmanager

Step 6. Open localhost UI
http://localhost:8081/#/overview

Step 7. Submit a job
./bin/flink run examples/streaming/WordCount.jar
This step assumes you have downloaded flink, we are not using their runtime execution, but just their examples and submitting jobs.

Step 8. Install maven (mvn), this is needed for scaffolding some stuff later (not sure about this, todo)
brew install maven

Step 9. Scaffold sample project

 mvn archetype:generate \
    -DarchetypeGroupId=org.apache.flink \
    -DarchetypeArtifactId=flink-walkthrough-datastream-java \
    -DarchetypeVersion=1.16-SNAPSHOT \
    -DgroupId=frauddetection \
    -DartifactId=frauddetection \
    -Dversion=0.1 \
    -Dpackage=spendreport \
    -DinteractiveMode=false

Dev advice from 4year exp guy

Bisvarup Mukherjee — Thu, 09 Jun 2022 04:02:12 +0000

In order to save time, don't give quick meaningless replies to issue tickets. The number of back and forth that will happen if your reply does not makes sense is less than you actually going in and making the fix.
Take up less work if needed but spend 100% concentration on the work you take up

Creating mid-sized somewhat ambiguous projects

Bisvarup Mukherjee — Sat, 04 Jun 2022 05:54:43 +0000

List the requirements of the system
Create an actor diagram
List the happy path
List the alternate paths
Sort the requirements in p0, p1 & p2
Further sort the requirements in essence, risk, & complexity
Create a sequence/flow diagram for happy path
Create sequence/flow diagram for alternate path
Focus on the feature that is most risky and that is not blocked on something first.
Once risk is reduced by some means (may be class diagram / flow diagram) move on to the other areas
Try to find commonality b/w the entities of the system
Reduce complexity of the system and features

Recommended way of creating software

Bisvarup Mukherjee — Mon, 30 May 2022 03:46:34 +0000

Inspired from some talks and books, here is the framework I would define

Part I

Write some tests of the part of the feature you want to make
Make the tests fail (duh! there is no code to pass it)
Write minimum code to pass the test cases
Refactor

Part II
From OOA & D book

Make the software/feature do what the customer wants it to do
Apply OO principles to add flexibility
Strive for a maintainable & flexible design

From Part I and Part II, the point 1 would correlate with Part I, points 1 & 2, Part II points 2,3 would correlate with Part I point 3, the refactor phase.

OO principles from OOA & D

Bisvarup Mukherjee — Mon, 30 May 2022 03:33:36 +0000

Encapsulate what varies
Code to an interface rather than to an implementation
Each class in your application should have one reason to change
Classes are about behaviour and functionality

LLD of a system

Bisvarup Mukherjee — Mon, 30 May 2022 03:31:55 +0000

Create a set of requirements that your system would handle.
Create the use case diagram of the system, and make sure all the requirements are properly handled in the use case diagram.
Do domain analysis, aka describe the problem in terms that the customer will be able to understand

Better estimate your features

Bisvarup Mukherjee — Tue, 24 May 2022 16:05:45 +0000

Take time to consider planning the project, breaking down into tasks, & creating some groundwork. (2-3D)
Writing in TDD format. Red-green-refactor. (Depends on project and dev experience)
Design completeness. (2-3D)
Design rework. (Depends)
Content completeness. (1-2D)
Content rework. (Depends)
Creating proper test situation for local/stage testing (sometimes for some features there needs to be some groundwork done before that feature can be tested). (Depends, sometimes .5D, to 3-4D)
Oncall shift.
Unseen p0 items. (Depends)
Vacation timeoff. (Depends)
Sick timeoff. (Depends)
Coworkers not having access to environment. (3-7D)
KT incase you are taking timeoff and there is a replacement in your place. (1D)
Coworkers who you have dependency on taking timeoffs. (2-3D)
Design reveiw.(2-3D)
Content review.(1-2D)
Internal testing.(1W)
Documentation.(2-3D)
Create insights & queries.
Craft feature emails to send to eng team (1D)
Create PPTs and other forms of comms for higher leadership to demo the feature (1-2D).
Adhoc interviews if you need to take them
Mandatory adhoc courses
Your new code breaking existing test cases

p.s1 Not all of these may be applicable for every feature, for example a small feature with <=2 storypoint may not need all of these considerations, but this framework would still help.

p.s2 The estimates given above are assuming a mid sized project (2-3 months).

How to get into big tech

Bisvarup Mukherjee — Sat, 21 May 2022 12:49:40 +0000

Right now I am SWE 2 at Uber and previously I was a UI 2 at Flipkart, so here is my opinion on a few questions I receive.

Q) How do you get interview calls?

Ans)From my personal experience, having a good college background helps in the initial years. If that is not the case for you, then getting you foot into one of the decent size brands also helps.

From my personal experience, once I got into Flipkart, I was getting interview calls from renowned companies that I was not getting earlier.

Now breaking into the first renowned company is the trick for many out there. For that I suggest to apply from all channels aka

Apply on job postings on the company page
Apply on Naukri
Apply on other job portals
Directly ping HRs
Directly ping Engineering managers, many of them say they are hiring for their team.
Ping engineers working on the company you are targeting.

Points 4,5,6 may get you referrals and that would help you get interviews.

Closing thought, I feel it requires a bit of luck as well to get the interview call, so keep trying.

Q) What kind of questions are asked during the interview? Specially DS questions?

Ans) Every specialization would have it's own set of rounds.

For example for FE (frontend engineering) it maybe machine coding round (where they may ask you to build some project) javascript round (where they will assess your FE skills), design and architecture round (where they will test your system design skills) coding round(where they may ask you DS algo based questions) and finally a manager round.

For DS algo based questions, I just did as many questions I could do (around 80 within a time period of 2-3 months) of leetcode mediums.
Just make sure you touch the popular topics and cover some breadth. That should be enough for the most part.