DEV Community: Pradeep Sharma

The worst and best side of building in public

Pradeep Sharma — Thu, 01 Jul 2021 14:27:46 +0000

For the past 3 months, I have been working on an open-source project Jina. In these 3 months, I have seen the best and the worst of the Internet. Sharing these anecdotes so they serve as a lesson for everyone building in public and for the open-source community.

Anecdote 1 - May 2021

Just like every month, Max starts the public zoom meeting to host contributors of open-source project- Jina. 200+ people join in. Max starts sharing the big news about the upcoming major 2.0 release. He’s sharing this news publicly for the first time. Max’s eyes are brimming with excitement, it’s a major stepping stone for the project. We all have been coding, designing and planning for many months to reach here.

While Max is on the second slide of his presentation, loud music suddenly starts playing and Max’s voice fades away in the background. I hear the racist abuse in the music. I see a drawing being made up on the screen. Whoever is drawing that, is definitely not good at drawing but you can clearly understand that it looks like the private parts of a man.

This is the clip from the event where it happened WARNING! NSFW Content in the video.

We got zoombombed and that was embarrassing! In the meeting, there were people from diverse backgrounds and gender.

Building an open-source project is challenging on multiple fronts other than the core technical challenges:

Understand what needs to be built and improved
Help people understand the project
Educate people about the domain
Reach out to people who might benefit from your project
Collaborate with other contributors
Deal with issues and pull requests
Deal with outdated versions/docs
Deal with different opinions
Sometimes deal with jerks like the ones who zoombombed us

The list is long! The point is

Open source is hard!

Making fun of someone’s work is easy!

Anecdote 2 - Jun 2021

After that zoombombing event, we did our best to not let trolls get in the way of our commitment to make this project successful and do that while being public and transparent about everything. But more than what we did, it is more about what the community did for us. We were overwhelmed with the support.

Fast forward 1 month

We doubled our growth
We were trending #1 on Github (the #1 out of 190 million repositories)

Jina made first public appearance in April, 2020 and now it has reached to

6,000 stars on Github
524 forks
155 contributors
1839 PRs

A big thank you for this amazing support 🙏

My learning

This is not the first time I experienced trolling. I have learned three things

It’s common to get trolled on the internet, there’s nothing personal about that
The best answer to trolling is to ignore them and keep doing your best (easier said than done)
The open-source community is awesome. For every 1 troll you have 6,000 supporters.

Cheers to Jina contributors! Thank you for the support.

Trolling is a real issue that we all need to collectively deal with.

Conclusion

Build in public, trolls are temporary and you'll get more support than you asked for.

Having said that, I seek your opinion: When building in public, how can we make communities safer for people, especially for people from underrepresented backgrounds?

What is Neural Search?

Pradeep Sharma — Thu, 13 May 2021 11:52:18 +0000

What is Neural Search and how to get started with it without prior experience in Machine Learning

TL;DR: Neural Search is a new approach to retrieving information using neural networks. Traditional techniques to search typically meant writing rules to “understand” the data being searched and return the best results. But with neural search, developers don’t need to wrack their brains for these rules; The system learns the rules by itself and gets better as it goes along. Even developers who don’t know machine learning can quickly build a search engine using open-source frameworks such as Jina.

Table of contents

What is Neural Search?
Evolution of search methods
Rules vs Neural Networks
Applications Of Neural Search
Get started with Neural Search

What is Neural Search?

There is a massive amount of data on the web; how can we effectively search through it for relevant information? And it’s not just the web where we need it: Our computers store terabytes of company and personal data that we need to work with; we need effective search to get our day-to-day job done. And what do I mean by effective search

Can we go beyond just matching keywords?
Can we search using natural language, just like we would write or speak?
Can we make the search smart enough to forgive our minor mistakes?
Can we search for things that aren’t an exact match but are “close enough”?

We can answer all those questions with one word: Yes. To understand how, we need to enter the world of Natural Language Processing. NLP is a field of computer science that deals with analyzing natural language data, like the conversations people have every day.

NLP is the foundation of intelligent search, and we have seen three different approaches in this field as follows.

Evolution of search methods

Rules (1950–1990s) Complex handwritten rules that emulate Natural Language Understanding. Drawbacks: Handwritten rules can only be made more accurate by increasing their complexity, which is a much more difficult task that becomes unmanageable over time.
Statistics (1990s — 2010s) Probabilistic decisions based on weights, machine learning and feature engineering. Creating and managing rules was solved with machine learning, where the system automatically learns rules by analysing large real-world texts. Drawbacks: These statistical methods require elaborate feature engineering.
Neural Networks (Present) Advanced machine learning methods such as deep neural networks and representation learning. Since 2015, statistical methods have been largely abandoned, and there has been a shift to neural networks in machine learning. Popular techniques using this method make it a more accurate and a scalable alternative. It involves
- Use of word embeddings to capture semantic properties of words
- Focus on end-to-end learning of higher-level tasks (e.g., question answering)

When you use Neural Networks to make your search smarter, we call this a Neural Search System. And as you will see, it addresses some of the critical shortcomings of other methods.

Note that the applications of Neural Search are not just limited to text. It goes well beyond what NLP covers. With neural search, we get additional capabilities to search images, audio, video, etc.

Let’s look at a comparison of the extreme ends of search methods — “Rules” vs “Neural Networks”:

Rules (Symbolic Search) vs Neural Networks (Neural Search)

While the Neural Search method has become more widespread since 2015, and should be the primary focus area of any new search system, we shouldn’t completely rule out Symbolic (rule-based) Search methods. In fact, using a combination of Neural Search and Symbolic Search may result in optimized results.

Let’s look at some of the powerful applications of Neural Search

Applications Of Neural Search

Semantic search

🔍 addidsa trosers

(misspelled brand and category, still returns relevant results similar to query “adidas trousers”)

Search between data types

With Neural Search, you can use one kind of data to search another kind of data, for example using text to search for images, or audio to search for video.

Search with multiple data types

With Neural Search, you can build queries with multiple query data types e.g. search images with text+image

Get started with Neural Search

For rule-based searches, Apache Solr, Elasticsearch, and Lucene are the de-facto solutions. On the other hand, Neural Search is relatively new on the scene, there aren’t so many off-the-shelf packages. Not to mention, training the neural network for such a system requires quite a bit of data. These challenges can be solved using Jina, an open-source neural search framework. To get started with building your own Neural Search system using Jina:

References

Neural Search term is less academic form of the term Neural Information Retrieval which first appeared during a research workshop in 2016
Learning to rank
Natural Language Processing
Neural Network
Building cross-lingual product search using Tensorflow
What is neural search and why should you care
Information Retrieval
Related to Information Retrieval
Search Engine Technology
Introduction to neural Information Retrieval
Content based image retrieval
An anatomy for Neural Search engine
How google search works, and a more in-depth video

7 tips & tools for effective "teamwork from home"

Pradeep Sharma — Mon, 23 Mar 2020 13:02:04 +0000

My learning of running a community of 15000 highly productive developers working from home. I have been working from home since 2016. I know some things about "work from home" and "teamwork" that you can trust. I condense my learning in 7 important and easy to follow practices.

While many of us are confident to work from home effectively as an individual. It is effective teamwork that drives output and keeps the organizations running. Ineffective teamwork leads to stress.

So how can you be a great team player when working from home? Follow these 7 simple tips

Have a dedicated space for work - define boundaries between your work space and family space
Communicate proactively - when you're working remotely, you need to communicate more effectively to make sure your team knows how you're progressing/thinking to achieve your work goals. Communicate more and be more transparent. Quick messages on team chat such as "hi, starting my work for the day" or communicating about status proactively even if it's not good e.g. "I didn't achieve today's goals, did some research on__" are example of an awesome remote team member
Always be on time for planned team calls, even 2 min late is LATE and can impact team morale negatively faster than coronavirus
Have some overlapping time with team - agree to have a fixed overlapping work time(at least 1-2 hrs everyday) with your team members so you can resolve those quick queries, brainstorm and move forward fast
Decide your work timings, break timings and family time; communicate the same with your family members, clearly ask them to not disturb and wait till your work time is over
Have an exercise routine - it can be running or workout at home or at gym. Do it for at least 10 mins everyday
Add some reliable tools in your toolbox- To make your life easier, here's a list of essential tools for the best "work from home" experience

Zoom for online face to face meetings
Google calendar to block your work and family time
Calendly to make meeting scheduling easier
Developer Diary every person is different and the best productivity advice is the one that comes from yourself. You can accelerate your awareness using this tool by understanding your work habits. This will help you take the right decisions to boost your and team's productivity.

Would love to hear your experience of working from home

Simple command to block distracting websites [Productivity]

Pradeep Sharma — Sun, 01 Oct 2017 09:47:21 +0000

For Linux based systems

Step 1: Open terminal
Step 2: Open the hosts file sudo vi /etc/hosts
Step 3: Add websites you want to block and save

127.0.0.1 facebook.com
127.0.0.1 www.facebook.com
127.0.0.1 web.facebook.com

For windows

The host file location(step 2) is> C:\Windows\System32\drivers\etc
everything else is the same

I'm tweeting similar things on twitter. Follow me and feel free to message me.

7 fucking rules for developers to get shit done

Pradeep Sharma — Wed, 20 Sep 2017 16:54:07 +0000

I am obsessed about being more productive and believe that everyone should be

Think about the amount of work you generally do in 5 days, what if you can do the same in 3 days. With better quality and without being stressed. I have done this, so can you.
This is what I have observed and many of us would agree, we start a project with enthusiasm but slowly we lose the momentum, we leave things for the last moment and eventually we miss the deadlines.
After many failures, frustration, giving up my smartphone for 6+ months. I was able to fine-tune my productivity. You don't have to go through the drill I have gone through. I have summarized the simple rules we all should follow. This will dramatically change how much we can achieve.

Everyone has same 24 hours, it’s about how you use them.

The 7 fucking rules

Know your fucking priorities: If this is your work time, add this article to your “To Read List and leave this post right now. There are tons of important and necessary things you have to do, but you can do only one thing at a time, which one should it be, you’re fully capable of making that right choice if you want to.
Be fucking mindful: Be mindful about what you are doing in “The moment”. This is easier said than done, only practice can help. Know that you will lose motivation and momentum at some point of time, you need to be ready to face those moments. There will be moments when a quick break to scroll Facebook endlessly may feel satisfying and harmless, after sometime you wouldn’t even realize how much time you have wasted just on that. Always be fucking mindful about what you’re doing in the present moment. Take a deep breath, be present in the moment.
Fucking eliminate interruptions: Block all possible interruptions and distractions during work time. Phone, news, fb, twitter, email, colleagues, blog, reddit, product hunt, game and everything that can possibly ask for your attention even for a moment. Make peace with the fear of missing out.
Have a big fucking picture for the day: Have an overview of what you are going to achieve today. Unplanned things will come your way, don’t worry, be like water. When we throw a pebble in the water, it gets disturbed but gets to it’s natural calm again easily.
Fucking focus on the first step: Do not fear the overwhelming long to-do lists. Decide your immediate goal and deadline to finish it.
Fucking block time for work and break: Divide the time you have in blocks of work-time and break-time. Generally 5 mins break after each 25 mins work can boost your creativity(known as pomodoro technique), you can figure out what suits you the best, if it’s 1 hour work and then 10 mins break, do that then. But do not work for long hours continuously. Leaving work for a break may seem unnecessary initially, it’s not. Having an unplanned break when you’re stuck may seem necessary, it’s not. Have a monk-like focus, be fully available for the task.
Fucking pause and think: Even after doing your best you’ll miss some deadlines. Now it’s time to pause and think objectively for a moment 1. Why is it taking time? 2. Can you finish it quickly by giving some extra time? 3. Can you move on to next task without completing this one and adding it to to-do list again? 4. To finish this task, do you need to do extra things e.g. learning a new concept, research or experiments. Can you allot time separately for that? 5. Are you not moving forward because you want to make it perfect? Can this perfectionism wait?

Some might be thinking there’s nothing new here in the article and some might be overwhelmed by the information. It is not about knowing, it's about actually doing it, doing it consistently. If I can do it, I’m sure, you can do it too. Just remember the rules.

When we are having fun like watching a movie, having a party etc. we create a stimulating environment by switching off lights, playing some music and many other things. We don’t make similar efforts for work while that’s what we do during most of our time. There is always a scope to do better.

I can be reached on twitter.
Update: I run a cult of top developers passionate about remote work called Invide. Do check it out and if you want to join this, use referral code: "ThePracticalDev" to speed up the curation process. It's 100% free but screening takes some time to maintain the standards.

7 Design lessons for a developer

Pradeep Sharma — Tue, 03 Jan 2017 12:50:07 +0000

I am a developer, I had been coding for last 6 years, building android apps and web apps. I had always faced problem with design thinking and did not even attempt to fix the design of my apps. But recently I started reading “The design of everyday things by Don Norman and worked on an app. I would like to list down my 7 extremely important lessons from last 6 months.

Users make mistakes all the time even with the simplest things. It's your job to minimize those errors and make them feel good about using your product not blaming themselves.
Make at least 3 user personas and keep them in front of your eyes all the time.
Assume that user can store maximum 3 things in their short term memory. Design the flow such as it does not exhaust the user's brain power, no matter how trivial those things sound, just 3 things.
Read about mental models.
The navigation and actions should match the target user's mental models.
Feedback is important for each action user performs, user should always get the affirmation of their actions and know the outcome of their actions instantly. Good feedback removes the cognitive load (e.g. What happened to the action that I performed?), bad or no feedback does the opposite.
When in doubt, talk to the users.

If you have any suggestions, I appreciate sharing it with me through comments or tweet. Follow me on twitter if you want to see more things in your feed related to programming, tech products, team building, productivity and entrepreneurship.

Hi, I'm Pradeep Sharma

Pradeep Sharma — Tue, 03 Jan 2017 12:32:13 +0000

I have been coding for 6 years.

You can find me on Twitter as @pradeep_io

I live in Delhi.

I mostly program in these languages: Javascript(node.js), Java, Python.

I am currently learning more about Machine Learning.

Nice to meet you.