DEV Community: Softhints

Convert String to Date in Pandas and Python

Softhints — Thu, 23 Jun 2022 07:56:25 +0000

If you need to convert string to dates in Python and Pandas you can check:

How to Convert String to DateTime in Pandas

It contains basic date conversion like:

pd.to_datetime(df['date'])

plus extra examples like:

custom date and time patterns
infer date pattern
dates different language locales
different date formats

pd.to_datetime(df['date'] , format='%Y%m%d HH:MM:SS', errors='ignore')

Also it will show if a given string contains dates with fuzzy matching.

Or mixed dates in a Pandas column:

    if '/' in date:
        return pd.to_datetime(date, format = '%Y/%m/%d')
    elif '-' in date:
        return pd.to_datetime(date, format = '%d-%m-%Y')
    else:
        return pd.to_datetime(date, format = '%d.%m.%Y')

SQL meet Data Science (Pandas) cheat sheet

Softhints — Mon, 20 Jun 2022 10:14:03 +0000

This is Pandas cheat sheet is intended for people who know SQL. The full version and high quality images are available here:

Pandas vs SQL Cheat Sheet

The cheat sheet is divided into different sections and aims to help beginners in SQL and/or Pandas. It offers easy navigation, examples and beautiful visualizations.

It includes most common SQL operations like:

Select
Where
Like, and, or
Group by
Join
Union
Limit
Update
Delete
Insert

Plus extra materials like:

Differences between SQL and Pandas
Pandas to SQL
Pandas from SQL
Pandas and SQL with SQLAlchemy and PyMySQL
Run SQL code in Pandas
Generate SQL statements in Pandas

Enjoy :)

This is the second part from a Pandas cheat sheet series.

Data Science Cheat Sheet for Python & Pandas

Softhints — Fri, 10 Jun 2022 13:21:47 +0000

A beautiful Python Cheat Sheet useful for the aspiring data scientists and contains ready-to-use codes for data wrangling.

The cheat sheet summarize the most commonly used Pandas features and APIs.

Data Science Cheat Sheet with Pandas and Python:

Pandas Cheat Sheet for Data Science - full cheat sheet

This cheat sheet will act as a crash course for Pandas beginners and help you with various fundamentals of Data Science. It can be used by experienced users as a quick reference.

The cheat sheet is part of series cheat sheets devoted to Data Science, Python and Pandas. Follow us for future versions on topics like:

data cleaning
data analytics
stats and math for data science
data wrangling
text processing

The full cheat sheet on the link above include

Organized in sections
Easy copy & paste
Fast navigation
Beautiful visualization

More resources

Example Section:

7 Good Habits for Successful Programmers

Softhints — Sat, 26 Sep 2020 08:31:21 +0000

Let's get honest: learning and mastering programming doesn't happen by accident or talent. In my fifteen years in the IT industry I have learned a lot about what works and what doesn't - I've wanted to be a better programmer and I was experimenting and searching for it.

The successful programmers who are able to reach the top and maintain it (finding nice job, working on interesting projects and getting high salary) share very similar habits which helped their progress. These good habits aren't anything secret or difficult, but they allow individuals to progress day by day.

I've compiled these habits of lucky professionals so we can all use them and enrich ourselves.

1. Minimize Time Lost

Efficiency is about producing a specific outcome with a minimum amount of waste, expense or unnecessary effort. It is of high importance to divide your professional life to days, days to hours and hours to minutes.

Practical advice

Reduce number of meetings
Decline meetings without agenda
Ignore long non professional talks
Minimize social media time at work
Don't multitask
Prioritize and organize on daily/weekly basis
Have long-term plan

Benefits

Less time sitting
Focused work
Dealing with harder problem
Learn faster

2. Sport! Sport! Sport!

Select an appropriate sport for yourself and practice it as long as you can. It's preferable to be a competitive sport if you like to enrich your social life and meet other people. To lead a highly effective professional life, a healthy body and mind is a must. Sport is the perfect tool against mind exhaustion, depression and lack of motivation.

Practical advice

Always find time for sport - don't skip it because of work
Get stuck on a hard task? - then go out for a walk or sport
Don't work immediately after sport routine
Find friends who share your sport interests
Use workdays for indoor sports and enjoy active weekends outside in nature
Do small sport routines at the office - even on the chair - exercise shoulders, legs etc

Benefits

Mental health
Self confidence
Leadership skills
Discipline
Stress management

3. Taking notes

I don't know everything which I did before but I know where to find it! Taking notes helps you retain more information and stay organized in highly complex areas like programming. The 15 minutes habit at the end of the working day is enough to make us more productive. Never start a new day without knowing where to start or searching the whole day for something from the last month.

Practical advice

Use special app like Typora
Research how other people do it
Daily updates on documentation
Once per week for analysis and reorganization
Keep only important things in great detail

Benefits

Stay organized
Spend less time searching
Use your brain and memory for important things

4. Learn new things

Make a habit of adding new skills whenever this is possible. The more you know the more creative you will be. The process of learning makes life more interesting and keeps our brain healthy. Have you ever experienced the case where you were doing something for years and then suddenly you find an easier and faster way to do it? Remember it's not about how much you know now but what you don't know which might improve your life.

Practical advice

Have fun while learning
Experiment with new ideas, tools, methods
Set apart time for discovery of new ideas
Socializing and learning go together
Test and adapt the new knowledge

Benefits

Becoming wiser
Creativity
Healthier brain
Broaden your mind

5. Meditate

Meditation is an ancient form of searching your inner-self and the answers related to it. This practice survived thousands of years and reached our society because it works and it's accessible. But don't be deceived that it's easy to make it right. It's better to search or ask experienced people about it. The habit of meditation a few minutes daily will help you solve harder programming challenges with less mind effort. Meditation is a habitual process of training your mind to focus and free yourself from thoughts.

Practical advice

Do it on daily base
Find nice peaceful place
Find what time during the day works best - morning
Use different meditation techniques and find what works better

Benefits

Improves Sleep
Controls Anxiety
Improves Emotional Health

6. Talk with more experienced

Find someone successful and learn more about him. Why do you think that he is successful? Is there something he has that you don't? Find more experienced and successful people in your company - drink a coffee with them or have a walk. They have much more experience than you and can share interesting stories and life hacks.

Practical advice

Become friend with such people
Ask questions which will help your growth as a person
Test and adapt useful experience of the others
Surround with successful people and learn from them

Benefits

New friends which might help you
Learn from others mistakes
Sometimes practical knowledge is more important than theoretical knowledge

7. Enjoy the nature

Spend more time outside in nature - everyday whenever you have this opportunity. Programming depends highly on memory and logical thinking. Walking in nature improves your short-term memory, frees your mind and boosts cognitive functioning. Forest walks are the most powerful way to fight with stressful states. As we know programming is related to stress - missing deadlines, hard problems, changing technologies.

Practical advice

Respect nature
Dress for the weather
Bring a bottle of water
Socialize with other people who enjoy nature
Meet friends in the parks not in the office buildings

Benefits

Improved physical and mental health
Reduce stress
Enjoy experience of the most complex system
Learn from the trees, birds, water
Best ideas and simplest solution come in nature

Conclusion

Best practices and good habits are easy to do for a few days or a week but almost impossible to stick to them for years. Remember 3 years are needed to learn a new habit and only 3 days to forget it! Be persistent and patient and only then you can get the benefits from those habits. Doing them from time to time is not enough if you like to go in the desired direction. I'll finish with:

What are we? What we do!
How good we are? How and how much we do it!

Please do share your comments and ideas on this interesting topic. I would like to learn more!

More interesting stories on softhints.com

Simple Guide to Deal with Painful Programming Headache

Softhints — Mon, 22 Jun 2020 16:15:46 +0000

Does programming give you a headache? Do you feel tired after few hours of coding? If so this article is for you. You will find most common reasons for programming migraine and how to relieve the pain.

Step 1: Identify the reason for the headache

The first step is to find what is the reason for the headache. It's extremely important to identify what is the cause and what is the type of the headache. This will help you to treat it better and address the problem in right direction. Below you can find most popular reasons associated with programming headache:

Eye strain
Bad colors / Dark theme(+ astigmatism)
Bad posture
Bad music / Noise
Over-thinking
Dehydration

Once the type of headache(you are suffering from) and the reason are clear you will know what practice to use to relieve it. In some cases, a head massage can help in others taking longer breaks. In the next step you can find how to ease the pain caused by headaches.

Step 2: Tips to relieve coding migraine

What are the best methods to relieve stress? I'll try to list the one which works for me.

Try Massage

You can do it yourself, or simply ask a friend / a specialist.** A few minutes massaging your neck, top of the head and temples can ease a tension headache, which may result from stress**. You can also buy simple massager like:

I'm using the first one and it really make the huge difference in the end of the long working day. After several minutes of massage I feel warming feeling which:

I can feel that all the way down in my toes!

Deal with Bad music / Noise

I was working simultaneously in open office and from home for several years (on weekly basis). I found out that my performance from the office was bad and I was less productive. It was really hard for me to concentrate and focus on the task which I had for the day. At the end of the working day I had also - subtle pain in my forehead.

I tried to change different things until I found the real cause for my suffers - the noise in the open space. The noise was on from the early morning till the late evening. Only when most workers were gone and the air-conditioner was off I was able to do something properly. What was the solution which helped me to deal with the constant noise? It's called - Noise Canceling Headphones.

Below you can find two models which will do the job:

Be careful because bad music might be another reason for programming migraine. So I prefer to:

Work on quite place in nature when it's possible
If not - use noise canceling headphones with nature sounds
Listen low acoustic music

Dehydration

Dehydration is one of the greatest migraine triggers. It might be related to drinking alcohol, too many cups of coffee, too salty food or simply don't drinking water. The focus on hard problems or staying too long in front of computer might deceive your thirsty feeling.

What could be done? You can

add a cup or bottle of water on your desk. If the cup is present on your desk it's more likely to drink it.
Frequent breaks will normalize your senses - so you will find if you are thirsty or not.
Drinking tea

Fix bad posture problem

Incorrect posture causes migraine and back pain. Most probably there is muscle imbalance which can be treated in long term with sport, stretching and exercises. In short term (or if you don't have 15 minutes per day for stretching) you can try to:

check if your monitor is ergonomically correct - the top 1/3 of the screen is at eye level.
sit up straight
avoid any prolonged stay
use ergonomic chair like:
Fortnite OMEGA-Xi Gaming Chair, RESPAWN by OFM Reclining Ergonomic Chair with Footrest (OMEGA-02)
Fortnite OMEGA-Xi Gaming Chair, RESPAWN by OFM Reclining Ergonomic Chair with Footrest (OMEGA-02)

Keep in mind that the long term goal is to stay less on the chair and be healthy. Even the most ergonomic chair will not help if you have bad habits. The key point here is to cure the muscle imbalance.

Bad colors / Dark theme(+ astigmatism)

There are studies that show some fonts and colors can result in migraine headache. Monitor screen colors, color theme, dark mode can be considered as triggers. For example dark mode theme in combination with astigmatism result in eye strain for me.

Best option here is to experiment with different setups and applications. Most probably your monitor have native option for regulating the working regime like - I-Style color:

Standard
Game
Cinema
Scenery
Text

I found out the Text regime is best working for me. In combination with programs like Red-Shift which decrease the levels of the blue light - I solved the headache related to colors and intense light.

Decrease eye strain

Another common problem is eye strain. It can be related to many factors like:

wrong view angle
staying for too long of monitor
bad or no light
bad / old monitor

You can try to change the position of your desk - closer or further from a window. To decrease direct sunlight or change the direction of your monitor. Play with the level and the inclination of the monitor.

Change of fonts and resolution is another option. If you read text in your browser you can increase the font size. A simple trick which helped me a lot in the past is to rinse my eyes with cold water. I did this regularly for one month and the headache which I had at that time due to bad monitor - disappear completely.

Buying a new monitor still can be an option. You can find some ideas below:

Over-thinking

Don't try so hard. Take difficult problems easy and step by step. Take your mind off the problem, switch to something else, just relax or simply enjoy the day outside. Do take an afternoon nap to rest your brain, eyes and nerve system.

Attack the problem from different angle or ask for help. Think deeply for simple things requires a lot of energy and mental strain. Try to balance it with the same amount of relaxation.

Step 3: Tips for prevention of coding migraine

In addition to previous steps you can find several simple things which you can do to ease the pain. Try these tips and get to feeling better:

Meditation

Meditation play huge role in the connection between our internal and external worlds. It helps to controls anxiety, promotes emotional health, fight stress and enhance focus. Start short meditation sessions on daily basis and you will find a clear picture of your mind. By improving attention and concentration you will solve problems faster with less mental power.

Frequent walks

The benefits of nature walks are difficult to be listed in a short article like this. But definitively one of them is - headache and stress prevention. The more time you spent out in the nature the more you will benefit from reduction in stress levels and burn out probability.

For sure nature walks improve concentration which will help you solve hard problems much faster and with less efforts. Fresh air is the natural food for your brain. Quick walk during the work day may do the difference between successful worker and struggler.

Pro Tip: You can remove your glasses when you walk in the nature or outside. This helps me to relax my eyes and mind (I have average near-sightedness). But please be careful!

Afternoon naps and more sleep

Do you know what is common between:

Aristotle
Leonardo Da Vinci
Albert Einstein
Thomas Edison

At least one thing which I know was their love for naps. Working as a programmer is mental draining occupation which requires long periods of concentration. One of the ways to rest from this long periods is by sleeping and switching off your brain for some time.

Sleep is extremely important for memory, concentration and relax of the mind. 8 hours of strong sleep is a good recipe for headache killer.

Conclusion

People who suffer from programming headache are susceptible to a number of factors that can trigger and boost such condition. Programing is related to staring at code all day with intense concentration which is challenging from a vision and mental perspective.

In combination with bad sleep habits, not drinking enough water, eating unhealthy food, constantly rushing deadlines might result in headache and stress.

What we could do to prevent headache is:

take regular short breaks to give your eyes and brain a rest
going for walk. Walking outside will clear your head, improves your productivity and give you fresh energy
live more active live - include sport and stretching in your daily routine
focus on small things like: drinking a cup of water, stare at distant object, afternoon nap or chat with colleague.

How to Install Oracle JDK in Ubuntu 18 in 2020

Softhints — Tue, 05 Nov 2019 14:02:32 +0000

In this short guide, I'll show you the new way of installation Oracle JDK: 8, 11 or 13 on Ubuntu 18 or Linux Mint 19 (and above). There are many articles describing the old steps which are not working anymore due to changes from Oracle. Below you can find the whole process - step by step.

Note 1: If you need OpedJDK then you can install it with command like: sudo apt-get install openjdk-8-jdk or by using SDKMAN - I prefer to use SDKMAN which gives many benefits like working with different versions and easy updates.

Note 2: Be sure that you read the Important Oracle JDK License Update before using Oracle JDK.

Step 1. Create Oracle account and download Oracle JDK

First in order to avoid confusion there are two Java downloads:

The second one is the one which has JDK and can be used for development. It's hosted on the Oracle website and needs Oracle account in order to download the JDK. Once the download is complete you can chose the Java version that you want to download - i.e. (or get recent from the link above):

For Ubuntu/Linux Mint you need to download .tar.gz file for your architectuer: 32 or 64 Bits.

Step 2. Install Oracle JDK in /usr/lib/jvm

I check huge amount of articles trying to find what is the best place to install oracle JDK on Ubuntu. There are different choices so I decided to select:

/usr/lib/jvm

which was the previous installation place for Oracle JDK 8.

Once the download is complete you can move the .tar.gz file to the folder of installation and extract it there:

cd Downloads
sudo mv jdk-8u231-linux-x64.tar.gz /usr/lib/jvm/
sudo tar xzvf jdk-8u231-linux-x64.tar.gz

This will install the JDK in folder - jdk1.8.0_231:

/usr/lib/jvm/jdk1.8.0_231

Step 3. Configure Java for update-alternatives (optional)

This step is needed in order to make newly installed Java visible for update-alternatives. What you need to do is:

sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.8.0_231/bin/java 1
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.8.0_231/bin/javac 1

Now if there are more than one Java version you can manage then easily by:

sudo update-alternatives --config java

and then select the best Java for your machine(in the example below Oracle JDK is not installed):

There are 2 choices for the alternative java (providing /usr/bin/java).

  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1101      auto mode
  1            /usr/lib/jvm/java-11-openjdk-amd64/bin/java      1101      manual mode
  2            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode

Press <enter> to keep the current choice[*], or type selection number:

Step 4. Set up variable JAVA_HOME (optional)

Many programs depends on environmental variable JAVA_HOME. So it's a good idea to set it up in order to avoid problems later. The setup of this variable for Debian based systems can be done by opening user profile settings:

sudo nano ~/.bashrc

Then you need to add somewhere near the bottom of the file:

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_231
export PATH=$JAVA_HOME/bin:$PATH

Then you need to reset the terminal (by command source ~/.bashrc) or open new one and test the variable by echo:

source ~/.bashrc
echo $JAVA_HOME

With this last step the installation of Oracle JDK 8 on Ubuntu/Linux Mint/Debian is complete.

Bonus: The scripts for installation of Oracle JDK 11 or 13

Download Oracle JDK 11 (the link is above) from the site and then execute:

sudo mv jdk-11.0.5_linux-x64_bin.tar.gz /usr/lib/jvm/
tar xzvf jdk-11.0.5_linux-x64_bin.tar.gz
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk-11.0.5/bin/java 11
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk-11.0.5/bin/javac 11

Download Oracle JDK 13 (the link is above) from the site and then execute:

sudo mv jdk-13.0.1_linux-x64_bin.tar.gz /usr/lib/jvm/
tar xzvf jdk-13.0.1_linux-x64_bin.tar.gz
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk-13.0.1/bin/java 13
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk-13.0.1/bin/javac 13

Note: if you wonder what are the numbers at the end of this command then this is the explanation:

a priority number. The highest the number, the higher the priority. That means that the alternative with the highest number will be executed by default, unless we manually set which is the default.

Resources

Complete list of 10 best Python IDEs and Code editors of 2020

Softhints — Sun, 03 Nov 2019 08:26:24 +0000

What python IDE to chose in 2020? The main purpose of this article is to show comprehensive list best and most popular Python IDEs for programming and education that make learning and coding much easier!

The information is based on personal experience, public web data and several surveys like:

The list of best Python IDEs and Code editors for 2020:

PyCharm ( Professional and Community edition) - Python integrated development environment specifically designed for the Python language.
VS Code - Visual Studio Code is a code editor redefined and optimized for building and debugging modern web and cloud applications. If you are working on multi-language web project which needs python then you can give it a try.
Vim - Vim is a highly configurable text editor with good support for Python
Sublime - Sublime Text is a sophisticated text editor for code; Many packages can make it fully functional python IDE
Jupyter Notebook - is a web-based interactive development environment; It's well known in data science community for analyzing, sharing and presenting information
Atom - multi-platform simplistic source code editor. It makes collaborating on code just as easy as it is to code alone. Many packages and ecosystem can convert the code editor into nice python IDE.
IDLE - comes with python. IDLE has two main window types, the Shell window and the Editor window. Perfect python tool for people starting with programming and python.
GNU Emacs - An extensible, customizable, free text editor - and more. It was popular choice for python in the past which lost his glory recent years.
Eclipse + PyDev - PyDev is a Python IDE for Eclipse. PyDev had huge problems in the past with stability and performance. This pytonic tool is evolving so waiting for good news in future.
IntelliJ and Python Plugin - Python plugin extends IntelliJ IDEA with the full-scale functionality for Python. Projects which needs Java and Python can be developed with ease in IntelliJ
Spyder - Spyder is a powerful scientific environment written in Python, for Python, and designed by and for scientists, engineers and data analysts.
Online Python IDE - repl - Write simple Python code and run it in your browser; includes some basic snippets and examples

Tabular comparison of the IDE/Source editor in 2020

IDE	Personal Score	Level	Price	Debugging	Light-weight	IDE	Multi-language	Projects
PyCharm Professional	★★★★★	Advanced+	Paid	✔	✘	✔	✔ (limited)	Various/Big
PyCharm Community	★★★★★	Beginner, Advanced	Free	✔	✘	✔	✔ (limited)	Various/Big
VS Code	★★★★	Advanced	Free	plugin	✘	✔	✔	Web/Big
Vim	★★★★	Advanced+	Free	pdb	✔	✔	✔	Various/Big
Sublime	★★★★★	Advanced	Free (*)	package	✔	✘	✔	Various/Mid
Jupyter Notebook	★★★★	Beginner, Advanced	Free	✔	✔	✘	✔ (limited)	Data-science
Atom	★★★	Beginner, Advanced	Free	package	✔	✘	✔	Various/Mid
IDLE	★★	Beginner	Free	✔	✔	✘	✘	Small scripts
Emacs	★★★	Advanced+	Free	package	✔	✔	✔	Various
Eclipse + PyDev	★★★	Advanced/Mid	Free	✔	✘	✔	✔	Various
IntelliJ and Python Plugin	★★★★	Advanced	Free (*)	✔	✘	✔	✔	Various
Spyder	★★★★	Advanced+	Free	✔	✘	✔	✔ (limited)	Data-science
Online Python IDE	★★	Beginner	Free	✘	✔	✘	✘	Small scripts

What question helps to find optimal tool for Python?

The popularity of code editors is decreasing with the years but they are far from dead. There are many people who still use "old school" tools like: vim, emacs, plain text editors etc - which write beautiful code and interesting projects. The questions below try to help novice in programming ( or people who want to try something new ) how to choose optimal choice for them.

Important questions which will help you to find the optimal tool for you:

IDE vs Text/Code Editor - because of plugins and additional tools is hard to draw the line.
- Integrated development environment (IDE) is feature rich tool which provides facilities like debugging, autocomplete, support for environments, project setup, project structure, console etc
- Text/Code Editor - a tool for editing and modifying the code mainly.

Note: With good plugin/package ecosystem the simple source editor might become lightweight but powerful IDE (it's matter of configuration and experience)

What is your level?
- beginner - IDLE (even online python editors) is perfect choice for the first steps in python language. In my opinion PyCharm is good choice for beginners - with some help from more experienced friend.
- mid - PyCharm, Sublime, Atom, Vs Code
- advanced - PyCharm, Vim, Emacs, Sublime, Atom, Vs Code
What do you (plan to)do with python?
- web development - PyCharm Professional, VS Code
- data science - Spyder, Jupyter Notebook, PyCharm Professional
- scripting - Sublime, Atom, PyCharm Community, Eclipse + PyDev
- QA - Sublime, Atom, PyCharm Community, Jupyter Notebook
What is your environment/OS?
- multiple/mixed OS - PyCharm, Sublime, Atom
- Linux, macOS - PyCharm, Sublime, Atom, Vim, Jupyter
- Windows - Sublime, VS Code, Eclipse + PyDev, PyCharm
- Web only - Online Python IDE-s
What hardware do you have?
- bad - IDLE, Atom, Sublime, Online Editor
- good - PyCharm, VS Code, Eclipse + PyDev
Programming languages
- only python - IDLE, Atom, Sublime, PyCharm, Vim
- python + more (Java etc) - IntelliJ, VS code

Note: Most of the answers are based on personal experience and my work with programmers with different background and knowledge. Also there are many other points to be considered as budget, git integration, team work, previous programming knowledge.

List of best Python IDE in 2020

Now it's time for (from a python developer with 4+ years in python, and developer with 14+ years in coding) top python IDEs / Code editors with description which I'll use or try in 2020. There is sections with pros/cons, how I'm using the IDE, what projects are used etc.

Most used Python IDEs

1. PyCharm

version - 2019.2.3 (expected version PyCharm 2020.1 in 2020)

Why

It has huge arsenal of features like:

Highly customizable - some examples:
Intelligent Assistance for Code
Powerful debugger - IntelliJ Debug, conditional breakpoints and step back
The fastest switching between files I've ever seen
It has nice performance and beauty
Good support for git and version control
PyCharm has a very good support for virtualenvs.
It's maintained and has two versions: paid ( Professional) and free (Community - open source)
- I used the free one - because it's enough for my needs
- There is also one more - Education - for students
It has multiple windows/views:
- version control
- code structure - working great for code but also for JSON, YAML etc
- terminal
- run / Debugger window
- project structure
- to do lists
Script / Configuration editor
Code / Python specific refactoring
Project and code navigation

What

I'm working on many different projects 25+. All of them different in nature, environments, code style and git setup (bitbucket and github), non python files like json, yaml and markdown.

Python projects per size:

small - several python files plus configuration is the whole project
medium - several folders with python files and configuration
big - hundreds of files
huge - thousands of python files, configurations, integration of different API-s

Projects by type:

data science - some of the projects are related to data science (beginner to medium level) - time estimation, error detection
data quality - frameworks responsible for data quality and analysis of data anomalies, finding outliers
automation - automation of various tasks like data collection, notifications, summary and report generation
office automation - create office documents, parse hundreds of xls/csv files, create doc files, parse PDF files

How

I'm using PyCharm on daily basis. Using it for several hours in a row without distraction (it has Distraction Free mode). In general I work with 1 open project at a time. From time to time I can open second or third project or additional file in separate window.

Nice/Python plugins - 1700+ plug-ins available

Chronomorph - automatic change of dark and light theme based on time
CSV plugin - nice lightweight table editor for CSV files
IdeaVim - Vim emulation for PyCharm
R Language for IntelliJ - perform various statistical computing
Markdown
YAML/Ansible support - support for YAML

Most of the plugins come preinstalled with PyCharm so you don't need to do anything more. This is nice advantage for PyCharm. If you want to check plugins installed in PyCharm then you can do:

Settings - CTRL + ALT + S
Plugins
Installed

Pros

Free version
It has standard and similar way of working between Idea products - IntelliJ and PyCharm. Considered myself as a Java and Python programmer this is a huge advantage - being able to work in same manner with Java or Python.
Big community and resources productivity guides - 42 IntelliJ IDEA Tips and Tricks
It helps to focus on important part (coding) and not on organizing, searching and style setup
Cross-platform and easy for installation/integration

Cons

It requires high amount of RAM for medium and big projects.
Some of the features are included only in the paid version - for example support for Notebooks or web projects
It requires some time to adopt all features and productivity guides

2. Jupyter Notebook

version - 6.0.1 (Releases can be tracked here: jupyter/notebook releases)

Why

Jupyter Notebook become a standard in data science world because it's powerful, easy for working for developers and easy to present at the same time. Many organization and companies use it in their everyday work.

Interesting Features:

Jupyter notebooks magics
Markdown and HTML integration
Support different kernels - helps working with Languages like Julia, R etc
Versioning with git
Visual representation
Easy automation and integration for Linux
- generating notebooks
- summarizing information
- starting and stoping servers, opening folders and files
Easy installation and setup

How

I have several environments with jupyter servers organized by projects and topics. Using it on daily basis for 1 or 2 hours per day. Automated scripts process web data and extract only important results for me - saving me huge amount of time or replacing one person who will need to do it.

What

I use it primary for data quality, automation QA and trend/error detection. Working with CSV, text and JSON files. Analyzing datasets and store temporary results. Share findings and code snippets.

Here you can find small collection of my notebooks: Jupyter notebooks and datasets for the interesting pandas/python/data science video series.

Nice/Python nbextensions
What is nbextensions:

The nbextensions contains a collection of extensions that add functionality to your Jupyter notebook.

All below can be installed and enabled from: Miscellaneous IPython and Jupyter extensions

Table of Contents - add table of content (TOC) to your jupyter notebooks - extremely handy for huge notebooks with hundreds cells
Variable inspector - show information about variables created in the notebook like: name, value, type, size
Hide Code input - focus on the results while hide the code. This is extremely useful when you need to share the notebook with non it representative which are afraid of seeing source code
Autopep8 - helps to write code formatted by PEP8 standard. Most of the time working with notebooks we focus on data and results but forget about the code style and format which makes the code hard to read later.

Pros

Cross-platform
Free
Markdown support
Multi language/environment support by Kernels
Popular standard in Python world

Cons

difficult debugging
required automation for big projects
There is a need of organization of folders and files - otherwise very soon it will become nightmare for finding and navigating in such projects
It needs best practices and time for management
- For example file naming - 20191027-XXXX-report
- add code/documentation for code cells - I have notebooks with hundreds of cells - it's not very nice to work with them after one or two years
- Avoid duplication and repeated work by using templates
sometimes information can be lost if there isn't good backup solution ( reasons for problems - version incompatibility, broken extension, corrupted file system )

3. Sublime

version - 3.2.2 (Sublime Dev Channel Changelog)

Why

Sublime is great text editor which can do much more than that. I'm using it to work it temporary information, interesting articles and texts. For example collecting meeting notes, decisions, to do lists etc.

I'm using it as a mental separation and not like a technical one. What I mean here is: if I'm on a meeting then I'll open Sublime in Markdown mode and I'll gather meeting notes there. If interesting idea or solution came on my mind - then I'll write down them with Sublime. Later on the same day I'll go through all files in Sublime and do something with them - for example add them to my Lab Diary

How

All day long I have open Sublime text editor. I'm using it in minimalistic mode - trying to avoid distraction and do what I need to do in the smallest possible number of clicks.

What

working with JSON files - reindent - view, edit and reindent JSON files
Markdown support - collecting meeting notes and TODO lists
File comparison and regex - searching with regex, edit and replace in small to medium files
- Regex find group of words and replace
- Regex examples of capture group and replace
Store temporary notes, links and code snippets
Small python code snippets and bash scripts

Nice/Python packages

Anaconda - Anaconda turns your Sublime Text 3 in a full featured Python development IDE including autocompletion, code linting, IDE features, autopep8 formating, McCabe complexity checker Vagrant and Docker support for Sublime Text 3 using Jedi, PyFlakes, pep8, MyPy, PyLint
Sublime Python IDE - add python completions and refactoring
Djaneiro - Django snippets and templates, keyword highlighting
Markdown Preview - Preview and build your markdown files quickly in your web browser

Pros

Cross-platform
Pretty good package system
Highly customizable
Light with good performance
Syntax Highlighting (various color themes)
Block selection and simultaneous editing (multiple selections)

Cons

(Personal experience)

It's not suitable for huge python projects
It's had to work in multiproject way
Needs extra time and knowledge to be converted in Python IDE

4. Atom

version - 1.42.0 (Atom Release notes)
and Atom releases in github

Why

Atom is helping me to collaborate with my colleagues by using plugin - Teletype - Collaborate in real time in Atom. From the official page:

Teletype(beta) for Atom lets developers share their workspace with team members and collaborate on code in real time.

It's quite nice feature for beginners and people who want to learn Python faster.

How

I'm using it 1 or 2 days per week when I need to support colleague of mine or I need help with some tasks. Atom support many different languages so I'm using it not only for Python but also for JSON and YAML files.

What

Small projects for beginners, JSON and YAML schemas, configurations. I like color themes and code styles of Atom for JSON / Markdown files.

Nice/Python packages

atom-python-run - Execute python (.py) programs using F5 or F6
Python Black - Uses Black for formatting Python code. It requires - black to be installed - pip install black
python-indent Simple and powerful Python PEP8 auto-indentation
atom-python-test - run py.tests and unitest.TestCase tests
Python Tools - handy tools to make developing Python code in Atom even more enjoyable
Python Autocomplete Package - Python packages, variables, methods and functions with their arguments autocompletion

Pros

Cross-platform
Package management
Native git versioning
Multi-language support with good syntax highlighting
Lightweight
Open source and free
Multiple panes and themes

Cons

it has unique way of work which needs some time of learning
lack of good single tap / project structure way of working. (For example in PyCharm I can work only with 1 open tab at time easily navigating in Project pane and recent files. Which I find hard in Atom)
needs time for configuration for Python

Interested in Python IDEs

5. Visual Studio Code

version - Visual Studio 2019 (expected Visual Studio 2020)

Visual Studio 2019 Release Notes

Why

Some people prefer VS code (Visual Studio Code) because it is very well designed for the development and debugging of latest web and cloud projects. For example some people working with Angular and ReactJS use it in their everyday work.

Visual Studio Code is another mature framework which have great community. It has nice editor and offers development features which makes work on web projects almost without any efforts. Maybe the only one IDE from the list which can rival in web development with VS Code is PyCharm Professional Edition.

Features which attract my interest:

plugins and customization
nice self-explaining interface and layout
multi-split window feature
vertical orientation

How

Testing Visual Studio Code with new small web projects and see how behaves in comparison to PyCharm. Test project life cycle - starting empty project, installing dependencies and use git integration.

What

Web related projects, angular plus reactjs. It might be good for people who want to learn python and has experience in other languages and Visual Studio code. Appropriate for web development.

Nice/Python plugins
* GitLens - Git blame annotations and code lens
* Python - rich support for the Python language
* Code Linting - linting your python code ( Note: Linting is the process of running a program that analyzes your code for programmatic and stylistic errors )
* Djaneiro - python and django code snippets
* autoDocstring - quickly generate docstrings for python functions

Pros

multi-platform
free
open Source.
multi-language support
good integration with git
live precompilation on change - if you do a change on your started web project in most cases will be propagated to the started app ( might depends on the OS settings - like number of file to monitor )

Cons

it needs configuration and setup for python
it's resource demanding
searching might be slow ( this might depend on the OS, hardware, project size and configuration)
worse navigation and indexing in comparison to PyCharm ( it need additional configuration for python )
slower start (also some operations might take more time than expected)
Some problems with Linux integration ( I faced problems during installation and configuration for some projects in the past )
Some developers complain that Change log for git is not good enough and use 3rd party software like GitKraken in combination with Visual Code studio.

6. Vim

Version Vim 8.1 is the latest stable version.

Why

Vim is extremely popular in geeks communities and people who has unique style of writing code. There is big lists or plugins, features and versions of Vim.

And one advice from Python's creator - Guido van Rossum

Emacs is dead, use vim.

How

Starting with small projects and tutorial from the web maybe once per month.

What

Small python and cross-language projects which requires focus and distraction free environment.

Pros

Open Source
Cross-platform
Simplistic
Distraction free
Very stable
Lightweight

Cons

it's not good for people who don't like to use terminal
difficult for beginners because it has stipple learning curve
It can be challenging even for experience programmers - because it has unique way of working which requires focus and concentration

7. Eclipse PyDev

Version - Release 7.4.0 (2019-10-25)

Why

Eclipse was my favorite IDE in the past for many years. I'll give chance to it again and again.

How

Testing existing projects from PyCharm in Eclipse PyDev.

What

Small and medium python projects to see what is new in Eclipse PyDev. What I want to ensure is that Eclipse PyDev is stable and with good performance for the most of python project life cycle.

Python features - this is the full table of features for PyDev

Django integration
Code completion with auto import
Remote debugger
Interactive console
Unittest integration
Code coverage
PyLint integration

Pros

open Source and free
multi-platform source editor
designed for python
multi-language support
interactive console
unittest integration
good syntax high lighting

Cons

very often the plugins are unstable or crashing the IDE. I had huge problems and lost many lines of code because of git problems with Eclipse (which force me to move to IntelliJ few years ago)
bad performance, unresponsive at times, crashes
plugins are not working smoothly

8. Emacs

latest version is: 26.3

Why

In the past Emacs was the favorite editor for Guido van Rossum(who later - 2017 - said that Emacs is dead - check section for Vim) and many others.

Tweet from Guido in 2016:

Emacs of course!

In Book: The Art of UNIX Programming is said:

Emacs is undoubtedly the most powerful programmer's editor in existence. It's a big, feature-laden program with a great deal of flexibility and customizability.

How

I'll give a try to it several times duing 2020.

What

Small projects and code snippets.

Pros

helps to focus
during the learning process and configuration for emacs you can learn a lot of your mistakes :)

Cons

installation depends on the platform
additional package for git integration - magit
more complex way of debugging
need additional steps for code formatting, syntax checking etc
specific customization and configuration for python
Emacs has a learning curve which require more time in comparison to other source editors.

Python IDEs/Code editors for beginners

9. IDLE

Why

It's the simplest possible way to run and work with python code on your desktop machine which is included in Python by default.

How

When I need to teach beginner or person without any experience in Python.

What

Simple projects like web browser game automations, basic web scraping applications and office automation.

Pros

No need of installation, setup etc. Just code!
Perfect for verifying small snippets and how python works

Cons

It has some limitation - you can't master python by using only IDLE. Soner or later you will need to try something better

10. Online Python editors

Why

Some companies and organizations prohibit installation of additional software (sometimes you might be able to run portable python from a USB) or even detecting what processes are running. In this cases using web Python code editor is the only available option.

How

Occasional use

What

Basic snippets and small programs

Pros

No need of installation, setup etc. Just code!
Perfect for verifying small snippets and how python works
Code sharing

Cons

many limitation - working with files, resources etc
depends on Internet

Conclusion - What is the best Python IDE in 2020?

For me the question has strange answer: There's no best IDE for Python in 2020 but there is a constellation/combination of IDEs and text editors which can make learning, coding and working with python code faster, easier and more reliable.

The closer IDE to the first place is PyCharm - because offers the richest set of features and options. Despite being relatively young IDE - Initial release - February 2010 (about 9 years ago - in comparison to Emacs - with versions since March 20, 1985) - it's well developed and mature environment which can manage big set of projects and assist developer in best possible way. For me :

PyCharm is the most pythonic IDE in 2020!

One advice - Don't search for best/most etc but search for optimal - the tool which can help you do your work with less efforts.

Happy Coding in 2020,
Softhints.

Python 3.8 New Features - code name walrus

Softhints — Fri, 18 Oct 2019 19:17:51 +0000

A brief and practical guide to the most important/interesting new features/changes coming with Python 3.8 (code name - walrus - joking :) ). So first lets announce the news:

Python 3.8 was released on October 14th, 2019.

The official documentation is here:

If you want to play with the new features without installing the latest python 3.8 then you can do it here: Python 3.8 (pre-release)

Brief video showing the new features

Overview

In this article we are going to have brief, high level review at some of the new features. Python 3.8 comes with a new warlus Operator, positional-only parameters and lots of other new and interesting changes, improvements and features.

It offers also another new features, plenty of language changes, one new module and improvements on others.

Many optimizations and C API changes can be found in the version. Several methods and commands are deprecated.

Personally I'll not move to python 3.8 soon. Nothing is so important or a game changer for the moment. Some features could be tricky or cause confusion which is another reason for me to stick to 3.7 for the moment.

New Features

For me the most interesting new features are 3:

The Walrus Operator
Positional-only parameters
f-strings support =

The Walrus Operator :=

In short this is a new operator which will be used as assignment or named expression. Official description from the PEP:

This is a proposal for creating a way to assign to variables within an expression using the notation NAME := expr.

If you wonder why it's called like this:

It is affectionately known as "the walrus operator" due to its resemblance to the eyes and tusks of a walrus.

The best way to understand the new operator is by viewing some examples like:

Avoid repeated expression calls:

num = 3
if (n := num % 2) != 0:
    print(f"Number is not even: number: {num}, remainder: {n}")

result:

Number is not even: number: 3, remainder: 1

Regex expressions

import re, datetime
text = 'The latest version of python 3.8 is released on 14-10-2019 .. '

# before walrus
whole_match = re.findall(r'[\d]{1,2}-[\d]{1,2}-[\d]{4}', text)
if whole_match:
    print(whole_match)

# after walrus
if (found_date := re.findall(r'[\d]{1,2}-[\d]{1,2}-[\d]{4}', text)):
    print(found_date)

List comprehension

If a value computed in a filtering condition is also needed in the expression body:

numbers = [3, '0', 0, '6', 2, 1, 0, '7']
print([final_number for number in numbers if (final_number := int(number)) != 0])

result:

[3, 6, 2, 1, 7]

Errors and usage:

Sometimes walrus operator can be misleading and lead to confusion or errors. So it might be a good idea the usage to be limited to simple cases and expressions in order to avoid problems with code readability.

Below you can find list comprehension and walrus operator which leads to error:

numbers = [3, 0, 0, '6', 2, 1, 0, '7']
[number for number in numbers if (number := int(number)) != 0]

SyntaxError: assignment expression cannot rebind comprehension iteration variable 'number'

Another interesting case and possible abuse is:

if (I := he ) or ( you := me ) and ( we := alltogether):
    I = walrus

You can read more here: I walrus

Full description is available here: PEP 572

Positional-only parameters

New syntax for function parameters is introduced - /. This is an example function:

def f(a, /, b, *, c):
    print(a, b, c)

in this function parameter a is positional only and cannot be a keyword argument. While on the other hand c must be a keyword argument.

Let's check several examples:

correct call for this function:

f(10, 20, c=30)

result:

10 20 30

Errors

TypeError: f() takes 2 positional arguments but 3 were given

f(10, 20, 30) # c must be a keyword argument

result in

TypeError: f() takes 2 positional arguments but 3 were given

TypeError: f() got some positional-only arguments passed as keyword arguments: 'a'

f(a=10, b=20, c=30)   # a cannot be a keyword argument

result in

TypeError: f() got some positional-only arguments passed as keyword arguments: 'a'

Positional only parameters can be used in two ways. Their names are not shown as possible keywords which makes them available for kwargs:

def f(x, /, **kwargs):
    print(x, kwargs)

f(10, x=1, y=2)         # x is used in two ways
10 {'x': 1, 'y': 2}

For more info: Python Positional-Only Parameters

f-strings support = for self-documenting expressions and debugging

In 3.8 is added support for = for f-strings. So f-string like:

f'{version=} is released on {release_date=}'

will be represented as:

version='3.8' is released on release_date=datetime.date(2019, 10, 14)

Few more examples on this

from datetime import date

version = '3.8'
release_date = date(2019, 10, 14)
python_one_version = date(1994, 1, 1)

since_python_one = release_date - python_one_version

print(f'{version=} is released on {release_date=}')
print(f'{version=} day of Release {python_one_version.day=:02d}')
print(f'{version=} day of Release {since_python_one.days=:,d}')

results:

version='3.8' is released on release_date=datetime.date(2019, 10, 14)
version='3.8' day of Release python_one_version.day=01
version='3.8' day of Release since_python_one.days=9,

Used in previous python versions like python 3.7 will result in:

  File "<fstring>", line 1
    (version=)
            ^
SyntaxError: invalid syntax

The main idea is:

#(really) bad old days this was pretty wordy:
print "foo=", foo, "bar=", bar

#f-strings make this slightly nicer to type:
print(f"{foo=} {bar=}")

result:

foo=datetime.date(2019, 10, 14) bar=1

Other new features

Some of the features cannot be easily explained with examples so I'll just list them:

Parallel filesystem cache for compiled bytecode files
Debug build uses the same ABI as release build
PEP 578: Python Runtime Audit Hooks
PEP 587: Python Initialization Configuration
Vectorcall: a fast calling protocol for CPython
Pickle protocol 5 with out-of-band data buffers

Other language changes

In this section there are some quite interesting changes which might need your attention when you migrated from older python version to 3.8 like:

continue statement is now allowed in finally clause
new method as_integer_ratio() for bool, int and Fraction types:

x = 7
print(x.as_integer_ratio())

result:

(7, 1)

Added support of \N{name} escapes in regular expressions:

import re

notice = 'Copyright © 2019'
copyright_year_pattern = re.compile(r'\N{copyright sign}')
print(copyright_year_pattern.search(notice).group(0))

which will error in earlier versions:

sre_constants.error: bad escape \N at position 0

Dict and dictviews are now iterable in reversed insertion order using reversed()
Warnings when comma is missed in code like [(1, 2) (3, 4)]

before:

  File ".code.tio", line 1, in <module>
    [(1, 2) (3, 4)]
TypeError: 'tuple' object is not callable

now:

SyntaxWarning: 'tuple' object is not callable; perhaps you missed a comma?
  [(1, 2) (3, 4)]
Traceback (most recent call last):
  File ".code.tio", line 1, in <module>
    [(1, 2) (3, 4)]
TypeError: 'tuple' object is not callable

Arithmetic operations between subclasses of datetime.date or datetime.datetime and datetime.timedelta objects now return an instance of the subclass, rather than the base class.

For example:

from datetime import datetime, timezone

class DateTimeSubclass(datetime):
    pass

dt = DateTimeSubclass(2012, 1, 1)
dt2 = dt.astimezone(timezone.utc)

print(type(dt))
print(type(dt2))

in python 3.8

<class '__main__.DateTimeSubclass'>
<class '__main__.DateTimeSubclass'>

before:

<class '__main__.DateTimeSubclass'>
<class 'datetime.datetime'>

When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the resulting KeyboardInterrupt exception is not caught, the Python process now exits via a SIGINT signal
Dict comprehensions have been synced-up with dict literals so that the key is computed first and the value second:

from unicodedata import normalize
names = ['Löwis', 'Łukasz', 'Dörwald']
print({(n := normalize('NFC', name)).casefold() : n for name in names})

result:

{'löwis': 'Löwis', 'łukasz': 'Łukasz', 'dörwald': 'Dörwald'}

Conclusion

Python 3.8 comes with new assignment operator, syntax for function parameters and many other improvements and features. At the first sight some of the changes look a bit controversial but they can offer new solution to old problems.

The examples covered in the article show potential for expanding the usage beyond the scope described in PEPs.

Once this version is well adopted I'll move to it in order to explore the full potential of it.

Previous python release is reviewed in this article: Python 3.7 features and release date

6 Best Practices for Gmail to Boost Your Workflow

Softhints — Fri, 20 Sep 2019 06:53:53 +0000

According to recent research done in August 2019 Gmail is mail client number one with 29 % market share (more info: Email Client Market Share). On the other hand the modern worker spent about 28 % of their time in email management. (More info here: The social economy: Unlocking value and productivity through social technologies ). If you spend one third of your time using Gmail, most probably you will need to analyze your workflow.

This article try to share tips and tricks based on my personal experience. The results of the article can be measured and verified after one month. Some of them are applicable not only for Gmail but also for other clients like Microsoft Outlook, Yahoo, Thunderbird, etc

You can find the official best practices from Google here:

The best practices:

Use labels and categorize Gmail
Create ToDo, Important and Non important folders
Open and use your Gmail only at a given time
Keep your inbox clean and neat
Use mail templates
Customize Your Gmail Interface Look With Themes
Gmail scheduling and automation

1. Use labels and categorize Gmail

There's no limit on the number of the labels you can create. You can organize the labels similarly to folders on your computer.

Note: Have in mind that labels don't behave always as folders. For example if you delete a mail then it's going to be removed from all labels!

In order to create labels in Gmail you can follow this steps:

start Gmail. (note you can't create labels from the Gmail app)
click More (on the left)
Click Create new label.
Select name for your label.
Create

Once the label is created it can be deleted or edited by:

start Gmail
Select label (on the left)
Click the Down arrow (next to it).
Click Edit if you want to edit the label
- or delete if you like to remove the label

Create well organized structure having this simple rules in mind:

Use optimal number of labels - too much will cause extra level of complexity while too few will be insufficient
Avoid duplication of labels - labels should be distinctive and unique.
Put the important labels at the top and less important at the bottom
create filters which will categorize your mail automatically

Now in order to expand the last section of the rules. Filters helps to organize incoming emails by

labeling
archiving
deleting
adding star
automated forward

New filter can be created by:

Gmail
In the search box at the top, click the Down arrow Down Arrow.
Enter your search criteria. You can test the criteria by clicking search and seeing what emails show up.
click Create filter (at the bottom of the search window).
Choose action(s) for your filter.
Click Create filter.

2. Create ToDo, Important and Non important folders

Add labels and filters which can help you to preorganize the incoming mails automatically. This is going to work fine in most cases but not in all. I have simple rules for dealing with new mails(Once the initial labeling is completed - which will help me to understand better from who is the mail):

read the email subject
check quickly the content
Now there are 3 actions which I can do:
- move the mail in folder ToDo
- read the mail (if there's no need of action or I can perform the action)
- leave the mail untouched

This rule can be tested best after absense or holidays when there are hundreds of mails in your mailbox. If your folder is well organized (keeping the top-bottom rule - most important mails on the top - then you can review hundreds of mails in matters of one or two hours).

3. Open and use your Gmail only at a given time

One common mistake is to keep your email client (and Gmail in particular) opened all the time and getting notified for every incoming mail. This will lead to distraction and switching between the tasks. In order to avoid this you can schedule a script or open your mail client two, three times per day. My personal preference is to start the gmail at 10 and 17.

Usually I start my working day at 8 in the morning. I want to complete the tasks from the previous day or to continue with my ToDo list. If I start the mail client first I'll lose track on my ToDo list or might have change in priorities.

Next time to open the mail is near the end of the working day(1-3 hours before it). This will give me enough time to do something if needed and also organize the next day better.

The good bosses don't expect you to be answering all the time or to be on stand by mode - waiting for their instructions. The good bosses check results and not how long you are available. On the other hand if someone expect your response 100% of the time (while this is not job requirement) you may think for a change.

4. Keep your inbox clean and neat

Why to tidy up your inbox? There are many reasons for this:

easier to find a mail
less time and efforts to deal with new mails
save you few extra clicks and mistakes from manual management
keep sensitive mails in a label and not in your inbox (sometimes you may need to share your inbox and you don't want someone else to see your sensitive information)
spent more time on important mails and less on spam(once something is identified as high/low informative and recurring mail - be sure to create filter for it!)

Most of the time my inbox is empty. This is because all other mails are automatically labeled and moved to different folders. In this way the inbox contains only "strange" or not categorized mails. If I find any kind of mail important - then I'll add new label and filter in order to deal with this kind of messages automatically.

Bad practice here is to have thousands of unread mails in your inbox. This will take extra time for searching and reading mails. Most of the time even you won't be able to find the message you need.

On the other hand if you work on several projects/people and they have labels - then you can search for the mail only in a given label.

5. Use mail templates

Many things will be repeated in your working days like- birthdays, anniversaries, new comers, reports, technical requests etc. All this kind of mails can be created from templates.

Of course you can do it by searching the mail and copy pasting it each time. But this is simply no efficient way of working. If you want to be more efficient then you can create templates. In order to create Gmail mail templates follow this steps:

First you need to enable the template feature

Open Gmail
Settings
Select Settings from the menu
Select tab Advanced
Enable Canned Responses (Templates)
Save the Changes

Now you can create and use new message templates

Create new email template or edit existing

Gmail
click Compose
Enter your template text
Click More
Canned responses
Choose an option - create or edit:
- Save draft as template > Save as new template - create new template
- click Save draft as template and under Overwrite Template, choose a response and click Save to overwrite it

Gmail use templates

Gmail
click Compose.
Click More
Canned responses
Insert Template
Choose a saved response to insert in your email.
Compose the rest of your message
Click Send.

Note: To delete a canned response, click Delete template, choose the response you want to delete, and click Delete.

Automate Gmail templates

Gmail
In the search box at the top
Click the Down arrow
Click Create filter - At the bottom of the search window
Choose what you’d like the filter to do
- Enter your search criteria (for example mail summary - birthday)
- or From - the sender of the mail
Click Create filter.

6. Customize Your Gmail Interface Look With Themes

Spending most of the time with Gmail you might want to change the default layout and look. According to some researches if you watch inspiring images you will be more creative and productive. Also good images can bring better mood which might be essential if your day is passing in front of the monitor.

a color can change our mood from sad to happy, from confusion to intelligence, from fear to confidence.

If you want to read more about it then you can check here:

The Effects of Color on the Moods of College Students

Apply a Gmail Theme

Gmail
Settings
Themes
Select a theme from the Themes screen
Save

When selecting images think for:

contrast - you want to read mails easily
distraction - some images might steal your focus
changes - frequent changes can bring a note of freshness in your day

Finally if you have time and the mood for it go and create your own theme.

7. Gmail scheduling and automation

Scheduling

Whether the mail will be received, open, read and digested depends on the recipient. But you still can influence on this by targeting optimal time for the person.

Check the person's time zone and working hours if possible. Then send the mail in the early working hours for him. If you are not sure about the time constraints of the person than you can check for mails from him (if any).

In order to do email scheduling with Gmail you can do:

create the mail
then click the down arrow next to Send
Schedule Send
Pick date & time
Finally click Schedule Send

Automation

Much is said and done about automation. In terms of emailing it's a huge opportunity and a problem. My advice for you is to schedule anything which is possible.

For example if you need to forward mails from a given person to another one or reports (containing given keyword in the summary) then go and create gmail filter.

If you need to send weekly report to someone - then again create a script which can generate automatical report for you. The web is full with example scripts. For example for python you can use something like:
send_an_email.py or others like:

Minimalistic module to send emails using GMail

Conclusion

Every journey starts with a single step. You can try several from the tips above and check if they are working for you. Best will be to give time for the changes.

Let say that you want to test practice number 2. Then prepare for the change ( small analysis of your mail might help in order to find optimal labels for you), apply the changes and verify the results in 1 month.

It's important at the beginning and the end to measure the time spent in Gmail (reading, writing, editing and searching mails) - the final goal is to reduce time spent by 50% and more.

Job automation in Linux Mint for beginners 2019

Softhints — Thu, 05 Sep 2019 14:21:49 +0000

Automation helps with boring and tedious work, save time and energy(Of course if you doing it right). Automation and task scheduling in Linux is done with daemon called crontab (CRON for short). The definition below best describes the tool:

cron is a Unix utility that allows tasks to be automatically run in the background at regular intervals by the cron daemon.

In this post you can find several examples of cron and bash scripts in order to automate some tasks like:

Crontab for beginners
Start random song from your Linux machine
Start motivation/relaxation video at given hour
Open random text file at Linux Mint start
Scrape webdata and load the LibreOffice ( or csv, excel etc)
Generate simple report as doc file, create jupyter notebook and send mail

All examples are written and tested on Linux Mint 19 Cinnamon. I plan additional articles and videos for some of the topics above.

Crontab for beginners

In order to schedule jobs for your user you need to write:

crontab -e

description of this command is:

Edit crontab file, or create one if it doesn’t already exist

Now you can list the scheduled jobs in Linux Mint / Ubuntu by:

crontab -l

This command will:

crontab list of cronjobs , display crontab file contents.

Example cron job:

0 16 * * * DISPLAY=:0 thunderbird %u

this will open Thunderbird mail client at 16:00. Where:

minute	hour	day	month	day (week)	command
0	16	*	*	*	DISPLAY=:0 thunderbird %u
55	9-15	*	*	*	DISPLAY=:0 ~/motivation.sh

Start random song from your machine

In order to be productive and keep your health you need to do regular breaks. Working on a computer make this habit difficult and you can easily lose impression for time. In order to have regular breaks and get some motivation a simple script can be used to play music at every hour.

Below you can find the simple script which selects random song from a folder, then start a player with the song and finally close the player.

#!/bin/bash
MUSIC=/mnt/x/Music/Motivation/
cd $MUSIC
ls |sort -R |tail -1 |while read file; do
    echo "$MUSIC$file"
    rhythmbox "$MUSIC$file" & sleep 5m
done
killall rhythmbox

Short explanation of this script:

#!/bin/bash - this is a convention so the Unix shell knows what kind of interpreter to run
MUSIC=/mnt/x/Music/Motivation/ - next is the folder which contains the songs from which a random one will be chosen. Define a variable with the folder and change the current folder to it
ls |sort -R |tail -1 |while read file; do - select a random song and assign it to variable file which will be used in next statement
rhythmbox "$MUSIC$file" & sleep 5m - start the music player with the selected song and keep the script for 5 minutes. The maximum length of song in this folder
killall rhythmbox - finally kill the music player

Once you have the script(for example name it motivation.sh, place it in your home and allow it's execution) you can easily schedule new cron job to run by:

open new terminal with CTRL + ALT +T
type:

crontab -e

add at the end of the file:

0 9-17 * * * DISPLAY=:0 /home/user/cron/motivation.sh

if you want the songs to be played every hour between 9 to 17. Happy listening!

Backed by science:

In 2005, research from the journal of Psychology of Music showed that software developers experienced more positive moods, better quality of work and improved efficiency when listening to music.

The Science Backed Ways Music Affects Your Brain and Productivity

Start motivation/relaxation video at given hour

Next example will demonstrate how to get pause for relax at your work with few lines of code and crontab daemon. This example helps me to schedule a meditation video and relax for 10 minutes several times per day. Basically you can use any video that you like or song- all depends on your personal preferences.

Usually I'm using meditation music or yoga videos shared by friends. If you don't have anything good in mind then you can download any video from Youtube and use it(just search for relaxing music or chill out).

The script is:

#!/bin/bash
vlc -f /home/user/Videos/Relaxing.mp4 & sleep 10m
killall vlc

save the script as motivation.sh in folder /home/user/cron/motivation.sh - or as you prefer
Start linux terminal
type:

crontab -e

add at the end of the file:

0 10 * * * DISPLAY=:0 /home/user/cron/motivation.sh

This will start VLC player with selected video at 10 in the morning. Will keep it running for 10 minutes and finally kill the player. So I get my break at 10:00 every working day.

How to download youtube video in Ubuntu / Linux Mint:

install youtube-dl by

sudo apt-get install youtube-dl

download video by:

youtube-dl https://www.youtube.com/watch?v=mkKDI6y2kyE

Bonus: For homework you can try to find out how to download multiple videos and update the script in order to play random one.

Backed by science:

regular breaks helps with:
- Increased productivity
- Improved mental well-being
- Creativity boost
- More time for healthy habits

Source: New Study Shows Correlation Between Employee Engagement And The Long-Lost Lunch Break

Open random text file at Linux Mint start

This is example demonstrates how to automate task at the start of your computer. For me it's very easy to like an article or saying but even easier to forgot about it. So I have a script which loads random piece of text at the start of the machine. In this way I can keep few good reads from a book, article or my to do list :)

Below you can find the script for the automation:

cd /home/user/Documents/Books/Motivation
ls |sort -R |tail -1 |while read file; do
    xed $file
done

: '
This script open random text file
'

This script show similar technique as the first example. There are two new things in it:

xed - using a xed text editor in order to open the text file
demonstrating how to use bash comments in order to get meaningful information for your scripts

Finally in order to schedule it for the start of your Machine you can do:

terminal
type:

crontab -e

and then:

@reboot /home/user/cron/openRandomText.sh

Or you can use the system start applications like (for Linux Mint):

Main Menu
Preferences
Startup Applications
Add ( the plus button)
Custom command and type:
- Name
- Command
- Startup Delay

Now you will have all important notes before starting your working day (or any other day).

Backed by Science:

We remember things because they either stand out, they relate to and can easily be integrated in our existing knowledge base, or it’s something we retrieve, recount or use repeatedly over time

Source: How to improve your memory, according to neuroscience

Scrape webdata and load the LibreOffice ( or csv, excel etc)

Web data is very important and precious nowadays. Collection is done fast, reliably and in automated manner with special programs - called spiders or scrapers. To write a personal spider is very easy by using modules like scrapy. I'm using spiders when I need to buy something and regularly checking the prices of the targeted product - in this way I can get really good promotion.

Below you can find very simple spider crawling website and collecting data:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'http://quotes.toscrape.com/page/1/',
            'http://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = 'quotes-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)

You can find more info at: Scrapy Tutorial

once your spider is complete you can create script which to automate its execution. Below you can find an example script:

#!/bin/bash
cd /home/user/environments/venv36
source ./bin/activate
cd /home/user/PycharmProjects/python/src/spiders/

now=$(date +"_%Y-%m-%d")

scrapy runspider my_spider.py -o - >data/my_spider/my_spider$now.csv -t csv
python helpers/pandas_after.py

libreoffice --calc /home/user/data/my_spider$now.csv

Now lets talk a bit about the code above:

first two lines are used to activate special python environment for this spider
then working folder is changed to the spider project
the line below run the spider and save the output as csv file
- scrapy runspider my_spider.py -o - >data/my_spider/my_spider$now.csv -t csv
command python helpers/pandas_after.py - call pandas function which clean, add new columns and sort data
the final line open the newly created CSV file in Libre Office

This script is scheduled on my machine with the following crontab:

55 9 * * * DISPLAY=:0 /home/user/cron/spider_script.sh

P.S. If you delete (incidentally by crontab -r) your crontab jobs they can be restored (if they were running) by checking the logs:

grep 'CRON.*username' /var/log/syslog

where username is your user name.

Generate simple report as doc file, create jupyter notebook and send mail

The final example shows a small python program which collect data from various sources like Github. The program below will collect Github information about given account:

class Github:
    def getStars(self):
        for repo in g.get_user('Softhints').get_repos():
            print(repo)
            stars = repo.stargazers_count
            forks = repo.forks_count
            print(stars)
            print(forks)
            print(repo.get_topics())

            open_issues = repo.get_issues(state='open')
            for issue in open_issues:
                print(issue)
            return stars, forks

I have similar small classes for different API-s and sites like Youtube, Facebook, twitter etc. They collect information which is interesting for me on weekly basis.

Another really useful tool is Selenium who can help with web automation. Below you can find basic automation for Google search:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# options.add_argument('--headless') 

executable_path = r'/home/userchromedriver_linux64/chromedriver'

options = Options()
driver = webdriver.Chrome(    executable_path=executable_path,    chrome_options=options)

driver.get("https://google.com")
search = driver.find_element_by_name('q')
search.send_keys("softhints python automation")

print(driver.page_source.encode("utf-8"))
driver.quit()

For this code you need to:

get latest chrome driver from: ChromeDriver - WebDriver for Chrome
update the script with the latest path
find the element on the web page
- on this step you can use really good browser extension - Katalon - Simplify API, Web, Mobile Automation Tests
sent information for the search
get the page source

The final data is collected into csv file stored on the computer, loaded into Jupyter Notebook and finally small report is sent to me. If I find the report interesting I can go into the notebook for more details.

The sample code for the script is shown below:

#!/bin/bash
cd /home/user/environments/venv36
source ./bin/activate
cd /home/user/PycharmProjects/automation/src/

python weekly.py
python create_report.py
python create_notebook.py
python send mail.py

Different sections of the example code can be found below:

Python how to create doc files

from docx import Document
from docx.shared import Inches

document = Document()

document.add_heading('Document Title', 0)

p = document.add_paragraph('A plain paragraph having some ')
p.add_run('bold').bold = True

document.add_heading('Heading, level 1', level=1)
document.add_paragraph('Intense quote', style='Intense Quote')

document.add_paragraph(
    'first item in unordered list', style='List Bullet'
)
document.add_paragraph(
    'first item in ordered list', style='List Number'
)

document.save('demo.docx')

more info here:

Python library for creating and updating Microsoft Word (.docx) files.

Python how to create jupyter notebook

import nbformat as nbf

nb = nbf.v4.new_notebook()
text = """\
# Weekly report
Report for github, youtube, etc."""

code = """\
# Import pandas 
import pandas as pd 

# reading csv file  
pd.read_csv("report.csv") """

nb['cells'] = [nbf.v4.new_markdown_cell(text),
               nbf.v4.new_code_cell(code)]
fname = 'report.ipynb'

with open(fname, 'w') as f:
    nbf.write(nb, f)

Notebook can be created in several different ways. I have two methods the first one (the one above use nbformat) the second one use json templates and create notebooks using the ipython API.

Python how to send emails

# Import smtplib for the actual sending function
import smtplib

# Import the email modules we'll need
from email.message import EmailMessage

# Open the plain text file whose name is in textfile for reading.
with open(textfile) as fp:
    # Create a text/plain message
    msg = EmailMessage()
    msg.set_content(fp.read())

# me == the sender's email address
# you == the recipient's email address
msg['Subject'] = 'The contents of %s' % textfile
msg['From'] = me
msg['To'] = you

# Send the message via our own SMTP server.
s = smtplib.SMTP('localhost')
s.send_message(msg)
s.quit()

more info here: Python email: Examples

Conclusion: Automation can be your best friend or your nightmare - all depends on you. Automation may increase dependency and complexity in general but it can save time and energy.

My personal goal is to automate anything which can be automated and it's going to be repetitive in time. Invest 1 day today and save 1 month in future. This rule helped me to do more work for less time. Of course there is a risk of being late with a given task but later this investment will pay off.