DEV Community: eLabFTW

Getting back that precious disk space

eLabFTW — Mon, 18 May 2020 20:33:55 +0000

Hello devs,

I you're like me and you like to try things out, work on several projects in several languages, chances are that you've been cluttering your computer with gigabytes of files that are not worth keeping.

I'm talking about all the cached packages from package managers, old docker images, and containers you started without the --rm flag.

This post is an attempt at regrouping the commands to clear that space. It's written for GNU+Linux users and will probably work for Mac. Some of the commands will also work on Windows (but then if you're doing dev on Windows I'm deeply sorry for you).

If you want to go along and execute the commands, start by checking your disk usage with df -h first so you know how much space you'll get back :)

WARNING: we're going to delete files here! It shouldn't cause any issue because it's only cached files that you can easily get back, but be careful about following a random blog post on the internets telling you to run a bunch of rm -rf commands ;) I shall not be held responsible for any damage caused by executing the commands described below.

I'm starting this article with 94 Gb free on /.

npm (javascript)

npm cache clean --force
yarn cache clean
rm -r ~/.cache/typescript/*

composer (php)

composer clear-cache

cargo (rust)

There is no baked in command for that. See GitHub issue.

rm -rf ~/.cargo/git ~/.cargo/registry

We don't delete the whole ~/.cargo folder as it might contain useful binaries in bin/.

docker

docker system prune -a

I like this command because it tells you how much disk space you got back after running it, and it really removes everything not currently in use. If you've been using docker for while and never used it, you might get as much as 40 Gb or more back! (yes images take up space!)

pip/pipenv (python)

rm -r ~/.cache/pip{,env}
rm -r ~/.local/share/{virtualenvs,jupyter}

go

go clean -cache -modcache -i -r

The -i flag causes clean to remove the corresponding installed archive or binary (what 'go install' would create). The -r flag causes clean to be applied recursively to all the dependencies of the packages named by the import paths.

css

Nope, just kidding.

some non-dev stuff

While we're at it...

rm -r ~/.cache/{mozilla,chromium,thumbnails}

You might want to have a look into this folder, as it's quite possible that a game you played 20 minutes in 2011 is taking up 3 Gb... du -sh ~/.cache/*.

Let's not forget the package managers!

Archlinux commands (BTW, I use Arch!):

sudo pacman -Sc (add another c to really remove everything)
yay -Sc

For Debian/Ubuntu/Mint users:

sudo apt-get clean
sudo apt-get autoclean

For Fedora/CentOS users:

dnf clean all

Conclusion

Please let me know in the comments how much disk space you got back! I had 94 Gb free at the beginning, now it's showing 128 Gb! That's 34 Gb back from the dead!! :)

Header image "GrandPerspective disk map" by Lars Plougmann is licensed under CC BY-SA 2.0

10 steps for securing a PHP app

eLabFTW — Mon, 14 Oct 2019 14:49:21 +0000

Hello PHP devs. In this post, I'll try and provide you with some concrete steps you can make to improve the security of your PHP app. I'm focusing on the PHP configuration itself, so we won't talk about SQL injections, HTTPS or other non PHP related issues.

I'll illustrate examples with bash lines from my docker-entrypoint.sh script, but of course you can apply it to a non docker environment.

Sessions

Use longer session id length

Increasing the session id length will make it harder for an attacker to guess it (via bruteforce or more likely side-channel attacks). Session ID length can be between 22 to 256 characters. The default is 32.

sed -i -e "s/session.sid_length = 26/session.sid_length = 42/" /etc/php7/php.ini

(don't ask me why it's 26 on Alpine Linux...)

You might also want to look at session.sid_bits_per_character.

Use a custom session save path with restrictive permissions

Only nginx/php needs to access the sessions, so let's put them in a special folder with restrictive permissions.

sed -i -e "s:;session.save_path = \"/tmp\":session.save_path = \"/sessions\":" /etc/php7/php.ini
mkdir -p /sessions
chown nginx:nginx /sessions
chmod 700 /sessions

Of course, if you use Redis to handle sessions, you don't care about this part ;)

Secure session cookies

session.cookie_httponly to prevent javascript from accessing them. More info.

sed -i -e "s/session.cookie_httponly.*/session.cookie_httponly = true/" /etc/php7/php.ini
sed -i -e "s/;session.cookie_secure.*/session.cookie_secure = true/" /etc/php7/php.ini

session.cookie_secure to prevent your cookie traveling on cleartext HTTP.

session.cookie_samesite to prevent Cross-Site attacks. Only for recent PHP/browsers.

Use strict mode

Due to the cookie specification, attackers are capable to place non removable session ID cookies by locally setting a cookie database or JavaScript injections. session.use_strict_mode can prevent an attacker initialized session ID of being used.

Restrict lifetime

Sessions should close with the browser. So set session.cookie_lifetime to 0.

Open_basedir

open_basedir is a php.ini config option that allows you to restrict the files/directories that can be accessed by PHP.

sed -i -e "s#;open_basedir =#open_basedir = /elabftw/:/tmp/:/usr/bin/unzip#" /etc/php7/php.ini

Here I have added unzip as it is used by composer. /elabftw is where all the source php files are located. And I don't remember why /tmp is here but there is surely a reason.

Disable functions

Be careful with this, as you can very easily mess up an app. But this is definitely something to look into. Let's say an attacker somehow managed to upload a webshell, with this correctly in place, the webshell won't really work as shell_exec and friends will be disabled. I'm providing a list that works for elabftw but it's not 100% complete.

sed -i -e "s/disable_functions =/disable_functions = php_uname, getmyuid, getmypid, passthru, leak, listen, diskfreespace, tmpfile, link, ignore_user_abort, shell_exec, dl, system, highlight_file, source, show_source, fpaththru, virtual, posix_ctermid, posix_getcwd, posix_getegid, posix_geteuid, posix_getgid, posix_getgrgid, posix_getgrnam, posix_getgroups, posix_getlogin, posix_getpgid, posix_getpgrp, posix_getpid, posix_getppid, posix_getpwnam, posix_getpwuid, posix_getrlimit, posix_getsid, posix_getuid, posix_isatty, posix_kill, posix_mkfifo, posix_setegid, posix_seteuid, posix_setgid, posix_setpgid, posix_setsid, posix_setuid, posix_times, posix_ttyname, posix_uname, phpinfo/" /etc/php7/php.ini

Disable url_fopen

The option allow_url_fopen is dangerous. Disable it. More info here.

sed -i -e "s/allow_url_fopen = On/allow_url_fopen = Off/" /etc/php7/php.ini

Disable cgi.fix_pathinfo

You don't want to allow non PHP files to be executed as PHP files, right? Then disable this. More info.

sed -i -e "s/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/g" /etc/php7/php.ini

Hide PHP version

To finish, a nobrainer:

sed -i -e "s/expose_php = On/expose_php = Off/g" /etc/php7/php.ini

That's it for now. I hope you'll find this post useful and improve your configurations ;)

Let me know in the comments if I've missed something important!

~Nico

Optimizing your PHP app speed

eLabFTW — Thu, 14 Mar 2019 13:08:54 +0000

This post is intended for PHP devs. I'll show you four ways to improve the speed of your PHP app easily. I'll skip the part where I tell you to use PHP 7, you must know by now that the speed improvement is dramatic… (and PHP 5.x is EOL anyway so…)

TL;DR: -a flag for composer, use opcache, use template engine cache, use fully qualified function names.

0. Use Composer optimization

The prod and dev environments are a little bit different. It shouldn't be a surprise to you if I tell you that you should not install the dev dependencies in prod, right?



composer install --no-dev

That's basic. But did you know you can optimize the autoloader? Because the classes won't be changing once it's deployed, you can add a flag to composer (-a) that will improve the speed of autoloading:



composer install --no-dev -a

Ok, but what does it do you'll ask?

From the help it says: "the Composer autoloader will only load classes from the classmap". It also implies "optimize-autoloader" so you don't have to add it too. You can see it as a way to say "hey, no more classes will be added so you don't need to scan the filesystem for a new class, just use the ones in the autoloader".

1. Use opcache

When a PHP file is read, it is converted in opcode, and then executed by the engine. Because in prod your PHP files won't change, you don't want to convert them to opcode every single time. Enter opcache.

Using opcache can dramatically increase the speed of your PHP application. Make sure that opcache_enable=1 is uncommented in your php.ini file. It will store the opcode for the executed files.

But don't enable it in your dev environment ;) (unless you enable opcache.validate_timestamps and set opcache.revalidate_freq to 0 (thx u/iluuu))

I recommend the opcache-status tool to monitor the use of opcache (at least in the beginning).

2. Use a template engine with cache

Templating engines like Twig can create a cache of the generated PHP code. The speed gain here can be dramatic. Again, you only want to cache the templates in prod, not in dev. But make sure that the cache is setup. For Twig see the documentation.

3. Qualify standard functions

Normally your code is namespaced (right?). So what happens when you call a function from the standard PHP library? The "compiler" (opcode producer) will look into the current namespace, then go up, and eventually use the global namespace. This means that if you add a "\" in front of standard functions (so effectively namespacing it in the global namespace explicitely), it will result in less opcode instructions, and that means faster code execution. Think it's one of those useless micro-optimization like the use of single vs. double quotes? You're correct, your app won't suddenly be faster (only very marginally), but if we can save CPU cycles, why not do it?

Note also that I prefer to add use function count; (for the count() function) in the use block, it's prettier than using \.

Conclusion

The little things I'm showing in this post won't make your app faster, at best it'll improve marginally. You can consider them as "good practice" because why not take some speed gains if you can, but if you want to get serious about optimizing your PHP app, get a profiler and optimize your SQL queries, because that's what is causing an issue, not the few milliseconds you might get with this kind of things ;)

That's all folks, have fun coding!

And if you like PHP, I'm always happy with contributions on the eLabFTW project, an open source electronic lab notebook. =)

Dependencies management in 2019: a review

eLabFTW — Sun, 03 Mar 2019 22:58:09 +0000

Hello folks,

The aim of this post is to make a (mostly) complete and accurate view of the different solutions available to you when you produce code and want to manage your dependencies like a boss, for different languages. We'll see how languages have different approaches to dependency management.

I'll be writing about these 9 languages:

PHP
Python
Rust
Go
Javascript
Elixir
Nim
C#
Ruby

Let's start with an easy one: PHP

Composer

composer / composer

Dependency Manager for PHP

Dependency Management for PHP

Composer helps you declare, manage, and install dependencies of PHP projects.

See https://getcomposer.org/ for more information and documentation.

Installation / Usage

Download and install Composer by following the official instructions.

For usage, see the documentation.

Packages

Find public packages on Packagist.org.

For private package hosting take a look at Private Packagist.

Community

Follow @packagist or @seldaek on Twitter for announcements, or check the #composerphp hashtag.

For support, Stack Overflow offers a good collection of Composer related questions, or you can use the GitHub discussions.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project and its community you agree to abide by those terms.

Requirements

Latest Composer

PHP 7.2.5 or above for the latest version.

Composer 2.2 LTS (Long Term Support)

PHP versions 5.3.2 - 8.1 are still supported via the…

View on GitHub

Composer is the de facto standard for dependencies management in PHP. It's very hard today to get anything done in PHP without it. I recommend reading this article from Théo Fidry if you want to know more about the limitations of composer and the workarounds (like using PHARs).

PEAR

pear / pear-core

This is the definitive source of PEAR's core files.

PEAR - The PEAR Installer

What is the PEAR Installer? What is PEAR?

PEAR is the PHP Extension and Application Repository, found at http://pear.php.net.

The PEAR Installer is this software, which contains executable files and PHP code that is used to download and install PEAR code from pear.php.net.

PEAR contains useful software libraries and applications such as MDB2 (database abstraction), HTML_QuickForm (HTML forms management), PhpDocumentor (auto-documentation generator), DB_DataObject (Data Access Abstraction), and many hundreds more. Browse all available packages at http://pear.php.net, the list is constantly growing and updating to reflect improvements in the PHP language.

Warning

Do not run PEAR without installing it - if you downloaded this tarball manually, you MUST install it. Read the instructions in INSTALL prior to use.

Documentation

Documentation for PEAR can be found at http://pear.php.net/manual/. Installation documentation can be found in the INSTALL file included in this tarball.

Tests

Run the…

View on GitHub

This is the old school player. It installs dependencies to the system. Only use it if your distribution is not shipping the extension in a package already. Note that you can use composer to install a PEAR package, see this part of phptherightway for more.

That's it for PHP. I'm pretty sure there are no other PHP dependencies management applications out there, but if I'm wrong please let me know in the comments section.

Second language: Python

Pip

pypa / pip

The Python package installer

pip - The Python Package Installer

pip is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes.

Please take a look at our documentation for how to install and use pip:

We release updates regularly, with a new version every 3 months. Find more details in our documentation:

If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms:

If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms:

Code of Conduct

Everyone interacting in the pip project's codebases, issue trackers, chat rooms, and mailing lists is expected…

View on GitHub

pip is the goto package manager for python, it has its drawbacks but is a very popular tool to install Python dependencies. BTW, you should Stop using sudo pip install. Use pip with moderation (and the --user flag!).

Venv

The official module for managing your project's dependencies inside a virtual environment, thus not polluting the rest of the OS.

Virtualenv

pypa / virtualenv

Virtual Python Environment builder

virtualenv

A tool for creating isolated virtual python environments.

Code of Conduct

Everyone interacting in the virtualenv project's codebases, issue trackers, chat rooms, and mailing lists is expected to follow the PSF Code of Conduct.

View on GitHub

This project started in 2011 and is very useful to isolate dependencies for a project. Part of this project is now venv in the standard python library. So you can think of virtualvenv as a more feature complete venv.

Pipenv

pypa / pipenv

Python Development Workflow for Humans.

Pipenv: Python Development Workflow for Humans

Pipenv is a Python virtualenv management tool that supports a multitude of systems and nicely bridges the gaps between pip, python (using system python, pyenv or asdf) and virtualenv Linux, macOS, and Windows are all first-class citizens in pipenv.

Pipenv automatically creates and manages a virtualenv for your projects, as well as adds/removes packages from your Pipfile as you install/uninstall packages. It also generates a project Pipfile.lock, which is used to produce deterministic builds.

Pipenv is primarily meant to provide users and developers of applications with an easy method to arrive at a consistent working project environment.

The problems that Pipenv seeks to solve are multi-faceted:

You no longer need to use pip and virtualenv separately: they work together.
Managing a requirements.txt file with package hashes can be problematic. Pipenv uses Pipfile and Pipfile.lock to separate abstract dependency declarations from the last tested…

View on GitHub

Pipenv started in early 2017 and was very quickly popular. It's my favorite, too. It works very similarly to composer with a lockfile and all the dependencies are installed in a special folder. To run the script in the environment, you need to prepend pipenv run. See my previous post about it. Note that the official Python doc is recommending pipenv for managing dependencies.

Something interesting to note, is that all of the above-mentioned projects are all hosted in the github.com/pypa namespace, the Python Package Authority, a group dedicated to maintaining the python packagers. I think it's nice to have this kind of "authority" for improving the ecosystem in Python.

The third language we'll talk about is: Rust

Cargo

Well, here it's a nobrainer, because (AFAIK) there is only one solution, and it's Cargo.

rust-lang / cargo

The Rust package manager

Cargo

Cargo downloads your Rust project’s dependencies and compiles your project.

To start using Cargo, learn more at The Cargo Book.

To start developing Cargo itself, read the Cargo Contributor Guide.

The Cargo binary distributed through with Rust is maintained by the Cargo team for use by the wider ecosystem For all other uses of this crate (as a binary or library) this is maintained by the Cargo team, primarily for use by Cargo and not intended for external use (except as a transitive dependency). This crate may make major changes to its APIs.

Code Status

Code documentation: https://doc.rust-lang.org/nightly/nightly-rustc/cargo/

Compiling from Source

Requirements

Cargo requires the following tools and packages to build:

cargo and rustc
A C compiler for your platform
git (to clone this repository)

Other requirements:

The following are optional based on your platform and needs.

pkg-config — This is used to help locate…

View on GitHub

Cargo is great. Cargo is shit. YMMV. Honestly, it's pretty cool to have one official tool for dependencies management, at least it saves you the time you would have spent choosing one! It installs dependencies per-project but can also be used for system-wide packages. Interestingly, it uses the TOML syntax for the configuration file, whereas other projects chose YAML or JSON.

Let's move on to the fourth language: Go

go get is is the way to install a dependency in Go. But the issue is that it'll be installed system-wide (in $GOPATH). Say hello to conflicts and breaking changes because you can only have one version of a lib…

This led to other approaches like git submodules or project specific $GOPATH and a lot of developers being angry at Go.

However, with Go version 1.11 (released in August 2018) there is now something called modules. Check out the wiki page to know more about Go modules. I believe this will be the future of Go dependencies management, because fuck system-wide dependencies.

What do I see? Is that the fifth language? Yep: Javascript

Aaaaahhh, javascript and the dependencies, what a wonderful world! :)

Npm

npm / cli

the package manager for JavaScript

npm - a JavaScript package manager

Requirements

You should be running a currently supported version of Node.js to run npm. For a list of which versions of Node.js are currently supported, please see the Node.js releases page.

Installation

npm comes bundled with node, & most third-party distributions, by default. Officially supported downloads/distributions can be found at: nodejs.org/en/download

Direct Download

You can download & install npm directly from npmjs.com using our custom install.sh script:

curl -qL https://www.npmjs.com/install.sh | sh

Node Version Managers

If you're looking to manage multiple versions of Node.js &/or npm, consider using a node version manager

Usage

npm <command>

Links & Resources

Documentation - Official docs & how-tos for all things npm
- Note: you can also search docs locally with npm help-search <query>
Bug Tracker - Search or submit bugs against the CLI
Roadmap - Track & follow along with our public…

View on GitHub

Everyone knows npm. You define your dependencies in a package.json and install them in the node_modules folder. Since version 5, there is a package-lock.json allowing you to have reproducible builds.

Yarn

yarnpkg / yarn

The 1.x line is frozen - features and bugfixes now happen on https://github.com/yarnpkg/berry

ℹ️ Important note

This repository holds the sources for Yarn 1.x (latest version at the time of this writing being 1.22). New releases (at this time the 3.2.3, although we're currently working on our next major) are tracked on the yarnpkg/berry repository, this one here being mostly kept for historical purposes and the occasional hotfix we publish to make the migration from 1.x to later releases easier.

If you hit bugs or issues with Yarn 1.x, we strongly suggest you migrate to the latest release - at this point they have been maintained longer than 1.x, and many classes of problems have already been addressed there. By using the nodeLinker setting you'll also have the choice of how you want to install your packages: node_modules like npm, symlinks like pnpm, or manifest files via Yarn PnP.

Fast, reliable, and secure dependency management

Fast: Yarn caches every package it has…

View on GitHub

Yarn had reproducible builds well before npm. It also doesn't suffer from the sometimes poor decisions taken by npm's project leaders. It's fast, works well. It's my tool of choice.

Bower

Come on now! We said 2019 in the title! Forget about bower.

I will not talk about Parcel or Webpack as these are bundlers, not package managers.

Note that you can use either yarn or npm with the exact same package.json!

One thing to note about packages in javascript, is that it has become completely crazy, with micro libraries consisting of one line of code more or less, being used by hundred of thousands of other packages. For fun, go into a Javascript project and try this command:

ls node_modules | grep '^is*'

How fun is it to have is-obj, is-object, is-plain-obj, is-plain-object or things like isarray, but also is-arrayish… Do we really need a package is-windows when all it does is:

return process && (process.platform === 'win32' || /^(msys|cygwin)$/.test(process.env.OSTYPE)); ?

And good luck if you want to look at the code of all of these libraries for your project. That lead to the left-pad chaos.

So my advice here is to think hard before installing a javascript dependency, as it might come with hundred of these little library, and one day, one of them will become malicious (see malicious packages on npmjs.com).

You can use Depcheck to try and cleanup a bit your dependencies.

Let's talk about a more exciting language, like: Elixir

Elixir is a bit like Rust when it comes to dependency management: one official tool that gets the job done: Mix (or Hex for Erlang). Nothing more to say here, it works well enough and lets you focus on other things.

Okay so what do we have left? Nim?

Nim is a pretty new and cool language, and the name of the package manager is sooooo cute <3: Nimble. Which reminds me of this guy:

Nothing to see here, just use the one provided and get coding!

The next language is C#

Nuget

NuGet / NuGet.Client

Client Tools for NuGet - including Visual Studio extensions, command line tools, and msbuild support. (Open issues on https://github.com/nuget/home/issues)

NuGet Client Tools

This repo contains the following clients:

Open Source Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Getting Started guide

For how to contribute to this repo follow the Contributing doc.

NuGet/Home repo

The NuGet/Home repo is the starting point for all things NuGet. It has the issue tracker and basic information about all things NuGet. Make sure to consult it before beginning your journey through NuGet code.

Feedback

File NuGet.Client bugs in the NuGet/Home.

License

Unless explicitly stated otherwise all files in this repository are licensed under the License in the root repository

View on GitHub

This is the standard package manager that comes with Visual Studio. It exists since 2010 and is developed by the .NET foundation. I don't have much more to say about it because it's really windowsy and that's clearly not my platform of choice when it comes to anything vaguely related to computing.

Last but not least: Ruby

Rubygem

rubygems / rubygems

Library packaging and distribution for Ruby.

RubyGems

RubyGems is a package management framework for Ruby.

A package (also known as a library) contains a set of functionality that can be invoked by a Ruby program, such as reading and parsing an XML file We call these packages "gems" and RubyGems is a tool to install, create, manage and load these packages in your Ruby environment.

RubyGems is also a client for RubyGems.org, a public repository of Gems that allows you to publish a Gem that can be shared and used by other developers. See our guide on publishing a Gem at guides.rubygems.org

Getting Started

Installing and managing a Gem is done through the gem command. To install a Gem such as Faraday:

gem install faraday

RubyGems will download the Faraday Gem from RubyGems.org and install it into your Ruby environment.

Finally, inside your Ruby program, load the Faraday gem and start hacking:

require 'faraday'

…

View on GitHub

This is the standard way to get dependencies for a ruby project. It can be seen as pip for ruby, because it installs packages system-wide. You can install them for your user with the --user-install flag (better: add gem: --user-install to ~/.gemrc to always install gems in your home).

So what about managing dependencies per-project you'll ask?

Bundler: a gem to bundle gems

rubygems / bundler

Manage your Ruby application's gem dependencies

Bundler is now maintained in the rubygems/rubygems repository.

Bundler: a gem to bundle gems

Bundler makes sure Ruby applications run the same code on every machine.

It does this by managing the gems that the application depends on. Given a list of gems, it can automatically download and install those gems, as well as any other gems needed by the gems that are listed. Before installing gems, it checks the versions of every gem to make sure that they are compatible, and can all be loaded at the same time. After the gems have been installed, Bundler can help you update some or all of them when new versions become available. Finally, it records the exact versions that have been installed, so that others can install the exact same gems.

Installation and usage

To install (or update to the latest version):

gem install bundler

To install a prerelease version (if…

View on GitHub

You get a Gemfile and Gemfile.lock file, and you're sure the project will run everywhere with the same versions of gems.

But there is something interesting about the ruby ecosystem: it exists a lot more projects to switch between different ruby versions (chruby, rbenv, rvm, uru) than to install dependencies.

Conclusion

As you can see, some languages have it all figured out for you (rust, nim), while for others you need to make a choice (python, javascript), and for some you have system-wide and project-wide solutions (python, ruby). I don't quite understand why Go, a "recent" language didn't do a better job at dependency management. But this fits well with the view that "Go was created like we were still in the 70's".

Take home messages

Install your dependencies per project with a reproducible build (lock down versions of packages) to avoid surprises and to get something portable.
Try to avoid installing too many dependencies (javascript, I'm looking at you!).
Don't ship the dependencies with your project but ship the lockfile ;)

That's all folks, please leave a comment if you think I forgot something important or if I got something wrong ;)

Have fun coding!

Improving the python REPL (autoload modules and history)

eLabFTW — Wed, 24 Oct 2018 10:36:14 +0000

Hello to you dear reader,

When doing python code, it might be useful to fire up the REPL (Read-Eval-Print-Loop) by simply typing python in a terminal to test some expressions.

But the problem is that it is very barebone. No history, no modules loaded by default. I don't know about you but 99% of the time I'll need to use numpy and pandas at the minimum. Probably matplotlib too.

Who has time to type import xyz every time? Not me.

So here is how to autoload modules in python's REPL:

Start by creating a ~/.pystartup file

As you can see, this file will also configure the ~/.pyhistory file so python doesn't forget everything when you close the REPL.

It is recommended to install these often used modules with pip install --user module-name, see my previous post: Stop using sudo pip install

Configure the PYTHONSTARTUP env var

Add this to your ~/.zshrc, ~/.fishrc, ~/.bashrc or whatever you use:

# Note that '~' is not expanded, so use the full path here
export PYTHONSTARTUP=/home/user/.pystartup

That's it, now try again to fire up python and check that the modules are there.

There you go :) Now go write so amazing code!

Cheers,
~Nico

Stop using sudo pip install

eLabFTW — Wed, 17 Oct 2018 21:23:48 +0000

We've all done it:

pip install numpy
# run into permissions issues
sudo pip install numpy # or "sudo !!" for the power users ;)
~~~{% endraw %}

End of story, it works, it's what is written in the README so what's wrong with it?

You see, when you install a library through pip, you are executing {% raw %}`setup.py`{% endraw %} with root permissions. This file could harbor malicious code. You are also messing with the system libraries and that invariably leads to issues down the line.

Relevant XKCD (thx u/truh):

![xkcd](https://imgs.xkcd.com/comics/python_environment.png)

A much better and secure way would be:{% raw %}

~~~bash
pip install --user numpy # libs will be installed in ~/.local/lib
~~~{% endraw %}

This is better, and can be used for installing applications, but it doesn't solve the problem of having different versions needs for different python projects. Enter [pipenv](https://pipenv.readthedocs.io/en/latest/). {% raw %}`pipenv` is to python what `composer` is to PHP. It lets you **easily** install and use libraries per project. It's not the only tool allowing you to do that, but it's the one I use so it's the one I'm gonna present you. Example:

~~~bash
pipenv install numpy matplotlib pandas
# to start your program
pipenv run ./crunch-data.py
# to install libs from another machine, after a git pull:
pipenv sync
# to get a shell in the env (like `source myenv/bin/activate` for venv)
pipenv shell
~~~

This allows a very reproducible environment for your program, without resorting to Docker and without messing up user or system libraries. Save yourself from future bugs and start using pipenv, venv, conda or virtualenv right away! It's much better than {% raw %}`requirements.txt`{% endraw %} + pip. :)

Cheers,
~Nico