DEV Community: theBridge2

Sequelize seeders for static data

theBridge2 — Tue, 24 Dec 2024 12:38:11 +0000

Very brief post on how to do seeders in sequelize. Seeders are how you create static data in a database that you want to be present without the users having to create it.

The goal of this is to add some static data to my very basic task type table in my todo app defined by this model:

module.exports = (sequelize, Sequelize) => {
    const static_task_type = sequelize.define("static_task_type", {
      id: {
        type: Sequelize.INTEGER,
        primaryKey: true,
        autoIncrement: true
      },
      task_type: {
        type: Sequelize.STRING,
        allowNull: false
      }
    });
    static_task_type.associate = function (models) {

    };

    return static_task_type;
  };

To create a seeder template:

npx sequelize-cli seed:generate --name add-static-task-types

This is filling in the seeder template with a couple of task types.

'use strict';

/** @type {import('sequelize-cli').Migration} */
module.exports = {
  async up (queryInterface, Sequelize) {


  await queryInterface.bulkInsert('static_task_types',[
      { id: 1, task_type: 'Deep' ,createdAt: new Date(),updatedAt: new Date()},
      { id: 2, task_type: 'Shallow' ,createdAt: new Date(),updatedAt: new Date()},
      { id: 3, task_type: 'Phone Call' ,createdAt: new Date(),updatedAt: new Date() },
      { id: 4, task_type: 'Errands' ,createdAt: new Date(),updatedAt: new Date()}]

      ) 
  },



  async down (queryInterface, Sequelize) {
    await queryInterface.bulkDelete('static_task_types', null, {
      truncate: true,
      cascade: true, // Optional: Will also delete dependent rows if foreign keys are used
      restartIdentity: true, // Optional: Resets auto-increment counters
    });
  }
};

NOTE: don't forget to add the createdAt and updatedAt columns, otherwise you will get the following error:

ERROR: null value in column "createdAt" of relation "static_urgencies" violates not-null constraint
ERROR DETAIL: Failing row contains (1, Now, null, null).

To run the seed:

npx sequelize-cli db:seed:all

Success as shown in DBeaver window:

Sequelize migrations

theBridge2 — Tue, 24 Dec 2024 12:08:03 +0000

Given the following sequelize model for my basic todo project's metadata table, I'd like to make some changes using sequelize's migrations.

module.exports = (sequelize, Sequelize) => {
    const Metadata = sequelize.define("metadata", {
      id: {
        type: Sequelize.INTEGER,
        primaryKey: true,
        autoIncrement: true
      },     
      urgencyId: {
        type: Sequelize.INTEGER,
        allowNull: true
      },      
      dataId: {
        type: Sequelize.INTEGER,
        allowNull: true
      },      
      taskTypeId: {
        type: Sequelize.INTEGER,
        allowNull: true
      },  
      projectId: {
        type: Sequelize.INTEGER,
        allowNull: true
      },
      data_id: {
        type: Sequelize.INTEGER,
        allowNull: true
      },    
    });
    Metadata.associate = function (models) {
      Metadata.belongsTo(models.data,{
        foreignKey: {
          name: 'data_id',
          allowNull: false,
          hooks: true
        },
        onDelete: 'cascade'

      });

    };

    return Metadata;
  };

Sequelize utilizes migration files that are essentially like database commits. To create a migrations file type in your command line:

npx sequelize-cli migration:generate --name data-columns-rename

This will create a migrations file with a template in /migrations folder and the name you provided in the command above (in this case data-columns-rename.js with the current date/time in front).

Template migrations file created for you:

'use strict';

/** @type {import('sequelize-cli').Migration} */
module.exports = {
  async up (queryInterface, Sequelize) {
    /**
     * Add altering commands here.
     *
     * Example:
     * await queryInterface.createTable('users', { id: Sequelize.INTEGER });
     */
  },

  async down (queryInterface, Sequelize) {
    /**
     * Add reverting commands here.
     *
     * Example:
     * await queryInterface.dropTable('users');
     */
  }
};

The async up command is the command it runs to try to make changes to the table, and the async down command runs to remove the changes. When you do your migrations, make sure the down commands are in reverse.

In my case, I'm trying to do some column name changes for some basic todo type table information so my filled in migrations table looks like this:

'use strict';

/** @type {import('sequelize-cli').Migration} */
module.exports = {
  async up (queryInterface, Sequelize) {
    await queryInterface.removeColumn('metadata','dataId')
// renameColumn: table, old column name , new column name
    await queryInterface.renameColumn('metadata','urgencyId','urgency_id')
    await queryInterface.renameColumn('metadata','taskTypeId','task_type_id')
    await queryInterface.renameColumn('metadata','projectId','project_id')

  },

  async down (queryInterface, Sequelize) {
    await queryInterface.addColumn('metadata','dataId', {
      type: Sequelize.INTEGER,
      allowNull: false
    })
    await queryInterface.renameColumn('metadata','urgency_id','urgencyId')
    await queryInterface.renameColumn('metadata','task_type_id','taskTypeId')
    await queryInterface.renameColumn('metadata','project_id','projectId')
  }
};

(These changes are simply me still not being used to snake case as desired by postgres)

Once the migrations table is ready, run it with this command

npx sequelize-cli db:migrate

Among other printouts from running the command, the key text for me to see success is:

== 20241224113716-metadata-columns-rename: migrating =======
== 20241224113716-metadata-columns-rename: migrated (0.065s)

And you can see the changes verified in this screenshot from DBeaver:

To run a specific migrations file:

npx sequelize-cli db:migrate --name 20241224113716-metadata-columns-rename

Let me know if there is interest in a post describing the basics of getting started with sequelize.

NPM Dependency error

theBridge2 — Mon, 23 Dec 2024 14:14:28 +0000

Making a quick post to remind myself in the future how to read these npm dependency errors.

Kudos to this stack overflow post: https://stackoverflow.com/questions/76039613/how-do-i-read-npm-dependency-conflict-errors

Here is my error with two key parts highlighted:

This is stating "for client@1.0.0 I have installed react 18.3.0 BUT react-dom@19.0.0 requires react 19.0.0."

Ok, so how do we resolve this? First of all, we need to make sure we understand the npm versioning scheme with the package.json file.

First step is to understand the semantic versioning system used by npm. A major version 5, minor version 10, and patch version 3 would be 5.10.3, or:

Semantic versioning = MAJOR.MINOR.PATCH

From this article https://flaviocopes.com/npm-semantic-versioning/ we get the following key points for how npm interprets what libraries and versions to install from your package.json file:

> any version higher than your current version
^ - allows updating minor and patch releases. Ex: ^5.1.3. allows updating 5.10.3, but will not allow going to major release 6.
~ allows only patch releases

Simplified version of my package.json file

{
"dependencies" : {
  "react": "^18.3.1"
  "@types/react-dom": "^18.2.21",
  }
}

So the first problem here you see with my package.json file is that there is a new major release out now for react which is major version 19. My "^" in my package.json will not allow that to be installed.

To make sure I understood all the new versions I might need to consider in my dependencies I ran the following command

npm outdated

To fix my issues, i updated my package.json file to allow all minor and patch versions of react and react-dom in major version 19:

{
"dependencies" : {
  "react": "^19.0.0"
  "@types/react-dom": "^19.0.0",
  }
}

Also before running the handy npm i command to fix all of this, I deleted my node_modules and package-lock.json folder. I'm pretty sure this isn't necessary now with npm improvements made over time. Never hurts to do this when debugging though since both node_modules and package-lock.json get recreated on the install command.

Now my issues are resolved! Hope this helps.

Full Stack Javascript Web Development Concept map

theBridge2 — Sat, 23 Mar 2024 17:01:48 +0000

Concept Map

The full stack web development world and javascript keep moving at a tremendous pace. This is my attempt to capture some of the higher level concepts, languages and tools that a beginner and intermediate developer should be aware of.

Note - I included some other technologies that are not in my primary stack, but these are the ones I am most familiar with:

Text editor - VsCode
Source control - Git
Package manager - Npm
Javascript technology - React
Server - Node JS
Middleware - Express
ORM - Sequelize
Database - Was SQL Server, now moving to Postgres

Writing Code

text editor - a text editor is where you write your code. These range from command line tools to fully featured tools

VsCode - mainstream text editor. Has lots of add-ons to enable additional capability
Sublime - slightly more lightweight than vscode
Notepad++ - basic text editor with some neat addons.
vim - command line text editor. More powerful than nano. Two modes, edit, and command mode.
nano - command line text editor. simpler and less capable than vim.

source control - this is of utmost importance in todays distributed development environment. Even with a single developer, source control allows careful control over change and the ability to revert back to known working states when things go wrong.

git - this is the mainstream development tool. This runs on your own pc locally and connects to a cloud tool like github or gitlab.
github - cloud service that is the source for sharing code.

Setting up a project

command line - a way to control the computer with out a graphical interface. A number of tools must be manipulated with the command line.

bash - created in 1989, unix shell and command language.
zshell - or zsh, created in 1990 is a unix shell with improvements over bash including the ability to use shell scripts. Comes as default language in macOS starting in Catalina.
command shell - windows command line built to automate routine tasks
powershell - windows command line built to extend command shell to run more sophisticated commands called cmdlets

build tools and bundlers - are tools to create your web application
-webpack - is a tool that takes your source code from your project and converts it to a bundle that can be loaded properly by your browser.
-create react app - (CRA) is a tool used to create a number of files as a starting point for the project and packages it for deployment. CRA also uses a webpack under the hood, but the user doesn't have to understand these details.

Using others code

package manager - package managers exist to help distribute all the pieces of code people have published. Using other peoples code makes writing code faster.

npm - stands for Node Package Manager and is the default manager for NodeJS.
yarn - stands for Yet Another Resource Navigator. Yarn was developed to overcome some issues npm had at the time, but both have kept pace with each other and are very similar in capability.

Running a server

virtualization - is done to make it easier to distribute an application to multiple hardware environments.

docker - is an open platform for developing shipping and running applications. Docker uses a client server model. docs
podman - is a daemonless, open source linux native tool designed to deploy applications using containers and images. Podman was designed to be more secure than docker.docs

environment variables - are used primarily to store data the application needs but should not be stored in the source code like user name and password for user and database logins.

Javascript Packages

People write code to solve problems. The common problems everyone solves over and over end up in packages that everyone ends up re-using. Don't be afraid to use other peoples code. This is just a small sampling of available packages.

Front end only - These packages are designed to run in your browser not on the backend server.

material UI - open source library that implements google's styling for html components docs
redux - Redux is a key tool used in managing state across an application. This can be used with any web technology including React, Vue and Angular docs
react tables - react tables is a library that exists for react that make creating tables easier. There are other competing libraries including one in material UI worth considering. docs
bootstrap - toolkit for styling websites. Has lots of themes and capabilities. docs
d3 - very power visualization library enabling dynamic visualizations. docs

For both front end and backend

lodash - utility library enabling things like deep object comparison that aren't easy to do with javascript out of the box. docs
package.json - file used for a number of things including saving your dependencies for both front and backend, so that your package manager can easily install the packages you need on another machine.

Backend only

nodemon - enables automatic reload of your server after code change docs
rimraf - enables easy deleting of server files including npm packages and optimized build filesdocs

Front End

Also referred to as the client. The front end is what runs in your browser and can be run on many different computers at the same time.

Document object model - connects web pages to scripts by representing the structure of the page in memory. docs

Structure - the structure of your webpage is important
HTML - (HyperText Markup Language) is the code used to structure a web page and its content. docs

Styling - The difference between a webpage that looks great and one that looks outdated is almost always the styling.
CSS - Cascading Style Sheet allows you to format your webpage with color, size, spacing. docs

Images - A picture is worth 1,000 words, using image on a webpage is key to rich content.
SVG - Uses geometric shapes to render graphics. Resolution independent. docs
XML - stands for Extensible Markup Language and is similar to HTML but has no predefined tags to use. SVG stores its data in XML. docs
Canvas - Canvas uses pixels. Gets blurry when resized. Can be more performant. docs

Javascript in the browser
React - react is a library that gives developers an application programming interface (API) to manipulate the DOM (this is React's ReactDOM package). React uses components and JSX to make building reusable code easier. docs
JSX - is a syntax extension for React Javascript code that lets you write HTML-like markup in a javascript file. This makes it easier to write reusable HTML. docs
State - a key react concept that guides setting and storage of data between renders. docs
Hooks - a key react concept for logic triggered by state change docs
Vue - is a framework for building web interfaces. Vue is lightweight and best for small projects prioritizing speed over functionality. doc
Angular - web development framework. Angular is best for dynamic more feature rich sites. docs

Note: React vs Vue/Angular - The key difference between React and Vue/Angular is that react is a library not a framework and therefore requires you to do more work to build your project in library and tool selection. Vue/Angular are frameworks with reduced choices for project setup. Vue/Angular should generally be easier to learn but less flexible than React which is harder to learn but much more flexible.

Networking

To connect the front end with the backend you have to have some understanding of networking.

Ip addresses - IP stands for internet protocol. An ip address is a string of characters that identifies each computer on a network.

localhost - is the ip address of the current system you are developing on. This is the ip address used most commonly in development. The address 'localhost' and 127.0.0.1 are equivalent.

DNS - stands for Domain Name System which is a lookup table between domains we know like google.com and IP addresses like 104.234.123.11.

IPV4 - is the 4th generation of the internet protocol. IPv4 is in the format X.X.X.X where each X can be any value between 0 and 255. IPv4 has 4.29 billion possible combinations (which has now been reached)

IPV6 - is the 6th generation of the internet protocol. The key change is to create more available addresses. Using 128-bit addresses vs IPv4s 32 bit addresses, IPv6 can support 2^128 internet addresses, which is a lot. IPv6 format is xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx.

port - a port is the connection point of network connections. Each port is associated with a specific process or service. If you don't have the right ip addresses and ports, you will not be able to access your front end via the browser and your front end won't connect to the backend.

Subnet mask - a subnet mask is used to identify which network an IP address belongs to. Subnet masks can be created to create subnetworks that can't see each other. Often if networking issues happen, one needs to understand subnet masks to make sure two computers are on the same network.

Networking tools

ping - the most basic tool, that answers the question, "can I see a computer through a network?"
browser network tab - a more advanced tool that can be used inside the developer tools in a browser. This will show data received and sent to the browser.
wireshark - a very advanced network tool that allows you to see all the packet traffic in a network.

Back End

Server - a computer that provides functionality for other programs and devices called clients.

nodeJS - the mainstream javascript server choice. docs

Middleware - middleware is the layer between the response and the request.

express - one of the most popular middleware tools, lightweight and easy to learn. docs
NestJS - opinionated more scalable, but harder to learn docs

ORM - stands for object relational mapping. This enables you to write less SQL and makes Create, Read, Update, Delete (CRUD) operations much faster for a developer to implement.

Sequelize - modern Typescript and NodeJS ORM for Oracle, Postgres, MySQL, MariaDB, SQLite, SQL Server+docs

Typescript

Typescript can be used in both the front end and the back end and gives strong typing to javascript for easier error prevention. The learning curve can be difficult at times but the benefits are worth it! docs

Summary

This took me a while to build these concepts and relationships in my head. Hopefully they are helpful for some of you still trying to understand the fundamentals of full stack web development.

Let me know if there are more concepts you think I should include. Obviously at some point this becomes more advanced. I did introduce some advanced concepts, but I think you have to hear about things several times before you make the effort to learn them.

Run PostgreSQL in docker

theBridge2 — Sat, 23 Mar 2024 02:56:16 +0000

Overview

1) Setting up

Get docker
https://www.docker.com/get-docker

Go to terminal to prove the install succeeded.



docker --help

Get postgres image to use in docker.
https://hub.docker.com/_/postgres



docker pull postgres:alpine

This is the smallest postgres install image to use for testing.

2) Start docker postgres container



docker run --name any_image_name -e POSTGRES_PASSWORD=mypassword -d -p 5432:5432 postgres

-> -e creates an environment variable
-> -p 5432:5432 exposes port 5432 to external to the container

3) Test docker image with psql

Link to my PostgreSQL cheat sheet

Useful reference: https://stackoverflow.com/questions/37694987/connecting-to-postgresql-in-a-docker-container-from-outside



docker ps     # Get container id
docker exec -it container_id bash  #Connect to docker container using bash prompt

Inside of docker container bash prompt



psql -U postgres # connect to postgres

Inside of a successful postgres connection with psql in docker container



CREATE DATABASE test;  #Create database
\list  # verify database created successfully
\q  # Quit
Control+C. #Exit out of bash shell inside docker container

Test from outside docker container:



psql -h localhost -p 5432 -U postgres
\list  # see the database you created.

Troubleshooting

This section is mac os/linux specific. For windows, you will need to find the corresponding troubleshooting commands.

Port issues

IF port error - debug ports in use with this command



sudo lsof -i -P | grep LISTEN | grep :5432

If you find anything like postgres or anything else listening on the port, run



sudo kill process_id

where process id is the second column of what was returned from the above lsof command.

Check to see if other docker containers are running trying to use the same port



docker ps -a  # get list of all containers
docker stop container_id
docker rm container_id

4) Connect with python, node JS etc.

Now that you proved you can connect to your docker containerized postgres instance, you can use your normal approaches to connect to postgres not inside a docker container.

PostgreSQL command line psql cheat sheet

theBridge2 — Thu, 21 Mar 2024 00:10:41 +0000

Tools Overview

postgres = open source relational database software
psql = command line tool to connect to postgres
pgAdmin = web based tool front end to psql - not discussing here

This cheat sheet is for psql. Other 3rd party tools exist for connecting to postgres databases. (Dbeaver is a popular choice)

Postgres cmd line tool psql

Official docs
https://www.postgresql.org/docs/current/app-psql.html

Command	Description
psql --help	list help
psql --username=postgres	login to postgres cmd line

When logged in:

Command	Description
\list	print out list of databases
\c <db name>	connect to a certain database
\dt	show all tables in current schema
\df	list functions of current database
\dv	list views of current database
\du	list users and roles
\q	quit postgres cmd line
SELECT * from mytable;	select statement to table

NodeJS Backend tools - Node Version Manager (NVM)

theBridge2 — Thu, 07 Dec 2023 18:49:36 +0000

Here is a very basic page on NVM. If you don't know what NVM is and you use nodeJS, you should know!

NVM

Node version manager is very powerful allowing you to easily install and switch back and forth between different node versions.

Install nvm per instructions.
https://github.com/nvm-sh/nvm

(I used their curl command to download script and run)

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

Node install with nvm:

node -v. # is v20.10.0 at this point.
nvm install 16 #installs node v16 and switches to it.
node -v # now equals v16.20.2.

To switch:

nvm use 20 # (may need to install 20 with nvm for it to be able to use it)

Javascript datetime to and from postgres with time zone

theBridge2 — Wed, 01 Nov 2023 10:39:45 +0000

First of all, shout out to this blog that goes much deeper into this and is how I learned this concept:

https://blog.untrod.com/2016/08/actually-understanding-timezones-in-postgresql.html

Problem:

Inserting a datetime from js to postgres and then selecting that same time back out will likely end up with the wrong timezone on your select statement unless you know what you are doing. This happens even if you are inserting UTC and attempting to select UTC.

Specifically what happened to me and the blog post above is instead of getting UTC back, I got UTC + 2 x my UTC offset, which is very very wrong.

Bad solution:

Use AT TIME ZONE 'UTC' in select statements. Ex: SELECT log_time AT TIME ZONE 'UTC' as log_time_utc FROM logs

This is bad because you will have to do this everywhere you use log_time. Lets find a better solution that just works as expected.

Solution:

Use timestamptz as your database column definition (which is timezone aware, vs just timestamp which is not)
Put a timezone offset string at the end of your datetime value prior to inserting, otherwise postgres won't understand what is being inserted (even for UTC you must put a "+0")

Example:
In my javascript helper utilities before inserting a new date, which are created with local time, I have been converting to UTC time.

function makeDateTimeNowForDbUtc(inDate){

if( inDate === undefined ){
  var inDate = new Date;
}
var outDate = inDate.getUTCFullYear() + '-' +
('00' + (inDate.getUTCMonth()+1)).slice(-2) + '-' +
('00' + (inDate.getUTCDate()).slice(-2) + ' ' +
('00' + (inDate.getHours()).slice(-2) ':' +
('00' + (inDate.getUTCMinutes()).slice(-2) + ':' + 
('00' + (inDate.getUTCSeconds()).slice(-2) + '.' +
('00' + (inDate.getUTCMilliseconds()).slice(-3);

outDate = outDate + "+0"; // addition for postgres
return outDate;

Now I don't have to worry about it and my dates go in as UTC and back out as UTC without any additional code!

NOTE: if you are getting confused by the time zone offset in the javascript date time use toUTCString to get straight UTC time from your date.

let dt = myPostgresSelectReturn();
console.log('UTC Value: ' + dt.toUTCString()); // Date time GMT where GMT = UTC

Hope this helps!

Python fundamentals (1 of 2): How to install and run python

theBridge2 — Fri, 27 Oct 2023 13:47:04 +0000

Installing python

To install python, go to the python downloads page: https://www.python.org/downloads/

To see python version installed, type python -VV in the command line.

To see which version of python you are running and where it is installed in linux:

ls -l $(which python)

Where do I run python?

Once python is installed, you can run python from several places.

Run python with IDLE

When you install python you get the Integrated Development and Learning Environment (IDLE) which is where most beginners start writing programs.

If you are going to write a program with any complexity however, I would recommend you use an IDE like vs code that allows you to set breakpoints and pause program execution. This is possible in IDLE, but it is fairly rudimentary.

Run python with VS Code

Download vs code from https://code.visualstudio.com/download

Add the python extension to VS Code, by clicking on the extensions side button:

and searching for python. The correct python package to install is this one:

Then go ahead and create a hello world python file:

print('hello world!')

Now hit 'F5' to run it in VS Code.

Select "Python File Debug the currently active python file" and it will run.

This will run in the terminal at the bottom of VS Code.

Note: if you use break points in vs code and want to see the value of variables or run python commands while the program is paused, go to the debug console at the bottom and start typing away! See below for an example:

Run python inside a shell

Python shell is created from your command line tool by typing in "python". You are rewarded with '>>>' to indicate you are properly in a python shell.

If this doesn't work for you, and you get an error like this:

then the problem is that you need to add this to your operating system's path variable.

See part 2 for how to figure out path variable issues. Part 2 - Python Fundamentals: Install packages and dealing with PATH variables

Basic Mac development setup

theBridge2 — Mon, 09 Oct 2023 11:13:00 +0000

All of my development has been on a windows machine to this point having only casually used my Mac for 10+ years.

Even with all that casual experience, I have been tripping up a lot. Here are the key things that got me going developing on a Mac.

Keyboard basics

Using a windows keyboard for now.

Action	Keys
Command button	Windows key
Home	command + left arrow
End	command + right arrow
move cursor one word at a time	Alt + Arrow
show applications/desktops	Control+Up
Screenshot entire desktop	Shift + Cmd + 3
Screenshot portion of screen	Shift + Cmd + 4 (+drag cross hair & make a box)
Copy	command + C
Cut	command + X
Paste	command + V

More on screenshots
https://support.apple.com/en-us/HT201361

Vscode install

https://code.visualstudio.com/docs/setup/mac

Vscode permissions

VS Code won't let you use the terminal until you do this:

OS Settings>Security and privacy>Privacy tab, Click checkbox "documents folder" under Vscode icon.

Command line basics

Command	Description
pwd	present working directory
ls	show directory contents
ls -a	show all files and directories including hidden ones.
ls -l	show ownership and permissions
find . -name 'mystring*'	find file name containing a string.
which <app name>	figure out if and where an application is installed. ex: Is brew installed? % which brew

Homebrew Mac/Linux Installer

"The Missing Package Manager for macOS (or Linux)"
https://brew.sh/

Command	Description
brew list	list all applications installed with homebrew
brew install	install an application

Environment Variables and path

Command	Description
env	See all environment variables
echo $PATH	See just the path variable

Add application temporarily to PATH variable, so it works right away (ex: postgres):

export PATH="$PATH:/Applications/Postgres.app/Contents/Versions/12/bin"

Add application permanently to PATH variable, so it works after restarting the terminal and stays:

Open your bash profile file at $HOME/.bash_profile with: $ vi ~/.bash_profile
At the end of the bash profile, add the line:

export PATH="$PATH:/Applications/Postgres.app/Contents/Versions/12/bin"

Hope some find this useful. Plenty more to add I'm sure.

Python fundamentals (2 of 2): How to install packages and deal with PATH variable.

theBridge2 — Fri, 22 Sep 2023 14:09:21 +0000

If you missed it, consider first checking out my first part Python fundamentals part 1 - installing and running

Package installation is a fairly straightforward process thanks to the Python's Preferred Installer Program or pip for short.

However, it seems most explanations assume you understand how to get the PATH variable which is required to get anything to run on the command line. Lets cover basic install first then the PATH variable.

Useful install related commands

Command	Purpose
pip install packageName	installs some package
pip uninstall packageName	uninstalls some package
pip list	lists packages installed
packageName --version	displays version of installed package
pip install --upgrade pip	upgrade the pip
pip show packageName	Prints useful list of info about package including where the package is installed on your computer

Installing a package

Let's start with the well known numpy library used for mathematical operations.

import numpy as np
print("test")

If this fails, you will get the error: "No module named 'numpy'"

The simple solution is to install the module. Every site everywhere explains this. The command is:

pip3 install numpy

Note that the '3' in pip3 is for python 3.

To prove you installed it type the command 'pip3 list' and you will see packages that are installed:

Installing a package to be used on command line: pytest

I'm going to be using the python package pytest since it is a harder example and will require an understanding of the PATH variable.

Here is the basic install command for python to be run in the command prompt:

pip3 install pytest

Then to confirm it works you can type:

pytest -h

If this is the first time you are doing this, more than likely this will fail. And you will get the same error trying to run it as you would if it wasn't installed at all.

Go back and take a look at text printed from the install messages and likely you will find something that looks like this:

which tells you something about a potential PATH issue.

What is the PATH variable?

Command line PATH variable

PATH is an environment variable. Environment variables are variables set outside of your code.
For python packages to be able to be used on the command line, it must know where they are. Generally python packages are installed in a scripts subdirectory next to your python.exe. See below for more details.

When you run the command line in any OS, it will not inherently understand what python or other commands not embedded in the OS are. In order to tell the command line what python is, you will need to add the executable file for python to command line tool's PATH variable.

Windows

echo %PATH%

Linux

echo ${PATH}

This prints all of the places the command line will look for commands.

When the command line looks for these command definitions it will look through the list of paths and stop when it is found. So if you have more than one version of python installed, it will find the first version installed and use that one. If you have more than one version of python installed and want explicit control over which one it uses, you should use a python virtual environment.

How do I fix my PATH issue?

To solve your PATH issue for your python command like pytest that needs to be run on the command line, follow these brief steps. For steps 1 & 2 for more detail I found this video to be useful: https://www.youtube.com/watch?v=3J96_vyfx8Y

Here is another good reference: https://realpython.com/add-python-to-path/

Steps:

1) Find your installed package location. Generally it is next to your python.exe location and then navigate into the subdirectory /scripts. Take a look at the PATH error in yellow above. Sometimes on windows it is installed to your roaming directory.

2) Copy and paste this pip path folder name to your environment variables both in the user and system variables section under PATH
3) Restart your command prompt to get the new system variables update
4) >pip uninstall pytest
5) >pip install pytest
6) pytest -h # should now work!

After fixing the path issue, now we get a wall of text in response to our pytest -h in the command line instead of our error.

I installed the package but it still isn't found!?

Type 'pip3 list -v' to see where the packages are installed.

To figure out where python is actually looking for imports use this code:

import sys
print(sys.path)

If the list does not include your package install path you can either reinstall it to the right location or modify your path with sys.path.append. Sys.path.append is not best practice, but it is simplest for a new python programmer to get started with.

import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))) "/../someFolder)
# where __file__ refers to the current file and can be replaced with a relative path

Best practice is to use some kind of virtual environment like venv or docker.

My next article will be on using a virtual environment to manage dependencies. Until then, you can use this for reference: https://realpython.com/python-virtual-environments-a-primer/

Carriage Return line feed: CRLF - Linux vs Windows

theBridge2 — Sat, 24 Jun 2023 11:55:20 +0000

CRLF? Carriage return (CR) and line feed (LF) come from the days of typewriters.

History

CR = '\r' = 0x0D (HEX) was used to move the cursor to the beginning of the line

LF = '\n' = 0x0A (HEX) was used to move the cursor down to the next line.

The combination of CRLF in type writers is what moved the cursor from the end of the current row to the start of the next row.

Windows vs Linux file compatibility

As computers developed linux moved away from using both CRLF and just used LF to indicate a new line inside a file while windows continued to use both.

If a file has only '\n' (LF) for its line endings then it is linux compatible.
If a file has '\r\n' (CRLF) for its line endings then it is windows compatible.

To convert from linux to windows use the sed command in a linux terminal:



sed -i 's/$/\r/' fileLinux.txt

To convert from windows to linux, use the following command in a linux terminal:



tr -d '\r' <fileWindows.txt > fileLinux.txt

credit: https://www.commandlinewizardry.com/post/removing-windows-line-breaks

How do I know which line ending a file has?

Notepad ++ does it:

VsCode will show you what the line endings are for a given file at the bottom right of the screen.