DEV Community: moogoo

Check Python Memory Usage

moogoo — Mon, 30 Jan 2023 08:52:33 +0000

Monitoring memory usage of a running Python program - GeeksforGeeks

Build-in libraries

tracemalloc — Trace memory allocations — Python 3.11.1 documentation

import tracemalloc

print(tracemalloc.start())
# ...
# my ugly code
# ...
print(tracemalloc.get_traced_memory())

# shows (current_usage, peak)

Tools

# need pip install memory_profiler
from memory_profiler import profile

@profile
def my_ugly_func():
  # ...
  # ...

# show memory usage line by line

Other

Tkinter initial use 41.9 MB

PostgreSQL notes

moogoo — Mon, 26 Dec 2022 00:01:46 +0000

Quick commands

# ALTER DATABASE name RENAME TO newname

# mysql: SHOW DATABASES
# pgres: \l
# mysql: SHOW TABLES
# pgres: \d
# mysql: USE mydbname
# pgres: \c  mydbname # \connect

# mysql: SHOW COLUMNS
# pgres: \d table

# mysql: DESCRIBE TABLE
# pgres: \d+ table ( \dt)

# exit: \q

SELECT version();
show data_directory;

HOW TO

export/import

dump command

pg_dump -U USERNAME DBNAME > dbexport.pgsql
# PGPASSWORD="mypassword" pg_dump -U myusername dbname 密碼 > output.sql
pg_dump -U postgres -f /tmp/dump.sql.gz --compress=5 --no-owner dbname

import command

psql -f backup.sql dbname dbuser
# import specific table
pg_dump -U xxx public.TABLE_NAME DATABASE_NAME > out.sql
pg_dump -U xxx -d DB_NAME -t TABLE_NAME > out.sql

export csv

psql -U user -d db_name -c "Copy (Select * From foo_table LIMIT 10) To STDOUT With CSV HEADER DELIMITER ',';" > foo_data.csv

Slow query log

# 1. find config file
psql -U postgres -c 'SHOW config_file'

# 2. edit conf
# default
#log_min_duration_statement = -1
log_min_duration_statement = 1000 # (log all queries executing more than 1 second)

possible log paths:

/var/lib/pgsql/PG_VERSION/data/log/
/var/log/postgresql/
/var/lib/postgresql/PG_VERSION/main/pg_log

restart postgres (not “service”)

su - postgres -c "PGDATA=$PGDATA {path/to/}pg_ctl -w restart"

create user

postgres=# CREATE DATABASE mydbname ;
postgres=# CREATE DATABASE mydbname  OWNER myusername ;
postgres=# GRANT ALL PRIVILEGES ON DATABASE mydbname to myusername ;
ALTER ROLE myusername WITH superuser;

Problems

sequences

duplicate key error... (after restore data)

SELECT setval('my_sequence_name', (SELECT max(id) FROM my_table));
; my_sequence_name default is my_table_id_seq

truncate & auto increment

TRUNCATE table_name RESTART IDENTITY;

Note abont JavaScript DOM objects

moogoo — Tue, 13 Dec 2022 04:25:17 +0000

DOM Element structure

+-------------+    +------+    +---------+    +-------------+
| EventTarget | <- | Node | <- | Element | <- | HTMLElement |
+-------------+    +------+    +---------+    +-------------+

The DOM Node interface is an abstract base class upon which many other DOM API objects are based, thus letting those object types to be used similarly and often interchangeably.

every kind of DOM node is represented by an interface based on Node

Element - Web APIs | MDN

JavaScript: Live vs. Static NodeLists

moogoo — Tue, 13 Dec 2022 04:24:06 +0000

There are two types of NodeList: live and static

live NodeList means that changes in the DOM automatically update the collection.

Node.childNodes is a live NodeList
document.querySelectorAll() returns static NodeList Node

Sample Code:

HTML

<div id="parent">
  <div class="child">1</div>
  <div class="child">2</div>
  <div class="child">3</div>
</div>

JavaScript

const liveParent = document.querySelector('#parent');
const staticChildren = document.querySelectorAll('.child');
console.log(liveParent.childNodes, staticChildren);
liveParent.appendChild(document.createElement('div'));
console.log(liveParent.childNodes, staticChildren);

Results:

You can see that the length of staticChildren still the same after the parent object append a child.

Tkinter & Tk resources

moogoo — Thu, 22 Sep 2022 00:48:37 +0000

Documentation

Tk Commands
TkDocs Home
Tkinter 8.5 reference: a GUI for Python
Welcome to this Tk tutorial! — Tk tutorial 2020 documentation Tkinter Hello, World! many examples/snippets

Default Widgets

Tkinter Dialogs — Python 3.10.7 documentation
- tkinter.messagebox - Tkinter message prompts - Python 3.10.5 documentation

Canvas

no border

highlightthickness=0, relief='ridge', bd=0

Image

foo = PhotoImage # foo may clear by garbage collection, can assign to class to prevent
button.image = foo

Quick references

Widgets useful methods

The following are some of the most useful methods: 

- winfo_class: a class identifying the type of widget, e.g. TButton for a themed button
- winfo_children: a list of widgets that are the direct children of a widget in the hierarchy
- winfo_parent: parent of the widget in the hierarchy
- winfo_toplevel: the toplevel window containing this widget 
- winfo_width, winfo_height: current width and height of the widget; not accurate until appears onscreen 
- winfo_reqwidth, winfo_reqheight: the width and height the widget requests of the geometry manager (more on this shortly) 
- winfo_x, winfo_y: the position of the top-left corner of the widget relative to its parent 
- winfo_rootx, winfo_rooty: the position of the top-left corner of the widget relative to the entire screen the position of the top-left corner of the widget relative to its parent 
- winfo_rootx, winfo_rooty: the position of the top-left corner of the widget relative to the entire screen 
- winfo_vieweable: whether the widget is displayed or hidden (all its ancestors in the hierarchy must be viewable for it to be viewable)

screen size

screen_width = win.winfo_screenwidth()
screen_height = win.winfo_screenheight()
#Print the screen size
print("Screen width:", screen_width)
print("Screen height:", screen_height)

def print_hierarchy(w, depth=0):
    print('  '*depth + w.winfo_class() + ' w=' + str(w.winfo_width()) + ' h=' + str(w.winfo_height()) + ' x=' + str(w.winfo_x()) + ' y=' + str(w.winfo_y()))
    for i in w.winfo_children():
        print_hierarchy(i, depth+1)
print_hierarchy(root)

Grid management

4.2. Other grid management methods

grid_forget & grid_remove

python - Is their a grid_remember()? Reversible grid_forget()? - Stack Overflow

grid_slaves()
w.grid_remove()
destroy()
grid_forget
w.grid()

# destroy
for widget in frame.winfo_children():
    widget.destroy()

life circle ?

update(), update_idletastks()

python - What's the difference between "update" and "update_idletasks"? - Stack Overflow

The important thing to remember is that update blocks until all events are processed. In effect, that means you have a mainloop nested inside a mainloop. It's never a good idea to have an infinite loop inside an infinite loop.

Event

Modifiers

Control
Mod1, M1, Command
Alt
Mod2, M2, Option
Shift
Mod3, M3
Lock
Mod4, M4
Extended
Mod5, M5
Button1, B1
Meta, M
Button2, B2
Double
Button3, B3
Triple
Button4, B4
Quadruple
Button5, B5

Event types

Activate
Destroy
Map
ButtonPress, Button
Enter
MapRequest
ButtonRelease
Expose
Motion
Circulate
FocusIn
MouseWheel
CirculateRequest
FocusOut
Property
Colormap
Gravity
Reparent
Configure
KeyPress, Key
ResizeRequest
ConfigureRequest
KeyRelease
Unmap
Create
Leave
Visibility
Deactivate

Some Examples

label.bind('<Button-1>', left_mouse_down) 
label.bind('<ButtonRelease-1>', left_mouse_up)
label.bind('<Button-3>', right_mouse_down)
label.bind('<ButtonRelease-3>', right_mouse_up)
label.bind('<B1-Motion>', moving_mouse)
label.bind('<Enter>', moving_into)
label.bind('<Leave>', moving_out)
label.bind('<FocusIn>', focus)
label.bind('<FocusOut>', unfocus)
<Double-Button-1>
<MouseWheel>
<Button-4>
<Button-5>
<Control-Button-1>
<space> # lower case
frame.bind('<Control-Shift-S>', key)

54.1. Levels of binding

bind & bind_all
bind_all: a certain event calls a handler no matter what widget has the focus or is under the mouse

event_generate cannot send args (data)
Generate Event Passing Data : Tkinter

Styles

via: 5.6. Relief styles

# Create style Object
style = Style()

style.configure('TButton', font =
               ('calibri', 20, 'bold'),
                    borderwidth = '4')

# Changes will be reflected
# by the movement of mouse.
style.map('TButton', foreground = [('active', '! disabled', 'green')],
                     background = [('active', 'black')])

Packages/Widgets

Widgets

Video Player

huskeee/tkvideo: Python module for playing videos (without sound) inside tkinter Label widget using Pillow and imageio 用了 imageio/imageio: Python library for reading and writing image data
PaulleDemon/tkVideoPlayer: Video player for tkinter. 用 av (ffmpeg4) build 不成功

Table/Excel/DataGrid

Style/Theme

Books

Interesting

py_regular_expressions/interactive_exercises at master · learnbyexample/py_regular_expressions

Python queue.Queue

moogoo — Sat, 16 Jul 2022 00:59:18 +0000

Python has 3 ways to implement a "queue" (FIFO rule).

list
collections.deque
queue.Queue

list is build-in, multi purpose structure. If get item out from a list, need to shift all item, may slow. collections.deque: list-like container with fast appends and pops on either end. Here introduce more about Queue:

from queue import Queue

q = Queue(5) # 5 is maxsize of the queue [1]
q.put('foo') # put item into a queue
q.put('bar')

print(q.qsize()) # approximate [2] size of the queue
>>> 2
print(q.full()) # check full or not
>>> False
print(q.empty()) # check empty or not
>>> False
a = q.get()
print(a)
>>> foo
b = q.get()
print(b)
>>> bar
c = q.get() 
# blocking  ... queue is empty, wait until an item is available. 

get_nowait() # get(False), if not available, raise queue.Empty

# blocking process
task_done() & join()

[1]
If queue is full, no free slow, put() will block while available

[2]
qsize() > 0 doesn’t guarantee that a subsequent get() will not block, nor will qsize() < maxsize guarantee that put() will not block.

macOS: xcrun: error

moogoo — Wed, 13 Jul 2022 02:58:59 +0000

After upgrade macOS to 12.4 (Monterey), execute make command will get this error message:

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

fixed: udate Xcode toolkit

$ xcode-select --install

wait about 20 minutes, my make command works.

Rough notes about Google reCAPTCHA (v2 checkbox type)

moogoo — Sat, 02 Jul 2022 10:54:34 +0000

According to official docs:

There are 2 versions and four types of reCAPTCHA:

reCAPTCHA v3
- returning a score, giving you the ability to take action in the context of your site: for instance requiring additional factors of authentication, sending a post to moderation, or throttling bots that may be scraping content.
reCAPTCHA v2
- "I'm not a robot" Checkbox
- Invisible reCAPTCHA badge
- Android

I choose reCAPTCHA v2 Checkbox, seems most site use that type.

render the widget Two ways can do that:

1) Automatically render

<html>
  <head>
    <title>reCAPTCHA demo: Simple page</title>
    <script src="https://www.google.com/recaptcha/api.js" async defer></script>
  </head>
  <body>
    <form action="?" method="POST">
      <div class="g-recaptcha" data-sitekey="your_site_key"></div>
      <br/>
      <input type="submit" value="Submit">
    </form>
  </body>
</html>

2) Explicitly render

<script src="https://www.google.com/recaptcha/api.js?onload=onloadCallback&render=explicit"
    async defer>
</script>

more attributes

Server side verifying

3 way to get user's response token
1) g-recaptcha-response POST parameter when the user submits the form on your site
2) grecaptcha.getResponse(opt_widget_id) after the user completes the reCAPTCHA challenge
3) As a string argument to your callback function if data-callback is specified in either the g-recaptcha tag attribute or the callback parameter in the grecaptcha.render method

API Request
URL: https://www.google.com/recaptcha/api/siteverify METHOD: POST

payload: {
  secret: 'MY_SECRET_KEY', // Required. The shared key between your site and reCAPTCHA.
  response: 'token', // Required. The user response token provided by the reCAPTCHA client-side integration on your site.
  remoteip: 'MY_IP', llOptional. The user's IP address.
}

JSON Response

{
  "success": true|false,
  "challenge_ts": timestamp,  // timestamp of the challenge load (ISO format yyyy-MM-dd'T'HH:mm:ssZZ)
  "hostname": string,         // the hostname of the site where the reCAPTCHA was solved
  "error-codes": [...]        // optional
}

React Router 6 send data from link to target page

moogoo — Tue, 21 Jun 2022 10:57:46 +0000

use Link, useLocation from React-Router 6, send my filterList data from list to detail page.

After enter filter in list page, click on row and go to detail page, then we can go back to list page with filters we added.

import {
  Link as RouterLink,
} from "react-router-dom";
import { useLocation } from "react-router";

List page:

<Button
  variant="contained"
  color="primary"
  size="small"
  style={{ marginLeft: 16 }}
  component={RouterLink}
  state={{ filterList: state.filterList }}
  to= {`/collections/${params.row.id}`}
>
  Edit
</Button>

Detail page:

<Button
  variant="outlined"
  startIcon={<KeyboardBackspaceIcon />}
  to="/collections"
  state={{filterList: location.state.filterList}}
  component={RouterLink}
>
  back to list
</Button>

AWS command-line

moogoo — Tue, 21 Jun 2022 09:35:19 +0000

some drafts for aws command...

Configure Account

ref: Named profiles for the AWS CLI

check ~/.aws/config for your profile settings.

get credentials from AWS console, and add key and secret_key to ~/.aws/credentials

change account (profile reference to .aws/config)

$ export AWS_PROFILE=my-profile

S3

check bucket list from specific account

$ aws s3 ls # not list, only ls

dump bucket

$ aws s3 sync s3://my-bucket my-local-dir

pbcopy and pbpaste: Mac command-line copy and paste tools

moogoo — Mon, 20 Jun 2022 14:55:16 +0000

phcopy and pbpaste are super useful command-line tools for Copy and Paste (ctrl+c / ctrl+v).

# copy content from a text file
$ cat my-file.txt | pbcopy
# paste (output) from clipboard
$ pbpaste

Install Docker and docker-compose on AWS EC2 (AMI)

moogoo — Mon, 20 Jun 2022 05:29:05 +0000

Amazon Linux 2

Install Docker

# Update AMI yum packages
$ sudo yum update
# Install Docker by yum
$ sudo yum install docker
# add docker group to ec2-user
$ sudo usermod -a -G docker ec2-user
$ id ec2-user

Install docker-compose

$ wget https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) 
$ sudo mv docker-compose-$(uname -s)-$(uname -m) /usr/local/bin/docker-compose
$ sudo chmod -v +x /usr/local/bin/docker-compose

Configure docker service

# start service at boot
$ sudo systemctl enable docker.service
# start service now
$ sudo systemctl start docker.service

check service
'