DEV Community: David Newberry

React, Git[Hub], and VS Code for Beginners - Video 3

David Newberry — Thu, 23 Oct 2025 03:26:47 +0000

Video 2 (about 18 min):
https://youtu.be/e_2t2MNYDFQ

In this video, I start by installing VS Code.
I start VS Code, and then open up the React project directory created in the first video.
I spend a minute looking at the file hierarchy of the project, and then open up the source file for a component called Welcome, and use it to talk a little bit about how components work.

I look a little more at the files. I don't see a files called App.js or App.tsx, which is sometimes the root of the React app. In this case, I overlooked root.tsx, so I'll explore that in the next video.
I start the React development server by running:

npm run dev

In the browser, I then load the site by visiting the URL printed in the terminal; as I recall, localhost:5173.

Finally for this video, I go back to the source in VS Code and demonstrate how the React dev server takes care of essentially refreshing and "Hot [Re]loading" the content that gets changed, if you edit the source code.

In the next video installment, I plan to:

Create a new branch in git
Create a new React component
Discuss how state is handled in React
Merge the working branch into main*

(*) As it happens, I didn't have time in that stream! But next time, for sure.

React, Git[Hub], and VS Code for Beginners - Video 3

David Newberry — Thu, 23 Oct 2025 03:26:47 +0000

Previous video blog: Part 2

Video 3 (about 18 min):
(Stream 2)
https://youtu.be/e_2t2MNYDFQ

In this video, I start by installing VS Code.
I start VS Code, and then open up the React project directory created in the first video.
I spend a minute looking at the file hierarchy of the project, and then open up the source file for a component called Welcome, and use it to talk a little bit about how components work.

I look a little more at the files. I don't see a files called App.js or App.tsx, which is sometimes the root of the React app. In this case, I overlooked root.tsx, so I'll explore that in the next video.
I start the React development server by running:

npm run dev

In the browser, I then load the site by visiting the URL printed in the terminal; as I recall, localhost:5173.

Finally for this video, I go back to the source in VS Code and demonstrate how the React dev server takes care of essentially refreshing and "Hot [Re]loading" the content that gets changed, if you edit the source code.

In the next video installment, I plan to:

Create a new branch in git
Create a new React component
Discuss how state is handled in React
Merge the working branch into main*

(*) As it happens, I didn't have time in that stream! But next time, for sure.

React, Git[Hub], and VS Code for Beginners - Video 2

David Newberry — Thu, 23 Oct 2025 03:13:29 +0000

Previous video blog: Part 1

Video 2:
(Stream 1, Part 2)
https://youtu.be/Vwr2P7OWAWI

So now, with luck, you have a React Router app. Before diving into it, create a repository on GitHub.

Signing up for GitHub is pretty straightforward.
Once you have an account, you can create a new repository.

Most of the default settings are fine. For simplicity I named it the same thing as my local repository. I did not include anything in the initial repo, because all the files I want to upload will come from the local repo.
Next I tried to follow the instructs to push connect the local repository to the remote repo and push the changes, but I ran into a snag. I needed to authenticate git to GitHub.

The solution I eventually found was to install the GitHub CLI tool, which is a separate binary from git, called gh. There were a few options for installing gh. I chose to download the binary and then run it by cd-ing into the directory and running:
```
./gh auth login
```
The ./gh indicates that I want to run a binary in my current working directory.

Another good option would be to first install Homebrew, and then use it to install gh. On my primary development computer I have Homebrew installed. I don't have to avail myself of it often, but I probably end up using it a few times a year to install some binary that would be hard to install otherwise.

Once I'm authenticated with GitHub, I'm able to push my the project.

I turn next to VS Code, starting the download.

That's it for this video. Next time, I actually open VS Code!

Next video blog: Part 3

React, Git[Hub], and VS Code for Beginners - Video 1

David Newberry — Thu, 23 Oct 2025 03:10:45 +0000

React, GitHub, VS Code Video Blog

This series of blogs is an accompaniment to a weekly coding live-stream that I am doing. This series is focused on the three topics listed above, and is aimed at people who are just getting started. For that reason, I am working on a computer that has never been used for development before, and installing all the software I need along the way.

I also decided not to rehearse these videos. My hope is that there is value in seeing the messiness of the process, and what it looks like to work through questions and issues in real time. The downside is that sometimes I'll get stuck, or change my mind and backtrack. When I post videos to YouTube, I will edit them a bit for time by speeding up stretches where I'm not talking.

You can join the weekly live stream Wednesday afternoons (around 1 PM, Pacific) on Twitch, and find past weeks' videos on my YouTube:
https://www.twitch.tv/paxfeline
https://www.youtube.com/@GrokProgramming

In these blogs, I will list out the important steps and call out any mistakes I make during the stream.

Video 1:
(Stream 1, Part 1)
https://youtu.be/K13tgiff5mo

Preliminary: Get git. On Mac, one way to do this is install Xcode.

If you have no intention of doing app development, or any non-web development, you might want to skip Xcode and get git directly from http://git-scm.com/. One way to do this on Mac is to use Homebrew, which is another thing to install, but makes getting some other binaries easier.
In order to run the commands to create a new React app and start the React dev server, I install Node.js, which comes with command line tools npm and npx. To install Node.js, I first installed nvm -- "node version manager" -- by running:
```
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
```
(Per https://github.com/nvm-sh/nvm?tab=readme-ov-file)

When nvm finished installing, it tried to modify my shell .profile script, which is run whenever a new shell session starts. In particular nvm wants to add some code that will export an environmental variable called NVM_DIR, and run a script to load nvm.
I used the command line program pico to create .profile and paste in the code nvm told me to, then opened up a new terminal window. This started a new shell session, which executed .profile (thus exporting the variable and loading nvm).
Next, I installed Node.js by running nvm install node.
At that point I could run the command that creates a new React Router app.

First, in the terminal, navigate to the directory you want to create your new app directory in.

Beware my mistake! I told the script to create the project at ~/first-react-app. (To be fair, I'm pretty sure the script asked "where" I wanted it.) Generally, a path like that is interpreted as "in the home directory, an entry called first-react-app." However, in this case, in my current working directory (which was my home directory -- so the tilde-slash was redundant), it created a directory named "~" and placed "first-react-app" inside that.

Don't make my mistake. Navigate to where you want your project, and then just enter the name of the project you're creating.

The command to create a new React Router app is:
```
npx create-react-router@latest my-react-router-app
```
(Per https://reactrouter.com/start/framework/installation)

During the creation process, you will be asked to give your project a name, if you would like to initialize a git repository (you probably do), and a few other questions.
To start the React development server, and see your new app, run the following commands to move into the project directory, install necessary packages, and then start the server.
```
cd my-react-router-app
npm i
npm run dev
```

End of Video 1

Next video blog: Part 2

Tagged String Literals

David Newberry — Thu, 13 Feb 2025 18:42:45 +0000

I love seeing a line of code and thinking, I can not parse that. And then learning to parse it.

This is valid JavaScript:

This is an instance of using a tagged template string.

This works too:

These examples are from MDN. Learn more there. I have no use for this knowledge right now, but one day I may!

Hearsay, part 2: Props

David Newberry — Wed, 05 Feb 2025 04:25:20 +0000

My previous post on hearsay discusses how this problem came up for me.

In hearsay, I want to be able to pass data into components using the props attribute, and specifying a JavaScript object literal. Something like this:

<hear-say src="Comp.html" props="{ n: Math.random() }"></hear-say>

Let's start with the fundamental task: the attribute should be interpreted as JavaScript code. So I created a custom getter:

get props()
{    
    const prop_att = this.getAttribute("props")?.trim() || "{}";
    const prop_func = Function("self", `return ${prop_att};`);
    return prop_func(this);
}

This creates a new function with a single argument self, and the body of the function simply returns whatever the expression evaluates to. So every time this component accesses self.props.n, it will call Math.random() and return a new value.

This is good, but a straightforward setter turned out to be a problem.

set props(val)
{
    this.setAttribute("props", JSON.stringify(val));
}

When I tried updating the value of props, I ran into a problem. A line of code like self.props = { ...self.props } (never mind accomplishing anything useful) would result in the value changing from { n: Math.random() } to, e.g., { n: 0.42 }. The code would be overwritten with a value.

It occurred to me that I could create another property, call it propsData, that could hold data written to props. This way you can write whatever you want to props, and it won't overwrite the code there.

I changed the setter to store data in propsData. And it places the data into a props-data attribute to make it easy to see when inspecting the DOM.

set props(val)
{
    this.propsData = val;
    this.setAttribute("props-data", JSON.stringify(val));
}

It's a little weird, the behavior isn't immediately intuitive I'm afraid, but on balance I think it's a good approach. Now I just had to give myself more work.

I thought it would be nice if the user could update props (or propsData, really) by mutating it. The solution that came to mind was to create a special Proxy for props that also incorporates propsData.

What I want it to do is sort of synthesize props and propsData. When code references props.foo, I want it to check both props and propsData for foo. If it exists in both places, it will prefer the propsData version. If foo doesn't exist in propsData, it will return whatever props.foo holds (if anything).

So far so good. But what if props holds nested objects? Code that sets, e.g., props.foo.bar.baz = 42 will run into an issue. As it goes down the props chain (getting first foo and then bar) the corresponding chain under propsData is empty (there is no propsData.foo, much less propsData.foo.bar). To compensate, if the getter finds an object property in the props chain and no corresponding propsData property, it will fill it in with an empty object.

This is basically how the current version of the code works, with one more provision. When evaluating an expression like props.foo.bar = 42, foo will return an object wrapped in a Proxy, which basically holds two things: the value for foo inside props, and inside propsData. This is exactly what it does for the props chain, but the propsData chain needs to be a little different.

At first I had it construct an object with one field for the props chain and one for the propsData chain. The Proxy target for foo in props.foo, for example, would look like { props: {bar: 42}, propsData: {} }. But that leaves the propsData chain disconnected from propsData. Instead, the Proxy target changed: a prop property was added. Now instead of the propsData property holding the value of, e.g., foo.bar, it would hold the value of foo, and prop would be bar.

This setup allows a backwards chaining effect. The Proxy for bar would look like { props: 42, propsData: propsData.foo, prop: "bar" }. The Proxy's setter runs target.propsData[target.prop] = val. target.propsData will be propsData.foo, so the effect is the same as setting propsData.foo.bar = val.

I hope you found this post somewhat useful. I fear this explanation is not as clear as I had hoped. If you have any questions, please feel free to comment.

Hearsay: "Talkative Web Components"

David Newberry — Tue, 04 Feb 2025 02:25:39 +0000

A few years ago, I learned enough React to cobble together a project I had in mind (RiffTube). Some parts of it I liked, and others I didn't. I've always leaned more toward vanilla JavaScript, and been wary of frameworks.

Recently I began reworking RiffTube, and updating Node packages caused everything to break. I eventually got things working again, but the process reminded me of what I don't love about React.

The idea for a framework started forming in my mind. There were two core pieces:

Components, similar to React
Communication via broadcasting

It seems to me that React's strategy for handling state is both redundant and at adds with the way JavaScript does everything else. The state of a web app can be specified by listing out all the values of all the variables (such as, and including, all the objects and all their properties).

React adds another layer of variables, etc., on top of that, which is what seems redundant. And then, when it's "official" state gets updated, it re-renders the component, which seems to me at odds with the JavaScript ethos.

So I started working on hearsay.

In hearsay, there is no "special" state. So if you want a component to do something, like update a property/UI, you can broadcast a message. You can specify a recipient, so that the appropriate recipient can respond -- but every component can "hear" the message, and decide if they need to respond or not.

Components are implemented with a fairly lightweight custom element (MDN article). The custom element, <hear-say>, provides a few capabilities and behaviors.

src attribute: Specify the path to an HTML file.
- The contents of this file will be attached as the shadow DOM for the custom element.
- A <script> element can be used to call a special setup() function allowing you to:
  - Attach custom element callbacks.
  - Specify a special init() method.
  - Specify a special react() method to respond to messages that are broadcast.
slot() method:

A helper method for filling slots in the shadow DOM.
props attribute:

Inspired by React, I wanted to be able to pass data into a component. Using this attribute, you can specify an object literal with regular JavaScript code.

The props attribute ended up being more complicated to implement than I realized at first. In particular, if props started out with a value like { n: Math.random() }, then doing this.props = { ...this.props, foo: 42 } would overwrite the code expression with a value, like { n: 0.123, foo: 42 }.

So I wanted to come up with a way to make using props easy and intuitive. The solution I came up with involves (what I think is) a pretty neat use of Proxy. The next post will be all about that.

Here is an example of a very simple hearsay component. It includes code that calls broadcast(), and also a react() method that responds to the message.

Clicker.html:

<script>
    setup({
        init: self =>
        {
            self.num = 0;
            self.slot("slot", self.num);
        },
        react: (self, data, recipient) =>
        {
            if (data == "inc")
            {
                self.num++;
                self.slot("slot", self.num);
            }
        },
    });
</script>
<h1>Clicker</h1>
<div>
    You have clicked
    <slot></slot>
    times!
</div>

<button onclick="broadcast('inc')">
    Click
</button>

The HTML file that wants to use this component would include the hearsay.js script file, and use a <hear-say src="Clicker.html"></hear-say> element to display it.

<html>
    <head>
        <script src="hearsay.js"></script>
    </head>
    <body>
        <hear-say struct="Clicker.html"></hear-say>
    </body>
</html>

More information and examples:

Hearsay on GitHub

Also see my hearsay "build journal"

Talking to a Gmail POP3 server with Python

David Newberry — Sat, 07 Sep 2024 18:37:33 +0000

There is a short (under 6 minutes) video to go along with this blog.

POP is a relatively old protocol; used to retrieve email from an email server. The first version was specified in 1984. The version still in use today, POP3, was specified in 1996. To try it out, I went about connecting to a Gmail POP3 server.

The first step was looking up the POP3 settings -- what server to connect to, on what port. Google led me here, where I found the following information.

pop.gmail.com

Requires SSL: Yes

Port: 995

It mentions that SSL is required. This wasn't something I dealt with 25 years ago, when I was last messing around with POP. I was afraid it would be a headache, but it turned out to be no challenge whatsoever; with a little help from the Python docs, I arrived at this code.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())

It connects, and tells me what version of SSL is in use... or something. Great success! Time to start a conversation with the server.

Borrowing from the official RFC for POP3, here's an example POP3 conversation between a client and server/

C: <open connection>
S:    +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
C:    USER mrose
S:    +OK mrose is a real hoopy frood
C:    PASS secret
S:    +OK mrose's maildrop has 2 messages (320 octets)
C:    STAT
S:    +OK 2 320
C:    LIST
S:    +OK 2 messages (320 octets)
S:    1 120
S:    2 200
S:    .
C:    RETR 1
S:    +OK 120 octets
S:    <the POP3 server sends message 1>
S:    .
C:    QUIT
S:    +OK dewey POP3 server signing off (maildrop empty)
C:  <close connection>

The first thing that happens is that the server sends a greeting to the client. Friendly. So I'll add code to receive a message from the server.

When you ask to receive data from a socket, you have to specify a buffer size. The docs recommend a power of 2, such as 4096. Many responses from the server will come through all at once. Some won't; sometimes a message from the server will be broken across sever reads, and the buffer may not be filled to its maximum even if there is more to come.

In the case of POP3, the way to tell if a message has come in completely depends on which message is coming in. Most of the time the server is sending a single line of text. (As we will see again later, these have a carriage return and line feed characters at the end of each line.) Certain messages that could have a much longer response use another way to show they're done: a period on a single line by itself.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        data = s.read(4096)
        print(data)

Run again and, we get a greeting. Another great success! Notice that the line ends with "\r\n" -- carriage return and line feed characters.

You have to pass a buffer size to the read method. It will then have a buffer that size available to read data from the server -- but there's no guarantee of how much data will come into the buffer at a time. This means that a protocol needs some way of specifying when a message is complete. There are numerous strategies possible. POP uses two: for all messages, lines are ended with \r\n. For a short (one line) message, this is all that's required. For multi-line responses, a period on a line by itself indicates that the message is complete.

TLSv1.3
b'+OK Gpop ready for requests from 2601:1c0:8301:b590:f408:d66a:3029:16ad dq2mb54750689ivb\r\n'

Now we need to start talking back to the server. Time to create an I/O (or O/I) loop; get some user input and send it to the server. Oops! I can't send a string directly; that gives me a TypeError. I need to convert the message to bytes. The string encode() method will do that (the default encoding of utf-8 works fine).

Only, when I run it -- oops again! Nothing happens when my message is sent to the server. Because I forgot that messages coming from the client also need to end with \r\n. Another tiny tweak gives us:

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        while True:
            data = s.read(4096)
            print(data)
            msg = input() + "\r\n"
            s.send(msg.encode())

Great, now I can actually try to log in!

TLSv1.3
b'+OK Gpop ready for requests from 2601:1c0:8301:b590:f408:d66a:3029:16ad g4mb5147337iow\r\n'
USER grokprogramming
b'+OK send PASS\r\n'
PASS trustno1
b'-ERR [AUTH] Application-specific password required: https://support.google.com/accounts/answer/185833\r\n'

OK, so following that link takes me to a page where I can set up an application specific password. One potential stumbling block I came across: your account has to have two-factor authentication turned on in order for you to be able to make an application specific password, as far as I can tell. Why would I not have two-factor authentication turned on in the year of our Lorde 2024? I can't say. I do now.

Armed with an application specific password (mind you take out the spaces), I can log in! Then I'll issue the STAT command which will tell me how many messages I have, and their combined size. After that, I'll issue the LIST command, which will return a list of messages with an ID and a size for each one.

TLSv1.3
b'+OK Gpop ready for requests from 2601:1c0:8301:b590:f408:d66a:3029:16ad e18mb76868856iow\r\n'
USER grokprogramming
b'+OK send PASS\r\n'
PASS baygdsgkmihkckrb
b'+OK Welcome.\r\n'
STAT
b'+OK 263 14191565\r\n'
LIST
b'+OK 263 messages (14191565 bytes)\r\n1 2778\r\n2 2947\r\n3 6558\r\n4 9864\r\n5 35997\r\n6 45462\r\n7 45462\r\n8 63894\r\n9 11487\r\n10 74936\r\n11 74925\r\n12 11632\r\n13 32392\r\n14 74997\r\n15 51961\r\n16 15375\r\n17 46513\r\n18 21519\r\n19 15966\r\n20 27258\r\n21 28503\r\n22 35615\r\n23 86353\r\n24 280'

I've hit a bug in the code. The response for LIST spans multiple lines and, in this case, will require multiple buffer reads. The whole message will end with a period on a line by itself. Here I've received one buffer worth of the message, and now I'll have to hit return and send a blank message to the server in order for the code to advance to the next iteration of the loop and read from the buffer again.

I'll tweak the code so that the user always has the option of reading from the buffer again, or not. I'll also finally decode the incoming bytes from the server, so that the text renders nicer.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        while True:
            data = s.read(4096)
            print(data.decode())
            while input("more? y/[n]: ") == "y":
                data = s.read(4096)
                print(data.decode())
            msg = input("> ") + "\r\n"
            s.send(msg.encode())

And here's a full session including retrieving an email and sending the disconnect message.

> USER grokprogramming
+OK send PASS

more? y/[n]: 
> PASS trustno1
+OK Welcome.

more? y/[n]: 
> STAT
+OK 263 14191565

more? y/[n]: 
> LIST
+OK 263 messages (14191565 bytes)
1 2778
2 2947
3 6558
<...>
260 41300
261 114059
262 174321
263 39206
.

more? y/[n]: 
> RETR 1
+OK message follows
MIME-Version: 1.0
Received: by 10.76.81.230; Thu, 28 Jun 2012 20:21:50 -0700 (PDT)
Date: Thu, 28 Jun 2012 20:21:50 -0700
Message-ID: <CADBp03TWFOKcTOaK_0P7VV2GB+TZsoSd_W4G5nZKKs7pdk6cWQ@mail.gmail.com>
Subject: Customize Gmail with colors and themes
From: Gmail Team <mail-noreply@google.com>
To: Grok Programming <grokprogramming@gmail.com>
Content-Type: multipart/alternative; boundary=e0cb4e385592f8025004c393f2b4

--e0cb4e385592f8025004c393f2b4
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

To spice up your inbox with colors and themes, check out the Themes tab
under Settings.
       Customize Gmail =BB <https://mail.google.com/mail/#settings/themes>

Enjoy!

- The Gmail Team
[image: Themes thumbnails]

Please note that Themes are not available if you're using Internet Explorer
6.0. To take advantage of the latest Gmail features, please upgrade to a
fully supported
browser<http://support.google.com/mail/bin/answer.py?answer=3D6557&hl=3Den&=
utm_source=3Dwel-eml&utm_medium=3Deml&utm_campaign=3Den>
..

--e0cb4e385592f8025004c393f2b4
Content-Type: text/html; charset=ISO-8859-1

more? y/[n]: y

<html>
<font face="Arial, Helvetica, sans-serif">
<p>To spice up your inbox with colors and themes, check out the Themes tab
under Settings.</p>

<table cellpadding="0" cellspacing="0">
  <col style="width: 1px;"/>
  <col/>
  <col style="width: 1px;"/>
  <tr>
    <td></td>
    <td height="1px" style="background-color: #ddd"></td>
    <td></td>
  </tr>
  <tr>
    <td style="background-color: #ddd"></td>
    <td background="https://mail.google.com/mail/images/welcome-button-background.png"
        style="background-color: #ddd; background-repeat: repeat-x;
            padding: 10px; font-size: larger">
          <a href="https://mail.google.com/mail/#settings/themes"
            style="font-weight: bold; color: #000; text-decoration: none;
            display: block;">
      Customize Gmail &#187;</a>
    </td>
    <td style="ba
more? y/[n]: y
ckground-color: #ddd"></td>
  </tr>
 <tr>
    <td></td>
    <td height="1px" style="background-color: #ddd"></td>
    <td></td>
  </tr>
</table>

<p>Enjoy!</p>

<p>- The Gmail Team</p>

<img width="398" height="256" src="https://mail.google.com/mail/images/gmail_themes_2.png" alt="Themes thumbnails" />

<p><font size="-2" color="#999">Please note that Themes are not available if
you're using Internet Explorer 6.0. To take advantage of the latest Gmail
features, please
<a href="http://support.google.com/mail/bin/answer.py?answer=6557&hl=en&utm_source=wel-eml&utm_medium=eml&utm_campaign=en"><font color="#999">
upgrade to a fully supported browser</font></a>.</font></p>

</font>
</html>

--e0cb4e385592f8025004c393f2b4--
.

more? y/[n]: 
> QUIT
+OK Farewell.

more? y/[n]: 
>

Yet another great success! I was able to log in to the POP3 server and retrieve a message. The script in its current state is pretty flexible, but it requires a lot of work from the user. I'll make a few final tweaks to make interacting with the POP3 server a little easier: if the user starts a message to the server with a "!" it will be stripped out, but the script will read in data from the server until it gets to a period on a line by itself -- in other words, for commands with long responses. No "!" and the script will read in a single line, looking for the \r\n characters.

import socket
import ssl

hostname = 'pop.gmail.com'
context = ssl.create_default_context()

def read_until(s, eom):
    # read into the buffer at least once
    data = s.read(4096)
    # continue reading until end of message
    while data[-len(eom):] != eom:
        data += s.read(4096)
    # return incoming bytes decoded to a string
    return data.decode()

def read_single_line(s):
    return read_until(s, b"\r\n")

def read_muli_line(s):
    return read_until(s, b"\r\n.\r\n")

with socket.create_connection((hostname, 995)) as sock:
    with context.wrap_socket(sock, server_hostname=hostname) as s:
        print(s.version())
        print(read_single_line(s))
        msg = input("> ")
        # empty msg will close connection
        while msg != "":
            if msg[0] == "!":
                msg = msg[1:]
                long = True
            else:
                long = False
            msg += "\r\n"
            s.send(msg.encode())
            if long:
                print(read_muli_line(s))
            else:
                print(read_single_line(s))
            msg = input("> ")
        s.close()

Real-time plotting with pyplot

David Newberry — Tue, 03 Sep 2024 06:35:03 +0000

I wanted to graph some data that I was generating from a simple polling app. I have tinkered with pyplot in the past, but I haven't tried creating anything from scratch. Luckily, it's really popular, and there are tons of examples to be found on StackOverflow and elsewhere.

I did a search and started with this SO answer related to updating a graph over time.

import matplotlib.pyplot as plt
import numpy as np

# You probably won't need this if you're embedding things in a tkinter plot...
plt.ion()

x = np.linspace(0, 6*np.pi, 100)
y = np.sin(x)

fig = plt.figure()
ax = fig.add_subplot(111)
line1, = ax.plot(x, y, 'r-') # Returns a tuple of line objects, thus the comma

for phase in np.linspace(0, 10*np.pi, 500):
    line1.set_ydata(np.sin(x + phase))
    fig.canvas.draw()
    fig.canvas.flush_events()

This code animates a sine wave changing phase.

First two lines import the libraries I want to use: matplotlib.pyplot does the plotting and handling the GUI.

The ion() method, if I understand (though I may not), makes pyplot drive the GUI. You could also use it inside a tkinter program, or use it to generate static images, but in our case it makes sense to let it handle the GUI of the plot for us. (That's what the flush_events() call later on is doing: allowing interactivity with the figure window.)

This example uses the numpy method linspace() to create the x values. It returns a numpy array, which is a fancy Python list.

The reason to use np.sin instead of math.sin is broadcasting. That's the numpy term for applying the function to every item in a list. In fact, it occurs to me that the same thing could be achieved without numpy using map:

map(lambda n: math.sin(n), x)

But numpy broadcasting is convenient and simple to use.

Now comes the pyplot setup. First, create a new "figure" (fig). Into this figure, add a subplot (ax) -- there could be many. 111 has the rather esoteric interpretation, "create a 1x1 grid, and place this subplot in the first cell."

Into this subplot (or set of axes), a line is plotted using the x and y values passed. (Points are connected with straight lines and plotted continuously.) "r-" is the shorthand way of specifying a solid, red line. We could specify multiple lines, so plot() returns a tuple; the code above uses tuple unpacking to extract the one value we want.

It's a good start, but I need to extend the x-axis over time. This code also doesn't update the bounds of the y-axis if necessary -- it's locked into whatever bounds it calculates for the first plot. A little more searching lead me to this SO answer. To quote them:

You will need to update the axes' dataLim, then subsequently update the axes' viewLim based on the dataLim. The appropriate methods are axes.relim() and ax.autoscale_view() method.

Sure, sounds good. Based on their example, I created a demo graph that grows in both x and y.

import matplotlib.pyplot as plt
import numpy as np
from threading import Thread
from time import sleep

x = list(map(lambda x: x / 10, range(-100, 100)))
x_next_max = 100
y = np.sin(x)

# You probably won't need this if you're embedding things in a tkinter plot...
plt.ion()

fig = plt.figure()
ax = fig.add_subplot(111)
line1 = ax.plot(x, y, 'r-')[0] # Returns a tuple of line objects

growth = 0

while True:
    x.append(x_next_max / 10)
    x_next_max += 1
    line1.set_xdata(x)
    line1.set_ydata(np.sin(x) + np.sin(np.divide(x, 100)) + np.divide(x, 100))
    ax.relim()
    ax.autoscale()
    fig.canvas.draw()
    fig.canvas.flush_events()

    sleep(0.1)

Now I'm getting somewhere. But, this is a blocking loop, and I need my data to be occasionally updated. If I had multiple threads, I'd have to worry about being thread-safe with updating my variables. In this case, I can be lazy because I know that the variable only gets updated once every 5 minutes (or however often the polling function runs); there is no danger of the variable getting overwritten in the middle of a line of code.

import matplotlib.pyplot as plt
import numpy as np
from threading import Timer
from time import sleep

x = list(map(lambda x: x / 10, range(-100, 100)))
x_next_max = 100
y = np.sin(x)

# You probably won't need this if you're embedding things in a tkinter plot...
plt.ion()

fig = plt.figure()
ax = fig.add_subplot(111)
line1 = ax.plot(x, y, 'r-')[0] # Plot returns a tuple of line objects

growth = 0
new_x = None

dT = 1

def grow():
    global new_x, x_next_max
    while True:
        new_x = x + [x_next_max / 10]
        x_next_max += 1
        sleep(dT) # grow every dT seconds

t = Thread(target=grow)
t.start()

while True:

    if new_x:
        x = new_x
        new_x = None
        line1.set_xdata(x)
        line1.set_ydata(np.sin(x) + np.sin(np.divide(x, 100)) + np.divide(x, 100))
        ax.relim()
        ax.autoscale()
        fig.canvas.draw()

    fig.canvas.flush_events()

    sleep(0.1)

The graph is only updated when the grow thread assigns a value to new_x. Notice that the flush_events() call is outside the "if" statement, so that it is being called frequently.

Polling for profit

David Newberry — Thu, 22 Aug 2024 01:14:22 +0000

I recently started job where work appears on an online dashboard. I just want to know when a certain kind of job (i.e. one that pays a certain amount) becomes available.

This isn’t the kind of thing there’s an API for, so I decided to just make my script mimic the behavior of the browser. To send HTTP requests from a Python script, the requests module came recommended on StackOverflow, so that’s the way I went.

The developer tools of Firefox (I assume the other major browsers as well) allows you to inspect HTTP requests and responses, including their headers, in the Network pane.

It offers a few different choices, slightly different but all similar and not exactly what I needed. So I started with the cURL code and started constructing the requests code.

(Eventually I took out the “If-None-Match” header that cURL included, not because of any issues but just to simplify the code. A couple others I might have accidentally removed; it still works fine.)

In retrospect I probably went further than I need to, converting the Cookies header string into a Python dictionary, when it’s only going to be converted back into string form to be sent through the network.

But I did write a neat little function to turn the string into a dictionary automatically, and learned about Python’s partition function along the way. Good times.

import requests

cookies = "(cookie data copied from Firefox)"

### you could remove this ####
cookies = cookies.split("; ")

def l(c):
    p = c.partition("=")
    k = p[0]
    v = p[2]
    return (k, v)

cookies = dict(map(l, cookies))
### ^ this could be removed ^ ###

r = requests.get('https://somejob.site/projects',
                 headers={
                    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:128.0) Gecko/20100101 Firefox/128.0",
                    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8",
                    "Accept-Language": "en-US,en;q=0.5",
                    "Upgrade-Insecure-Requests": "1",
                    "Sec-Fetch-Dest": "document",
                    "Sec-Fetch-Mode": "navigate",
                    "Sec-Fetch-Site": "same-origin",
                    "Priority": "u=0, i"
                 },
                 cookies=cookies)

(These headers are probably unnecessary, but I just wanted to pretend to be the browser.)

When working on this, I used IDLE at first. It's a terrible text editor, but it makes it easy to make little changes and then run the code. You can also run the script from the terminal using the python executable:

python "polling script.py"

Once the request returned the source code for the webpage, it was time to tackle the HTML. First I just looked through the source to find out where the data was and for hooks that the program could use to get to it.

I found that all the information about jobs was stored inside a “data-react-props” attribute. This was used more than once in the code, but the tag I wanted to get at also had a “data-react-class” attribute with the value “workers/WorkerProjectsTable”.

This was all very brittle (they could easily break my script by changing the layout of the HTML code slightly), but it was a start. (As it happens it did break slightly, after working for some time. I fixed it, and it worked again. Then it went back to the first layout again, so it’ll be easy to revert or fix again.)

For the Python side of things, correctly parsing HTML is non-trivial. Rather than trying to get all the data I needed directly from the source code string, I opted to use a HTML parsing module to help.

HTMLParser fit the bill. Starting with an example, I stripped it down to just the code that is triggered every time a start tag is read in.

class MyHTMLParser(HTMLParser):
    def handle_starttag(self, tag, attrs):
        # handle tag

parser = MyHTMLParser()
parser.feed(r.text)

The code assumes that the data-react-class attribute will come before the data-react-props attribute. Importantly, these are both attributes of the same tag. A for-loop is used to iterate over all the attributes in the tag. A Boolean variable is set to True if the right data-react-class attribute is found, and if so then the subsequent data-react-props attribute is found.

found_tag = False
for key, val in attrs:
    if key=="data-react-class" and val=="workers/WorkerProjectsTable":
        found_tag = True
    elif found_tag and key=="data-react-props":
        d = json.loads(val)

The value of the data-react-props attribute is in JSON format. I think I started working with this by iterating through it looking for the data I wanted. This lead to some laughably convoluted code when I was looking back over it just now:

d = json.loads(val)
# oh no, did I really do this?
for k in d:
    if k == "dashboardMerchTargeting":
        v = d[k]

OK, let’s simplify away the loop and conditional and just access the data.

d = json.loads(val)
v = d["dashboardMerchTargeting"]

At this point, v is a Dictionary, and one of its keys is “projects.” Contained therein is a List of Dictionaries that each contain information about a project.

The code iterates over the list, skipping over items whose pay is less than a threshold number.

for p in pl:
    if p["pay"] < "$25/hr": continue # skip jobs under $25 an hour
        print(p) # dangerous, apprently

(Yes, I am using string comparison for this. It’s fragile, but it works as far as it needs to.)

The items can be printed out directly, as I did at first. In the process I ran into a problem with IDLE crashing. It turned out to be because one of the project titles had an emoji in it, and this caused IDLE to freeze up.

My solution was to use encode with errors set to “ignore” and then decode back to a string. This uses the default encoding of utf-8.

Since I wanted to change, omit, and add some fields, I chose to make a new object based on p (a Dict with project info). I could have almost as easily modified p and used it to store my information, but I did this instead.

{
    "name": p["name"].encode(errors='ignore').decode(),
    "coding?": str(p["isCoding"]),
    "pay": p["pay"],
    "added": datetime.datetime.now(),
}

So far I only use the name and added fields, but I included the other fields in case I decide I want them later.

So what do I actually do with the above object? Well, I put it in a Dictionary, using the project id as the key.

live_projects[p["id"]] = {
    # seen above
}

The live_projects is defined early on as an empty Dictionary. The request and parsing code is moved into a function to make it easy to call repeatedly.

live_projects = {}

def do_it():

    # all that code

    t = Timer(60.0 * 5, do_it) # every 5 minutes
    t.start()

do_it()

I added the live_projects Dictionary so that I could keep track of what projects were available, and not keep announcing the same one each time it polled.

To keep track of what projects were added or removed, I used the Set class. Above the loop, an empty set is creted.

current_projects = set()

Inside the loop that adds projects to the live_projects Dictionary, I added this line.

current_projects.add(p["id"])

This creates a set contains the id of each project it finds when polling that matches my criteria.

To compare this to the previous state (as contained in the live_projects Dictionary), at the top of the handle_starttag method I save the current set of keys as “lpks” (live_project keys).

lpks = set(live_projects.keys())

Now after going through the current list of projects, I can compare the sets. The live_projects Dictionary may be out of sync just after polling. To find projects that have been taken down, I get the difference between the current_set (generated from polling) and the keys of live_projects Dictionary (the lpks set).

Python allows you to take the difference of two sets (i.e. set a but without any elements from set b) using the minus operator. However, keep in mind that this is not regular subtraction.

not_yet_live = current_projects - lpks
still_live = current_projects - not_yet_live
no_longer_live = lpks - current_projects

(Algebraic substitution suggests that still_live should lpks, but it doesn’t hold for set difference.)

Finally, this code prints out the new state if there have been any changes. It also takes care of removing elements from the live_projects Dictionary that are no longer available.

if len(not_yet_live) > 0 or len(no_longer_live) > 0:
    ctime = datetime.datetime.now()
    print()
    print(ctime)

    if len(no_longer_live) > 0:
        for k in no_longer_live:
            print("- (live " + str((ctime - live_projects[k]["added"])) + ") " + live_projects[k]["name"])
            del live_projects[k]

    if len(still_live) > 0:
        for k in still_live:
            print("  " + live_projects[k]["name"])

    if len(not_yet_live) > 0:
        for k in not_yet_live:
            print("+ " + live_projects[k]["name"])

That covers just about everything, except for a few lines of debugging code. I was playing around with using bit masks as flags for different kinds of debugging messages.

Here is the complete code.

import requests
from html.parser import HTMLParser
import json
from threading import Timer
import datetime
import os

cookies = "your cookies"

cookies = cookies.split("; ")

#print(cookies)

def l(c):
    p = c.partition("=")
    k = p[0]
    v = p[2]
    return (k, v)

cookies = dict(map(l, cookies))

#print(cookies)

DEBUG_SETS = 2**0  # i.e. 00000001
DEBUG_ATTRS = 2**1 #      00000010

DEBUG = 0

live_projects = {}

def do_it():

    r = requests.get('https://fake-url.job.tech/projects',
                     headers={
                         "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:128.0) Gecko/20100101 Firefox/128.0",
                        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8",
                        "Accept-Language": "en-US,en;q=0.5",
                        "Upgrade-Insecure-Requests": "1",
                        "Sec-Fetch-Dest": "document",
                        "Sec-Fetch-Mode": "navigate",
                        "Sec-Fetch-Site": "same-origin",
                        "Priority": "u=0, i"
                     },
                     cookies=cookies)

    #print(r.text)

    class MyHTMLParser(HTMLParser):
        def handle_starttag(self, tag, attrs):
            global live_projects

            lpks = set(live_projects.keys())

            if DEBUG & DEBUG_ATTRS: print("Encountered a start tag:", tag)

            found_tag = False
            for key, val in attrs:

                if DEBUG & DEBUG_ATTRS: print(f"debug: {key}={val}")

                if key=="id" and val=="workers/WorkerProjectsTable-hybrid-root":
                    found_tag = True
                elif found_tag and key=="data-props":
                    d = json.loads(val)
                    v = d["dashboardMerchTargeting"]
                    pl = v["projects"]

                    current_projects = set()
                    for p in pl:
                        if p["pay"] < "$22/hr": continue


                        if p["id"] not in live_projects:
                            live_projects[p["id"]] = {
                                "name": p["name"].encode(errors='ignore').decode(),
                                "coding?": str(p["isCoding"]),
                                "pay": p["pay"],
                                "id": p["id"],
                                "added": datetime.datetime.now(),
                            }

                            current_projects.add(p["id"])

                    not_yet_live = current_projects - lpks
                    still_live = current_projects - not_yet_live
                    no_longer_live = lpks - current_projects

                    if DEBUG & DEBUG_SETS:
                        print(f"\ndebug current_projects (a): {current_projects}")
                        print(f"debug lpks (b): {lpks}")
                        print(f"debug not_yet_live (c = a - b): {not_yet_live}")
                        print(f"debug still_live (a - c): {still_live}")
                        print(f"debug no_longer_live (b - a): {no_longer_live}")

                    if len(not_yet_live) > 0 or len(no_longer_live) > 0:
                        ctime = datetime.datetime.now()
                        print()
                        print(ctime)

                        note = ""

                        if len(no_longer_live) > 0:
                            for k in no_longer_live:
                                print("- (live " + str((ctime - live_projects[k]["added"])) + ") " + live_projects[k]["name"])
                                del live_projects[k]

                        if len(still_live) > 0:
                            for k in still_live:
                                print("  " + live_projects[k]["name"])
                                note += "  " + live_projects[k]["name"]

                        if len(not_yet_live) > 0:
                            for k in not_yet_live:
                                print("+ " + live_projects[k]["name"])
                                note += "+ " + live_projects[k]["name"]

                    if len(not_yet_live) > 0:
                        os.system(f"osascript -e 'display notification \"{note}\" with title \"{ctime}\"'")


    parser = MyHTMLParser()
    parser.feed(r.text)

    t = Timer(60.0 * 5, do_it) # every 5 minutes
    t.start()

do_it()

The Annoying Closure

David Newberry — Thu, 08 Oct 2020 05:53:44 +0000

Today I came across some code whose behavior confused me at first. The code involved attaching onclick functions to HTML elements inside a loop. Something like this:

let divs = document.getElementsByTagName( "div" );

for ( var i = 0; i < divs.length; i++ )
{
    divs[i].onclick = function () { alert(i); }
}

There's a subtle problem with this code, which relates to closures. We'd expect to see the index of a div when we click on it; instead, no matter what div you click on, you see the total number of divs.

When the anonymous onclick function is created, it has access to the variables in parent scopes -- that's why we can access the variable i (even though it's not working the way we want). This creates a closure, which binds the variable i in the onclick function to the variable i in the outer scope (the global scope in this case, but it could just as easily be another function).

When creating a variable with var, its scope will normally be the function in which the variable was created (or else in the global scope). Modern JS also allows variables to be created with let, which behaves somewhat more complexly.

A variable created with let is bound to the nearest inner block scope -- this could be a function, an if-statement, a loop; pretty much anywhere curly braces are used. If you're familiar with C-type languages, this scoping behavior will feel very familiar.

So one solution is to simply change var to let. When the closures are created, they will bind not to a function-scoped variable, but to a block-scope variable. As best I can tell, a loop creates a new block-scope each time its body executes. In this context, the closures bind to the values we want.

let divs = document.getElementsByTagName( "div" );

for ( var i = 0; i < divs.length; i++ )
{
    divs[i].onclick = function () { alert(i); }
}

While I was researching this question, I came across the MDN article on Closures. It gets into this issue in a section called "Creating closures in loops: A common mistake."

If you're like me, you might be wondering what other solutions to the issue there are. The let keyword has only been with us for a few years. The MDN article lists a few other solutions. Two solutions introduce an intermediary function. In my simplified example, you could do the following:

let divs = document.getElementsByTagName( "div" );

for ( var i = 0; i < divs.length; i++ )
{
    (function () {
        var index = i;
        divs[i].onclick = function () { alert(index); }
    })();
}

Or, perhaps more readably, you could just give the loop-body-function a name, move it outside the loop, and call it, passing i as an argument.

The MDN article also mentions using forEach instead of a for-loop per se, which also works by creating an intermediary scope to be bound to.

I'll finish by mentioning two other approaches that the MDN article doesn't discuss.

1) You could add a property to the element the onclick function is attached to, and reference this property. For example:

let divs = document.getElementsByTagName( "div" );

for ( var i = 0; i < divs.length; i++ )
{
    divs[i].index = i;
    divs[i].onclick = function () { alert(this.index); }
}

2) You could use bind:

let divs = document.getElementsByTagName( "div" );

for ( var i = 0; i < divs.length; i++ )
{
    divs[i].onclick = (function (index) { alert(index); }).bind( divs[i], i );
}

In the above example, the element the onclick function is being attached to is passed to bind as the first parameter. This allows the function to access this if desired.

Rho - Unifying REC component state

David Newberry — Mon, 05 Oct 2020 03:56:33 +0000

When we last left off, we had a component called RhoExpressionComposer that stored information about what Rho function it represents as part of its state. This made it possible to prototype the expression composition interface, but unfortunately the information about the expression being composed was scattered and not accessible as a unified tree.

To fix this, we need to unify where the expression is stored to a single component. For prototyping, I used App as the root component which stored the expression in its state, and provided a setter callback that is provided to child components (either by passing it down as a prop, or the use of a context).

In order to allow the setter callback to update the correct part of the expression tree, we need to come up with a way to specify a kind of path up the tree. This path could simply be an array of integers that represent indexes. For example, the root REC component's path will be an empty array. This is something of a special case.

Consider what happens as we start with an empty expression and begin to build one. Let's build the expression say we want to create this simple expression: 1 + (2 + 3).

The root REC component will allow the user to select a Rho function. Say the user selects addition. The Rho function (addition) and the REC component's path get passed to the setter callback. It sees an empty path, and sets the root Rho function.

Now, the root REC component displays the addition function's composition interface. On the left and the right of the plus sign are REC components, which represent the first and second arguments that will be passed to the addition function. The code that displays this interface (the addition function's "render" method) tells its child REC components what their paths are -- in this case, [0] and [1].

Say the user then selects a "constant" for the right Rho function. The setter callback is executed, and again it's passed the desired Rho function, and the REC component's path -- in this case, [0].

The setter callback starts at the root Rho function (addition), and then sets its first argument (child 0, the index specified in the path) to the specified Rho function.

The value returned by the Rho constant function is simply its input value, so that any constant can be set. So, another call to the setter callback is made, with the path [0, 0], and the constant value 1. (This is handled by the Rho constant function's render method / composition interface. For this edge case the callback 'setFunctionFromPath' is slightly misnamed. This may be renamed setArgumentFromPath.)

Now on the right side of the root addition, the setter would be passed the path [1] and the Rho function addition.

The setter would then be called twice more: with path [1, 0], and with path [1, 1], but with the "constant" Rho function. Finally it would be called twice again: [1, 0, 0] and the constant value 2; [1, 1, 0] and the constant value 3.

One technical consideration I came across when writing the setter function was that each time any part of the expression tree got updated, the entire tree was re-created, in the sense that each node was cloned. Since each node is duplicated, the connections between them also need to be explicitly re-set.

Here is a snapshot of the code discussed above. Also see the render method of the Rho addition function in RhoFunctions.js for an example of how the path prop is modified and passed to child REC components.
App.js, RhoFunctions.js

In this way, the system can build up a unified expression tree. Next step will be to create a component that allows the creation of multiple lines of code, where each line is a Rho expression. Once that's done, it will make sense to add in a runtime; and start putting together the display where the turtle is moved about, and draws.