DEV Community: Tyrel Souza

6502 Code I wrote running on an actual NES.

Tyrel Souza — Wed, 01 Feb 2023 02:29:24 +0000

As I mentioned in my December post on my own blog, I'm doing a 6502 course on Pikuma.

I'm about 75% of the way done, and I think I need to circle back to some earlier stuff about how the PPU works, but it's super fun.

Over the holidays I was able to stop at my father's and pick up my old NES.
I swapped out the ZIF connector for a new one, and cleaned up some contacts on the RCA ports, and it works great!
Once I found out that it was working - I played Sesame Street ABC 123, as that's the only one I had up in my office - I ordered an EverDrive N8.
That came last week.

The pictures are tall due to how I took them, so sorry I'll attach them at the end of the post.

Once I got the EverDrive N8 I made sure it worked by playing a Battletoads ROM.
Battletoad tested - I then copied Atlantico.NES to my Everdrive.
Atlantico is the game that Gustavo is walking us through making in the current part of the course - not a real published game.
I loaded it up and HOLY COW - something I actually wrote in Assembly is running on real hardware.

If you want to watch the video, it's very simplistic at the 75% mark, this was before the Collisions chapter, and no sound yet.
I posted about it on my Fediverse server.

The feeling of getting something running, locally, and seeing it working on screen, despite being a programmer for ~~20 years, is AMAZING.
Writing code that executes on the system you grew up playing the early 90's, wow.

I do wish the CRT TV my wife had was square, things get cut off on it.
I even got a remote, so I could try to fix that in the menu, alas, only picture option is brightness.
(Not that I realistically thought I could scale it, CRT Pixels are only Pixels.

TurboC2 Setting Header and Include Locations

Tyrel Souza — Tue, 17 Jan 2023 05:00:00 +0000

This weekend I purchased a book from this seller on Craigslist - "Advanced MS-DOS Programming: The Microsoft Guide for Assembly Language and C Programmers" and before opening it, I wanted to get a C environment running.

I found a copy of TurboC2 on Archive.org and tossed that into my DOS Box install. I wrote a "Hello world" and pressed compile and it couldn't include "stdio.h", what the heck?

It seems that the Archive.org copy of Turbo C 2 ships with configuration that sets where the Includes and Lib directories to C:\TC. I keep all my programs in C:\PROGS so of course it can't find any header files for me!

To fix this you can either move your TurboC install to C:\TC, which feels wrong to me, or you could configure it in the options properly.

Steps

Go to the Directories entry in the Options Menu.
You can see the default provided configuration directories
Fill out your appropriate directories for all three of the options.
Make sure all three are configured properly.
Then you can save your config, so you only have to do this once.

The Screenshot Way

Go to the Directories entry in the Options Menu.

You can see the default provided configuration directories

Fill out your appropriate directories for all three of the options.

Make sure all three are configured properly.

Then you can save your config, so you only have to do this once.

Unfortunately - this file is a binary file. You can't just edit it in a text editor and carry on, so this is the only way I know how to change these locations.

Hopefully this helps anyone else who runs into any include errors with Borland Turbo C 2!

Dotfiles - My 2022 Way

Tyrel Souza — Tue, 10 Jan 2023 05:00:00 +0000

New Year's eve eve, my main portable computer crashed. Rebooting to Safe mode, I could mount this MacBook's hard drive long enough to SCP the files over the network to my server, but I had to start that over twice because it fell asleep. I don't have access to rsync in the "Network Recovery Mode" it seems - maybe I should look to see if next time I can install things, it's moot now.

I spent all January 1st evening working on learning how Nix works. Of course, I started with Nix on macOS (intel at least) so I had to also learn how nix-darwin works. I have my dotfiles set up to use Nix now, rather than an INSTALL.sh file that just sets a bunch of symlinks.

I played around for a litle bit with different structures, but what I ended up with by the end of the weekend was two bash scripts (still working on makefile, env vars are being funky) one for each operating system rebuild-macos.sh and rebuild-ubuntu.sh. For now I'm only Nixifying one macOS system and two Ubuntu boxes. Avoiding it on my work m1 Mac laptop, as I don't want to have to deal with managing synthetic.conf and mount points on a work managed computer. No idea how JAMF and Nix will fight.

My filetree currently looks like (trimmed out a host and a bunch of files in home/)

.
├── home
│   ├── bin/
│   ├── config/
│   ├── gitconfig
│   ├── gitignore
│   ├── gpg/
│   ├── hushlogin
│   └── ssh/
├── hosts/
│   ├── _common/
│   │   ├── fonts.nix
│   │   ├── home.nix
│   │   ├── programs.nix
│   │   └── xdg.nix
│   ├── ts-tl-mbp/
│   │   ├── brew.nix
│   │   ├── default.nix
│   │   ├── flake.lock
│   │   ├── flake.nix
│   │   ├── home-manager.nix
│   │   └── home.nix
│   └── x1carbon-ubuntu/
│   ├── default.nix
│   ├── flake.lock
│   ├── flake.nix
│   ├── home-manager.nix
│   └── home.nix
├── rebuild-macos.sh
└── rebuild-ubuntu.sh

Under hosts/ as you can see, I have a brew.nix file in my macbook pro's folder. This is how I install anything in homebrew. In my flake.nix for my macos folder I am using home-manager, nix-darwin, and nixpkgs. I provide this brew.nix to my darwinConfigurations and it will install anything I put in my brew nixfile.

I also have a _common directory in my hosts, this is things that are to be installed on EVERY machine. Things such as bat, wget, fzf, fish, etc. along with common symlinks and xdg-config links. My nvim and fish configs are installed and managed this way. Rather than need to maintain a neovim config for every different system, in the nix way, I can just manage it all in _common/programs.nix.

This is not "The Standard Way" to organize things, if you want more inspiration, I took a lot from my friend Andrey's Nixfiles. I was also chatting with him a bunch during this, so I was able to get three systems up and configured in a few days. After the first ubuntu box was configured, it was super easy to manage my others.

My home/ directory is where I store my config files. My ssh public keys, my gpg public keys, my ~/.<dotfiles> and my ~/.config/<files>. This doesn't really need any explaination, but as an added benefit is I also decided to LUA-ify my nvim configs the same weekend. But that's a story for another time.

I am at this time choosing not to do NixOS - and relying on Ubuntu for managing my OS. I peeked into Andrey's files, and I really don't want to have to manage a full system configuration, drivers, etc. with Nix. Maybe for the future - when my Lenovo X1 Carbon dies and I need to reinstall that though.

Advent of Code 2022 + End of Year Updates

Tyrel Souza — Fri, 16 Dec 2022 05:00:00 +0000

Advent of Code this year is kicking my butt so I haven't been doing any tech blogging really lately. If you want to follow my progress, I think I might be done as of day 15 - This one seems to be a traveling salesman/knapsack problem related. Here's my repo: https://gitea.tyrel.dev/tyrel/advent-of-code/src/branch/main/2022/python.

I'm not on the computer that runs it, but I've been spending a lot of time playing with Apple's System7 in the BasiliskII emulator. Might have some fun projects with that coming up, but wanted to do some more learning before I start anything. So I have been going through a course on 6052 Assembly programming for the NES, and I'm about 73% done with that, it's really great!

It's By Gustavo Pezzi at Pikuma, if "oldschool" programming floats your boat then I definitely recommend it. It's all programming through making roms with CC65/CA65 assembler, and using FCEUX to see your results, super neat.

I've been picking up some more Go work at work. My current team is sort of disbanding so I'm going to be moving away from doing just Python. It's been a year since I've done Go stuff, since I left Tidelift, so I'm really rusty.

Speaking of Rust, I was trying to do Advent of code in Rust also, and made it TWO whole days in Rust. It's still on my bucket of stuff to learn, but my free time seems to be running out lately, and I have a lot of things on my plate to get done.

Reminder, you can find me on The Fediverse as opposed to Twitter these days.

Neighbor's Water Heater Automation (part 1)

Tyrel Souza — Fri, 04 Nov 2022 04:00:00 +0000

The Setting

My neighbor has a Bosch tankless water heater he put in last year. This water heater has one slight problem that when the power even blips a single second, it gets set back to its lowest temperature of 95°F. My neighbor (we'll call him Frank for this post because Frank Tank is funny) Frank wants to set his heater to 120°F in his house. The problem arises in that his water heater is under the house in his crawl space.

Without an easy way to set his temperature, he needs to crawl under his crawl space and turn a dial EVERY. SINGLE. TIME.

He asked me if I knew of anything off the shelf that would help. I did not. So I said the only logical thing someone like me would have done. "I can totally automate that!"

The Lay Of The Land

He has a Bosch Tronic 6000C, with what appears to be a rotary encoder knob to set the temperature. I only spent a few minutes under his house while planning this and didn't think to any measuring of how many detents to rotate, or how long the dial took to rotate to 120°F, so my first pass of this project is done with estimations.

Project Time - Round 1!

I have a few random servos laying around, and an NodeMCU ESP8266 module. I figure these would be the perfect solution! ... note: was half right...

I found some code online by Kumar Aditya that is for the two items in my current parts list (ESP8266 and SG90)

The Original code runs a web server on port 80, and runs a web page with some jQuery (wow it's been a while) to change the angle of the servo. I realized this wasn't what I needed because my servos could only go 180° and I might need to go multiple rotations. I found a youtube video on how to make a SG90 run infinite in either direction, so I did those modifications. I then modified the front end code a little bit.

The new code on the back end was actually exactly the same, even though the effect was slightly different. It would run on port 80, listen at / and /angle, but the angle here was more of direction and speed (a vector?). The way the servo was built, 160° was "Stop", higher than that was rotate clockwise, lower was rotate counter clockwise.

I put three buttons on my page that would be "Lower" (150), "STOP" (160), and "Higher" (170). I then did some standard debouncing and disabling of buttons using setTimeout and such.

For a final touch I added in a range slider for "Time". This held how many seconds after pressing Higher or Lower, that I would send the STOP command again.

This seemed to work relatively well, but I figure I should just use a stepper motor if I was attempting to emulate one this way. I dug around in my closet and was able to find some parts.

Project Time - Round 2!

I was able to rummage up a 28BYJ-48 stepper with control board, and a HW-131 power module.

With these I needed a new library so I stripped the c++ code down to its basics, just getting me a server with the index page for the first pass.

On the Javascript side of things, I then decided I would add a temperature slider, from 90° to 120° (which writing this realize it should be from 95°... git commit...) with a confirmation button, and a small button to initialize down to 95°.

The initialize button would need to trigger an initialization where I rotate counter clockwise an appropriate amount of time (Length TBD) in order to force the rotary encoder dial to always start at a known state of 95. The green submit button sends the new desired temperature as a post.

Server side, I was using a library called AccelStepper. This I set some made up max speeds and steps per rotation, actual values TBD.

I added an endpoint called /setTemperature that takes in a temperature and sets a local temperature variable. From there, I calculate the temperature less 95, to find out how many degrees I need to increase by, for now I'm considering this rotations.

I then apply a multiplier (TBD also... there's a lot of these as you can see!) and call stepper.moveTo() and it actually feels like it's pretty accurate.

The endpoint /initialize runs stepper.moveTo with ten rotations CCW, and then resets the "known location" back to zero (this also runs on power on for now).

In Action

The result of this second round of coding is a lot more that I expect to happen once I can finally get down beneath his house. Frank will lose power, his water heater will reset to 95°F, the NodeMCU will reboot, and reinitialize itself. Frank will then open his browser to the NodeMCU's server, set the desired temperature, and take warm showers.

Version 2 will come once I actually test EVERYTHING. My first quesiton is if a rubber band on a lego tire with a servo wheel adaptor (which I 3d modeled and printed...) will work sufficiently. Programming wise, I need to figure out how many steps is one degree. Is the rotary encoder one degree per detent? Is it a constant speed? Is it like an alarm clock where you can sometimes jump by 10?

Stay tuned to find out the exciting conclusion once I can go down below Frank's house.

Code

The code is currently at https://gitea.tyrel.dev/tyrel/frank_tank.git

Office Meeting Sensor

Tyrel Souza — Fri, 04 Nov 2022 04:00:00 +0000

NOTES

This post is ported over from my wiki, so the format isn't as storytelling as a blog post could be, but I wanted it here.

Bill of Materials

Home Assistant Parts

Third Party Plugin Requirements

Zoom Plugin

I followed the Read Me from https://github.com/raman325/ha-zoom-automation#installation-single-account-monitoring and set up a Zoom Plugin for my account, that will detect if I am in a meeting or not.

Pi Zero

I have a tiny project Enclosure box that I dremeled a hole for the GPIO pins in the cover and I then sandwich the Blinkt onto the Pi Zero with another dremeled hole running to the micro usb power, and that's it for hardware.

For software, I installed the python packages for Pimoroni and Blinkt, which came with a lovely set of sample projects. I deleted everything except the mqtt.py file, which I then put my Mosquitto server settings.

I then added a new service in systemd to control the mqtt server


[Unit]Description=Meeting Indicator[Service]Type=simpleExecStart=/usr/bin/python2 /home/pi/mqtt.pyWorkingDirectory=/home/pi/Pimoroni/blinkt/examplesRestart=alwaysRestartSec=2[Install]WantedBy=sysinit.target

Pleased with the results, and testing by sending some messages over mqtt that changed the color, I then dove into Node-RED

Node-Red

This is my first project using Node-RED, so I'm sure I could optimize better, but I have two entry points, one is from running HomeAssistant app on my mac, which gets me sensor data for my webcam, and the other is the aforementioned Zoom Presence plugin I created. These are Events:State nodes.

When either of these are True, they call first my ceiling light to turn on, which next will then add a msg.payload of


rgb,0,255,0,0
rgb,1,255,0,0
rgb,2,255,0,0
rgb,3,255,0,0
rgb,4,255,0,0
rgb,5,255,0,0
rgb,6,255,0,0
rgb,7,255,0,0

as one string. This leads to a Split, which will in turn, emit a new MQTT message for each line (I split on \n) and turn on all 8 LEDs as red. This is inefficient because I am still using the sample code for the blinkt which requires you to address each LED individually, my next phase I will remove the pin requirement and just have it send a color for all of them at once, one line.

When either of the sensors states are False, I then flow into a Time Range node, in which I check if it's between 9-5 or not. If it is, then I turn all the LEDs Green, and if it's outside 9-5 I just turn the LEDs off. I do not turn OFF the overhead light, in case it was already on. I don't care about the state enough.

I also intentionally trigger at the Office Hours node, which will inherently turn the Green on at 9:01am, and off at 5:01pm. As well as turn on Red for any long standing meeting times I have.

Images

Videos

Source

Nodered configuration source json https://gist.github.com/tyrelsouza/c94329280848f0319d380cc750e995c2

Comparing Go GORM and SQLX

Tyrel Souza — Mon, 17 Oct 2022 04:00:00 +0000

Django ORM - My History

I'm not the best SQL developer, I know it's one of my weak points. My history is I did php/mysql from the early 2000s until college. In college I didn't really focus on the Database courses, the class selection didn't have many database course. The one Data Warehousing course I had available, I missed out on because I was in England doing a study abroad program that semester. My first job out of college was a Python/Django company - and that directed my next eight years of work.

Django, if you are unaware, is a MVC framework that ships with a really great ORM. You can do about 95% of your database queries automatically by using the ORM.


entry, created = Entry.objects.get\_or\_create(headline="blah blah blah")



q = Entry.objects.filter(headline\_\_startswith="What")
q = q.filter(pub\_date\_\_lte=datetime.date.today())
q = q.exclude(body\_text\_\_icontains="food")

Above are some samples from the DjangoDocs. But enough about Django.

My Requirements

Recently at my job I was given a little bit of leeway on a project. My team is sort of dissolving and merging in with another team who already does Go. My Go history is building a CLI tool for the two last years of my previous job.I had never directly interacted with a database from Go yet. I wanted to spin up a REST API (I chose Go+Gin for that based on forty five seconds of Googling) and talk to a database.

GORM

Being that I come from the Django (and a few years of ActiveRecord) land, I reached immediately for an ORM, I chose GORM. If you want to skip directly to the source, check out https://gitea.tyrel.dev/tyrel/go-webservice-gin. Full design disclosure: I followed a couple of blog posts in order to develop this, so it is in the form explictly decided upon by the logrocket blog post and may not be the most efficient way to organize the module.

In order to instantiate a model definition, it's pretty easy. What I did is make a new package called models and inside made a file for my Album.


type Album struct {IDstring`json:"id" gorm:"primary_key"`Titlestring`json:"title"`Artist string`json:"artist"`Pricefloat64 `json:"price"`}

This tracks with how I would do the same for any other kind of struct in Go, so this wasn't too difficult to do. What was kind of annoying was that I had to also make some structs for Creating the album and Updating the Album, this felt like duplicated effort that might have been better served with some composition.

I would have structured the controllers differently, but that may be a Gin thing and how it takes points to functions, vs pointers to receivers on a struct. Not specific to GORM. Each of the controller functions were bound to a gin.Context pointer, rather than receivers on an AlbumController struct.

The FindAlbum controller was simple:


func FindAlbum(c \*gin.Context) {var album models.Albumif err := models.DB.Where("id = ?", c.Param("id")).First(&album).Error; err != nil {c.JSON(http.StatusBadRequest, gin.H{"error": "Record not found!"})}c.JSON(http.StatusOK, gin.H{"data": album})}

Which will take in a /:id path parameter, and the GORM part of this is the third line there.


models.DB.Where("id = ?", c.Param("id")).First(&album).Error

To run a select, you chain a Where on the DB (which is the connection here) and it will build up your query. If you want to do joins, this is where you would chain .Joins etc... You then pass in your album variable to bind the result to the struct, and if there's no errors, you continue on with the bound variable. Error handling is standard Go logic, if err != nil etc and then pass that into your API of choice (Gin here) error handler.

This was really easy to set up, and if you want to get a slice back you just use DB.Find instead, and bind to a slice of those structs.


var albums []models.Albummodels.DB.Find(&albums)

SQLX

SQLX is a bit different, as it's not an ORM, it's extensions in Go to query with SQL, but still a good pattern for abstracting away your SQL to some dark corner of the app and not inline everywhere. For this I didn't follow someone's blog post — I had a grasp on how to use Gin pretty okay by now and essentially copied someone elses repo with my existing model.gin-sqlx-crud.

This one set up a bit wider of a structure, with deeper nested packages. Inside my internal folder there's controllers, forms, models/sql, and server. I'll only bother describing the models package here, as thats the SQLX part of it.

In the models/album.go file, there's your standard struct here, but this time its bound to db not json, I didn't look too deep yet but I presume that also forces the columns to set the json name.


type Album struct {IDint64`db:"id"`Titlestring`db:"title"`Artist string`db:"artist"`Pricefloat64 `db:"price"`}

An interface to make a service, and a receiver are made for applying the CreateAlbum form (in another package) which sets the form name and json name in it.


func (a \*Album) ApplyForm(form \*forms.CreateAlbum) {a.ID = \*form.IDa.Title = \*form.Titlea.Artist = \*form.Artista.Price = \*form.Price}

So there's the receiver action I wanted at least!

Nested inside the models/sql/album.go file and package, is all of the Receiver code for the service. I'll just comment the smallest one, as that gets my point across. Here is where the main part of GORM/SQLX differ - raw SQL shows up.


func (s \*AlbumService) GetAll() (\*[]models2.Album, error) {q := `SELECT * FROM albums;`var output []models2.Albumerr := s.conn.Select(&output, q)// Replace the SQL error with our own error type.if err == sql.ErrNoRows {return nil, models2.ErrNotFound} else if err != nil {return nil, err} else {return &output, nil}}

This will return a slice of Albums - but if you notice on the second line, you have to write your own queries. A little bit more in control of how things happen, with a SELECT * ... vs the gorm DB.Find style.

To me this feels more like using pymysql, in fact its a very similar process. (SEE NOTE BELOW) You use the service.connection.Get and pass in what you want the output bound to, the string query, and any parameters. This feels kind of backwards to me - I'd much rather have the order be: query, bound, parameters, but thats what they decided for their order.

Conclusion

Overall, both were pretty easy to set up for one model. Given the choice I would look at who the source code is written for. If you're someone who knows a lot of SQL, then SQLX is fine. If you like abstractions, and more of a "Code as Query" style, then GORM is probably the best of these two options.

I will point out that GORM does more than just "query and insert" there is migration, logging, locking, dry run mode, and more. If you want to have a full fledged system, that might be a little heavy, then GORM is the right choice.

SQLX is great if what you care about is marshalling, and a very quick integration into any existing codebase.

Repositories

Notes

I sent this blog post to my friend Andrey and he mentioned that I was incorrect with my comparision of sqlx to pymysql. To put it in a python metaphor, "sqlx is like using urllib3, gorm is like using something that generates a bunch of requests code for you. Using pymysql is like using tcp to do a REST request." Sqlx is more akin to SqlAlchemy core vs using SqlAlchemy orm. Sqlx is just some slight extensions over database/sql. As the sort of equivalent to pymysql in Go is database/sql/driver from the stdlib.

2016 Monitoring a CO2 tank in a Lab with a raspberry pi

Tyrel Souza — Thu, 02 Jun 2022 20:54:00 +0000

This was written in 2017, but I found a copy again, I wanted to post it again.

The Story

For a few months last year, I lived around the block from work. I would sometimes stop in on the weekends and pick up stuff I forgot at my desk. One day the power went out at my apartment and I figure I would check in at work and see if there were any problems. I messaged our Lab Safety Manager on slack to say "hey the power went out, and I am at the office. Is there anything you'd like me to check?". He said he hadn't even gotten the alarm email/pages yet, so if I would check out in the lab and send him a picture of the CO2 tanks to make sure that nothing with the power outage compromised those. Once I had procured access to the BL2 lab on my building badge, I made my way out back and took a lovely picture of the tanks, everything was fine.

The following week, in my one on one meeting with my manager, I mentioned what happened and she and I discussed the event. It clearly isn't sustainable sending someone in any time there was a power outage if we didn't need to, but the lab equipment doesn't have any monitoring ports.

Operation Lab Cam was born. I decided to put together a prototype of a Raspberry Pi 3 with a camera module and play around with getting a way to monitor the display on the tanks. After a few months of not touching the project, I dug into it in a downtime day again. The result is now we have an automated camera box that will take a picture once a minute and display it on an auto refreshing web page. There are many professional products out there that do exactly this, but I wanted something that has the ability to be upgraded in the future.

Summary of the Technical Details

Currently the entire process is managed by one bash script, which is a little clunky, but it's livable. The implementation of the script goes a little like:

Take a picture to a temporary location.
Add a graphical time stamp.
Copy that image to both the currently served image, and a timestamped filename backup.

The web page that serves the image is just a simple web page that shows the image, and refreshes once every thirty seconds.

The Gritty Technical Details

The program I'm using to take pictures is the raspistill program. If I had written my script to just call raspistill every time I wanted a picture taken, it would have potentially taken a lot longer to save the images. This happens because it needs to meter the image every time, which adds up. The solution is Signal mode and turning raspistill into a daemon. If you enable signal mode, any time you send a SIGUSR1 to the process, the backgrounded process will then take the image.

Instead if setting up a service with systemd, I have a small bash script. At the beginning, I run a ps aux and check if raspistill is running, if it's not, I start up a new instance of raspistill with the appropriate options and background it. The next time this script runs, it will detect that raspistill is running and be almost a no-op.

After this, I send a SIGUSR1 (kill -10) to take the picture which is then saved, un-timestamped. Next up I call imagemagick's convert on this image, I crop out the center (so I couldn't use raspistill's"-a 12" option) because all I care about is a 500x700 pixel region.

This is then copied to the image that is served by the web page, and also backed up in a directory that nginx will listen to.

Writing an EPUB parser. Part 1

Tyrel Souza — Wed, 01 Jun 2022 05:41:00 +0000

Parsing Epubs

Recently I've become frustrated with the experience of reading books on my Kindle Paperwhite. The swipe features, really bother me. I really like MoonReader on Android, but reading on my phone isn't always pleasing. This lead me to look into other hardware. I've been eyeing the BOOX company a while ago, but definitely considering some of their new offerings some time. Until the time I can afford the money to splurge on a new ebook reader, I've decided to start a new project, making my own ebook reader tools!

I'm starting with EPUBs, as this is one of the easiest to work with. At its core, an EPUB is a zip file with the .epub extension instead of .epub with many individual XHTML file chapters inside it. You can read more of how they're structured yourself over at FILEFORMAT.

The tool I've chosen for reading EPUBs is the Python library ebooklib. This seemed to be a nice lightweight library for reading EPUBs. I also used DearPyGUI for showing this to the screen, because I figured why not, I like GUI libraries.

My first task was to find an EPUB file, so I downloaded one from my calibre server. I convert all my ebook files to .epub and .mobi on my calibre server so I can access them anywhere I can read my OPDS feed. I chose Throne of Glass (abbreviating to TOG.epub for rest of post). Loading I launched Python, and ran


>>> from ebooklib import epub >>> print(book := epub.read\_epub("TOG.epub")

This returned me a <ebooklib.epub.EpubBook object...> , seeing I had an EpubBook I ran a dir(book) and found the properties available to me


['add\_author', 'add\_item', 'add\_metadata', 'add\_prefix',
 'bindings',
 'direction',
 'get\_item\_with\_href', 'get\_item\_with\_id', 'get\_items',
   'get\_items\_of\_media\_type', 'get\_items\_of\_type', 'get\_metadata',
   'get\_template', 'guide',
 'items',
 'language',
 'metadata',
 'namespaces',
 'pages', 'prefixes',
 'reset',
 'set\_cover', 'set\_direction', 'set\_identifier', 'set\_language', 'set\_template', 'set\_title', 'set\_unique\_metadata', 'spine',
 'templates', 'title', 'toc',
 'uid',
 'version']

Of note, the get_item_with_X entries caught my eye, as well as spine. For my file, book.spine looks like it gave me a bunch of tuples of ID and a "yes" string of which I had no Idea what was. I then noticed I had a toc property, assuming that was a Table of Contents, I printed that out and saw a bunch of epub.Link objects. This looks like something I could use.

I will note, at this time I was thinking that this wasn't the direction I wanted to take this project. I really wanted to learn how to parse these things myself, unzip, parse XML, or HTML, etc., but I realized I needed to see someone else's work to even know what is going on. With this "defeat for the evening" admitted, I figured hey, why not at least make SOMETHING, right?" I decided to carry on.

Seeing I was on at least some track, I opened up PyCharm and made a new Project. First I setup a class called Epub, made a couple of functions for setting things up and ended up with


class Epub:
    def \_\_init\_\_(self, book\_path: str) -> None:
        self.contents: ebooklib.epub.EpubBook = epub.read\_epub(book\_path)
        self.title: str = self.contents.title
        self.toc: List[ebooklib.epub.Link] = self.contents.toc

I then setup a parse_chapters file, where I loop through the TOC. Here I went to the definition of Link and saw I was able to get a href and a title, I decided my object for chapters would be a dictionary (I'll move to a DataClass later) with title and content. I remembered from earlier I had a get_item_by_href so I stored the itext from the TOC's href: self.contents.get_item_with_href(link.href).get_content(). This would later prove to be a bad decision when I opened "The Fold.epub" and realized that a TOC could have a tuple of Section and Link, not just Links. I ended up storing the item itself, and doing a double loop in the parse_chapters function to loop if it's a tuple.


def parse\_chapters(self) -> None:
    idx = 0
    for \_item in self.toc:
        if isinstance(\_item, tuple): # In case is section tuple(section, [link, ...])
            for link in \_item[1]:
                self.\_parse\_link(idx, link)
                idx += 1
        else:
            self.\_parse\_link(idx, \_item)
            idx += 1

_parse_link simply makes that dictionary of title and item I mentioned earlier, with a new index as I introduced buttons in the DearPyGUI at this time as well.


def \_parse\_link(self, idx, link) -> None:
    title = link.title
    self.chapters.append(dict(
        index=idx,
        title=title,
        item=self.contents.get\_item\_with\_href(link.href)
    ))

That's really all there is to make an MVP of an EPUB parser. You can use BeautifulSoup to parse the HTML from the get_body_contents() calls on items, to make more readable text if you want, but depending on your front end, the HTML may be what you want.

In my implementation my Epub class keeps track of the currently selected chapter, so this loads from all chapters and sets the current_text variable.


def load\_view(self) -> None:
    item = self.chapters[self.current\_index]['item']
    soup = BeautifulSoup(item.get\_body\_content(), "html.parser")
    text = [para.get\_text() for para in soup.find\_all("p")]
    self.current\_text = "\n".join(text)

I don't believe any of this code will be useful to anyone outside of my research for now, but it's my first step into writing an EPUB parser myself.

The DearPyGUI steps are out of scope of this blog post, but here is my final ebook Reader which is super inefficient!

I figure the Dedication page is not as copywrited as the rest of the book, so it's fair play showing that much. Sarah J Maas, if you have any issues, I can find another book for my screenshots.

Garage Door Opener

Tyrel Souza — Mon, 10 Jan 2022 03:46:00 +0000

I bought a house on October 9, 2020. This house has a garage door, and like any normal person of course I had to automate it.

One of the first things I did when I moved into my house was research some automation. I initially bought a half dozen ESP8266 devices and tried to figure out what I could do with them. I found Home Assistant and set that up on my server computer, along with ZoneMinder for security cameras.

NodeMCU ESP8266 module

I knew I would need some sort of relay (domain purchased from is gone) and reed switches to trigger the door and sense its position, so I purchased some from the internet. But my friend Paul said all I needed was a MOSFET so I bought one of those too. I tried to figure out how to trigger the door with a mosfet, but I was never able to. I won't document those failures.

Magnetic Reed Switch

Home Assistant has a plugin called ESPHome where you can write yaml files to configure an esp8266 module. This then builds a binary which you need to flash onto the esp8266 at least once via usb. From then on you can then on you can upload from a web form and drop the bin file in manually, or just press the UPLOAD button from ESPHome. I set my relay up on pin 19/D1 for the digital pin, and 16/GND,10/3v3 for the power. The Reed switch I tossed on 15/D7 and 11/GND but that could have been anywhere. See Schematic below. It still doesn't have an enclosure.

Relay in blue, and wires going to the NodeMCU

Schematic

With the relay triggering, I still had one problem - I'd trigger the door and it wouldn't do anything! Something else was a problem. The wiring for the garage door terminates in four screws, two of which are the door trigger. I tried poking it with a multimeter and having someone push the door button on the wall, but I was never successful that way, as any contact between the two poles would just open the door anyway.

After some unsuccessful thinking, I figured it was time to purchase an oscilloscope. I've always wanted one, in fact I bought an old Heathkit one once, but never got it working as I would have had to replace all the capacitors, and the insides is all perfboard - A NIGHTMARE.

I found this USB Logic Analyzer and Oscilloscope on amazon and figure I'd try it out. It came while my father was in town, so he was pretty interested in seeing it work as well. Of course I'm very fond of open source software so I downloaded PulseView and found out that the LHT00SU1 is a fx2lafw driver device. The interface to connect is really simple, you just look for your driver, and Scan for any usb devices that are using that driver and they show up to choose from.

I plugged the 1ACH and GND cables in and hooked them on the +/- wires where they attach to the door motor. Once you have a device connected, you then click Run on the top left, and trigger what ever mechanism (my garage door button) and see what happens.

I was very pleasantly surprised when I saw some movement on the A0 line!

Pulses of about 180ms and 140ms

I added some markers to figure out what I needed to imitate in ESPHome, and saw that it's about 150ms high with a 225ms low, then another 150ms high and then low again.

This looks like this in yaml:


switch:- platform: gpiopin: D1id: relay- platform: templatename: "Garage Remote"icon: "mdi:gate"turn\_on\_action:- switch.turn\_on: relay- delay: 150ms- switch.turn\_off: relay- delay: 225ms- switch.turn\_on: relay- delay: 150ms- switch.turn\_off: relay

I'm pretty sure I jumped and screamed with excitement when it opened!

Once the door was opening and closing, I was able to add more yaml to set another binary sensor to show whether it was open or closed (from the reed sensor):


binary\_sensor:- platform: gpiopin:number: D7inverted: truemode: INPUT\_PULLUPname: "Garage Door Closed"

All together this is shown on my Home Assistant Lovelace dashboard using two cards, one that shows a closed door, and one with an open door (both actual pictures of the door!) with a button to open it. Once it opens or closes the other card switches into place, Home Assistant at least at the time didn't have good conditional cards like I wanted.


type: conditionalconditions:- entity: binary\_sensor.garage\_door\_closedstate: 'on'card:type: picture-glancetitle: Garage (Closed)image: 'https://tyrel.dev/house/garage\_door.jpeg'entities:- entity: switch.garage\_remotehold\_action:action: none

Closed door state and button

Happy with the state of my Garage Door opening button, I can now yell at my phone to open the garage door (it's a "secure" switch so it requires the phone to to be open before OK Google will trigger the door).

There's a couple more pictures in my Instagram post about it.

I know I could have bought a device to do this myself, but this is fully mine, my code, and my experiment with learning how to automate things at home, I gained way more out of this project than I did if I just bought a MyQ or what ever is popular these days.

Bill of Materials

Notes

This is no longer in service, as I replaced the door and have a Chamberlain MyQ system now. Less fun, but at least it's serviceable.

Postmortem of a fun couple bugs

Tyrel Souza — Thu, 11 Nov 2021 19:55:00 +0000

Story at my previous job:

Tieg: Hey Tyrel, I can't run invoke sign 5555, can you help with this?

This is How my night started last night at 10pm. My coworker Tieg did some work on our CLI project and was trying to release the latest version. We use invoke to run our code signing and deployment scripts, so I thought it was just a quick "oh maybe I screwed up some python!" fix. It wasn't.

I spent from 10:30 until 1:30am this morning going through and looking into why Tieg wasn't able to sign the code. The first thing I did was re-run the build on CircleCI, which had the same error, so hey! at least it was reproducible. The problem was that in our Makefile scripts we run tidelift version > tidelift-cli.version and then upload that to our deployment directories, but this was failing for some reason. We let clients download this file to see what the latest version is and then our CLI tool has the ability to selfupdate (except on homebrew) to pull this latest version if you're outdated.

Once I knew what was failing, I was able to use CircleCI's ssh commands and log in, and see what happened, but I was getting some other errors. I was seeing some problems with dbus-launch so I promptly (mistakenly) yelled to the void on twitter about dubs-launch. Well would you know it, I may have mentioned before, but I work with Havoc Pennington.

Havoc Pennington: fortunately I wrote dbus-launch so may be able to tell you something, unfortunately it was like 15 years ago

Pumped about this new revelation, I started looking at our keychain dependency, because I thought the issue was there as that's the only thing that uses dbus on Linux. Then we decided (Havoc Pointed it out) that it was a red herring, and maybe the problem was elsewhere. I at least learned a bit about dbus and what it does, but not enough to really talk about it to any detail.

Would you know it, the problem was elsewhere. Tieg was running dtruss and saw that one time it was checking his /etc/hosts file when it was failing, and another time it was NOT, which was passing. Then pointed out a 50ms lookup to our download.tidelift.com host.

Tieg then found Issue 49517 this issue where someone mentions that Go 1.17.3 was failing them for net/http calls, but not the right way.

It turns out, that it wasn't the keyring stuff, it wasn't the technically the version calls that failed. What was happening is every command starts with a check to https://download.tidelift.com/cli/tidelift-cli.version which we then compare to the current running version, if it's different and outdated, we then say "you can run selfupdate!". What fails is that call to download.tidelift.com, because of compiling with go1.17.3 and a context canceled due to stream cleanup I guess?

Okay so we need to downgrade to Go 1.17.2 to fix this. Last night in my trying, I noticed that our CircleCI config was using circle/golang:1.16 as its docker image, which has been superseded by cimg/go:1.16.x style of images. But I ran into some problems with that while upgrading to cimg/go:1.17.x. The problem was due to the image having different permissions, so I couldn't write to the same directories that when Mike wrote our config.yml file, worked properly.

Tieg and I did a paired zoom chat and finished this up by cutting out all the testing/scanning stuff in our config files, and just getting down to the Build and Deploy steps. Found ANOTHER bug that Build seems to run as the circleci user, but Deploy was running as root. So in the build working_directory setting, using a ~/go/tidelift/cli path, worked. But when we restored the saved cache to Deploy, it still put it in /home/circle/go/tidelift/cli, but then the working_directory of ~/go/tidelift/cli was relative to /root/. What a nightmare!

All tildes expanded to /home/circleci/go/tidelift/cli set, Makefile hacks undone, (removing windows+darwin+arm64 builds from your scripts during testing makes things A LOT faster!) and PR Merged, we were ready to roll.

I merged the PR, we cut a new version of TideliftCLI 1.2.5, updated the changelog and signed sealed delivered a new version which uses Go 1.17.2, writes the proper tidelift-cli.version file in deployment steps, and we were ready to ROCK!

That was fun day. Now it's time to write some rspec tests.

Finished my GitHub CLI tool

Tyrel Souza — Fri, 05 Nov 2021 04:08:00 +0000

I never intended this to be a full fleshed CLI tool comparable to the likes of the real GitHub CLI. This was simply a way to refresh myself and have fun. I have accomplished this, and am now calling this "Feature Complete". You can play around with it yourself from the repository on gitlab.

HTTPX.LINKS

Today I learned some fun things with httpx mainly. The main thing I focused on today was figuring out pagination. The GitHub API uses the link header which I had never seen before.

The format of the header is a url, and then a relationship of that url. Lets take my friend Andrey's repos for example:

{'link': '<https://api.github.com/user/6292/repos?page=2>; rel="next", <https://api.github.com/user/6292/repos?page=7>; rel="last"', ...}

The link header there has two items split by a comma, each with two fields split by a semicolon, the first of which is a URL inside angle brackets... and AGHH this is going to be annoying to parse! Luckily httpx responses have this handled and client.get(...).links returns a lovely dictionary of the proper data.

With a response.links.get('next') check, you can get the url from response.links['next']['url']. So much nicer than writing some regular expressions.

TESTING

With that accomplished, I then added pytest-cov to my requirements.in and was able to leverage some coverage checks. I was about 30% with the latest changes (much higher than anticipated!) so I knew what I wanted to focus on next. The API seemed the easiest to test first again, so I changed around how I loaded my fixtures and made it pass in a name and open that file instead. In real code I would not have the function in both my test files, I would refactor it, but again, this is just a refresher, I'm lazy.

I decided earlier that I also wanted to catch HTTP 403 errors as I ran into a rate limit issue. Which, I assure you dear reader, was a thousand percent intentional so I would know what happens. Yeah, we'll go with that.

Py.Test has a context manager called pytest.raises and I was able to just with pytest.raises(httpx.HttpStatusError) and check that raise really easily.

The next bits of testing for the API were around the pagination, I faked two responses and needed to update my link header, checking the cases where there was NO link, was multiple pages, and with my shortcut return - in case the response was an object not a list. Pretty straight forward.

The GHub file tests were kind of annoying, I'm leveraging rich.table.Table so I haven't been able to find a nice "this will make a string for you" without just using rich's print function. I decided the easiest check was to see if the Table.Columns.Cells matched what I wanted, which felt a little off but it's fine.

The way I generated the table is by making a generator in a pretty ugly way and having a bunch of repo['column'], repo['column'] responses, rather than doing a dict comprehension and narrowing the keys down. If I ever come back to this, I MIGHT reassess that with a {k:v for k,v in repos if k in SELECTED_KEYS} and then yield a dictionary, but it's not worth the effort.

Overall I'd say this project was fun. It gave me a glimpse back into the Python world, and an excuse to write a couple blog posts. My next project is to get a Django site up and running again, so I can figure out how to debug my django-dbfilestorage.

Closing Thoughts

If I had to do this again, I would probably have tried some test driven development. I've tried in the past, but I don't work on a lot of greenfield projects. I tend to be the kind of engineer who jumps HEAD FIRST into code and then tests are an after thought.

I also kind of want to rewrite this in Go and Rust, two other languages I've been fond of lately, just to see how they'd compare in fun. I haven't done any API calls with Rust yet, only made a little Roguelike by following Herbert Wolverson's Hands-On-Rust book. The Tidelift CLI is all Go and a bazillion API calls (okay like ten) so that wouldn't be too hard to use like SPF13's Cobra CLI library and make a quick tool that way.

One fun thing I learned while moving things over to GitLab is that my user Tyrel is a super early adopter. I was in the first 36,000 people! I showed a screenshot of my user ID to my friend Sunanda at GitLab and we had fun finding that out.