DEV Community: Graham Trott

The Catch-22 of programming

Graham Trott — Sat, 29 Nov 2025 21:06:33 +0000

The Catch-22:

Only skilled programmers can guide AI to create good software.
It takes a long time to gain the skills to become a skilled programmer, starting with a long period of simple coding at the entry level.
AI is taking away the jobs of entry-level programmers.

Becoming a programmer

In the early days of computing, the route to becoming a programmer was straightforward. You just did it. Training courses were initially nonexistent, so we had to learn from textbooks and data sheets. The ones who had the motivation to stick with it were the ones who became professional programmers.

After that, in some ways it got a lot easier. Information became more plentiful, and with the arrival of first email then the World Wide Web, there were people you could ask and articles you could read without having to order a book and wait for it to arrive. Courses became available for pretty much anything.

In other ways, though, it didn’t get easier. Software became more complex, and a myriad of tools and libraries became regarded as essential things to know. Job listings started to fill up with acronyms and left many applicants feeling their efforts had all been for nothing, as the prize continued to remain just out of reach.

Programming with an LLM

Then along came AI, and suddenly it seemed a programmer's life became simpler again. No more need to remember every function in a library; just ask AI to find the one you need. But just as we started to get used to this new way of working, these upstart AI systems started to go for our jobs. And now we’re told that pretty much all programming will soon be done entirely by AI engines.

Or so it seems. However, if you actually ask AI to write software, you’ll soon discover that what comes back might be classifiable as a prototype but is likely to be totally unfit for delivery as a finished product. The only way to ensure a good product is to create a bullet-proof prompt to describe what you want, then closely monitor the entire process from start to finish, enforce good working practices and test everything as it gets written. The only person who can do that is a competent programmer.

So we have a Catch-22 situation. To get a good product you need to be an expert to drive AI in the right way. But because AI is taking over all the mundane programming work, few will ever spend the time and effort to become expert. Who in their right mind spends hours churning out boilerplate code when AI will spit it out in seconds? But if you don’t do it yourself you’ll never really learn how to, and you won’t be able to spot the mistakes made by AI. Likewise, you’ll never really know what any given programming language is capable of, and whether AI is making best use of it.

Or to put it more concisely: To get a job you need skills, but to get skills you need a job.

The crux of the matter

This is a situation that is only just developing but is set to get pretty serious quite quickly as the supply of top-level programmers dries up. The risk is that AI-generated products will frequently never develop past the prototype stage, while many will remain incomplete and generally unfit for purpose.

What are our options?

At the heart of the problem is the way we communicate with computers. There are basically 2 ways:

Using traditional high-level languages. This means code written in C++, Java, Python or any other mainstream programming language. As explained above, fewer people will become skilled at this work as AI takes over the lower levels of coding.
Using natural language, e.g. English. This is the new way that has come into its own with the arrival of AI. However, program code must be unambiguous and human language is usually anything but. This is the enemy of AI, as it will usually work with what it thinks you mean rather than question a possible ambiguity, so the result may be subtly or even grossly different to that which you wanted.

Software can be categorised in several ways, but one that I find useful is to distinguish between tools and products. Tools include operating systems and code libraries, and products are the things that are built with the tools. The thing that for me stands out like a lighthouse beacon is that coders use the same language for both, when in fact they often require very different forms of language. A computer algorithm expressed in English is clumsy and hard to follow, and a set of business rules that makes perfect sense in English becomes impenetrable when converted to Python or Java. Real-world requirements are often long chains of ‘if-then-else’ logic, with a frequent need for ‘goto’ to jump from one place to another, which is of course a complete no-no in the world of structured programming. Maybe what we need is a different form of language.

However much automation is provided, there will always be skilled toolmakers. Although it has limited storage capacity, the human brain is capable of insights no machine can match. So why waste such talent by asking such people to simply mind AI while it does all the hard work? Right now, the answer is that there is no separate class of coder with the ability to express clearly what is needed and guide the AI to create it, without having taken the long route to becoming a professional user of conventional languages and tools.

Some people are better than others at expressing clear logic. Many become lawyers, and they use a highly constrained form of English that most of us find difficult to understand, but which has that vital property of being unambiguous. Maybe we need something akin to this when we want to guide AI?

An example

To help clarify where I'm heading, here's an example. Let's suppose the task is to create a new app with a windowing user interface. The first thing is to decide where it will run, so I'll say on any desktop OS. This probably suggests Python should be the language to code for, and I'll constrain it a bit further by specifying PySide to handle the graphics.

The app will mimic a traditional web page, with a central data area, left and right side bars, a title bar and a footer. There's a menu bar with a single menu having 2 items, About and Exit, each with simple behaviours. It looks like this:

Here's an AI prompt to create the app as shown above:

Using PySide6, create a main window with the following attributes:
Location: 100,100
Size: 640,480
Title: My New App
The window contains 3 rows.
The first row is a panel taking 20% of the window height, containing a label with the text ‘Header’.
The second row has a height that is what is left after the other 2 rows are created. It has 3 elements: 1 - a panel taking 20% of the window width, with a label having the text ‘Left’. 2 - a panel taking whatever width is left after the other 2 items are created, with a label having the text ‘Center’. 3 - a panel taking 15% of the window width with a label having the text ‘Right’.
The third row is a panel taking 15% of the window height, containing a label with the text ‘Footer’.
The window has a menu bar with a single ‘File’menu having 2 items:
An ‘About’ item that when clicked shows a simple dialog.
An ‘Exit’ item that terminates the app when clicked.

This is a fairly clumsy way to program. It's largely unstructured, and when changes are needed whole sections may have to be rewritten. Most importantly, to avoid ambiguity it requires discipline in the choice of wording; the kind of discipline that only comes with experience. Since there’s no “compiler” to parse the text and spot problems with language or syntax, errors can easily slip through unnoticed.

So let’s see if we can use a simplified form of English to describe this window. The vocabulary I’ve chosen is arbitrary and many syntax variations can be accommodated to provde alternative ways of expressing the same thing, as in English. To be clear, when I refer to this as English I mean it's text that is mostly if not completely confined to English words, not mathematical symbols. And it’s terse; this is a recipe, not a poem. Let’s call this the pseudocode version.

! Variables used in the script
window Window
layout Content
panel Panel
label Header
panel Panel2
label Left
label Center
label Right
label Footer
menu Menu
menuitem Item

! Create a window
create Window
   at 100 100
   size 640 480
   title `My New App`

! Add the main content; a series of nested panels and labels
create in Window Content type rows

! The first row
create in Content Panel height 20%
create in Panel Header text `Header`

! The second row
create in Content Panel type columns
create in Panel Panel2
create in Panel2 Left width 20% text `Left`
create in Panel Panel2
create in Panel2 Center text `Center`
create in Panel Panel2
create in Panel2 Right width 15% text `Right`

! The third row
create in Content Panel height 15%
create in Panel Footer text `Footer`

! The menu
create in Window Menu text `File`
add to Menu Item text `About`
on click Item show dialog type info text `A sample application`
add to Menu Item text `Exit`
on click Item exit

show Window

This is 47 lines, compared to roughly 100 for the Python code that resulted from the AI query.

This version has a big advantage over the natural language version. It’s compilable to a form that will run directly from the command line. This means it can be used as a prototype product, rather than being just a specification of how to build the product. That’s demonstrated by the last line, show Window, which will display the window on a suitably-equipped system. Compilers and runtimes to handle this kind of script are not hard to write; in fact, something very close to this already exists in the Python world. See EasyCoder. (Disclaimer: This is my own product.)

How does this work with AI?

Can AI use pseudocode instead of natural language to generate a fully native application in whatever underlying language you prefer? To answer this question, I did the following:

First, I submitted the original natural-language request to DeepSeek, which generated Python code. The image above is a screenshot of the running code.

I then asked DeepSeek if it could recognise the pseudocode version. It came back with the confirmation that the two versions were in fact identical, and that if a suitable dictionary were built it could easily handle pseudocode instead of natural language. It described the approach as

a declarative UI specification system where:
Pseudocode = Human-readable UI description
Keyword dictionary = Mapping to actual UI components and properties
Generator = Interprets pseudocode → creates actual UI code

then went on to comment that

This is similar to how many modern UI frameworks work (React, Vue components) but with an even more abstracted, human-readable syntax.

and offered the following:

Benefits of This Approach:
Rapid Prototyping - Write UI structure in simple English-like syntax
Consistency - Enforced patterns through keyword definitions
Platform Independence - Same pseudocode could generate PySide6, Tkinter, web UI, etc.
Non-technical Friendly - Designers can specify layouts without coding
Version Control Friendly - Pseudocode is clean and diff-able

Compilable pseudocode reduces complexity

In the example above, many assumptions have been made. The way items are positioned and sized in their parent containers, the foreground and background colors, the fonts used for text, border widths and so on. In the browser world these are all handled by stylesheets, and in fact PySide handles most common stylesheet definitions. In any practical application, some form of stylesheet will have to be defined, holding global values and values that are specific to a class of on-screen widget. Doing all of this using natural language is painful to say the least, but a variant of the same pseudocode can be used to create JSON stylesheet files that can be read by the compiler. The language will need a vocabulary that allows the user to define which styles belong where, but this is not a hard task. DeepSeek offered to design such a system for me, but I’m not yet ready quite yet to dive further into this.

By scripting with compilable pseudocode, products can be developed quickly with or without AI, or by combining both. If people start to use pseudocode, AI will gradually be able to provide more assistance in creating applications by examining examples of those already created.

How does this change the way we work?

Unlike natural language, pseudocode scripts can be compiled on any computer and then run; hence the last line in the script shown above - show Window. The entire script can be run from the command line, directly from the source text. The performance may not be sparkling, but as a means of catching errors it's unbeatable as corrections can be made instantly and the run started again. Typical compilation times are in the tens of milliseconds, even for quite large scripts. The runtime even includes a visual debugger that allows breakpoints to be added and variables examined with the application halted.
The script uses a lot of variables to represent screen elements. Some of these are reused but others hold values that are of interest for the lifetime of the application. The natural language version might name such elements, but the syntax to refer to them will be clumsy and impede understanding. Having them available as code elements allows them to be accessed at any time simply by name.
Since performance is rarely an issue with prototypes, the compiler and runtime for these scripts can check very aggressively for potential coding errors during the development phase. Typical examples are the wrong type of variable being used, or an attempt to use a variable that has no value assigned to it. Once the prototype is seen to run without errors the script can be submitted to AI for recoding in Python, JavaScript, C++ or whatever is best for the target environment. This maximises performance.

New opportunities

This takes us back to the title of this article - the catch-22. Pseudocode allows very large chunks of functionality to be represented by a fairly concise English syntax. New functionality, created by experts in their fields, gets added as extra pseudocode vocabulary, which is easily understood by AI and by people familiar with the domain in which they are working. It is fairly easy to take a given piece of functionality and create a new language feature for it, and it does not usually require any great knowledge of how that functionality is coded. Take Google Maps, for example, which has a remarkably simple API for such a complex and comprehensive product. We will still need highly skilled programmers to create new core functionality, but pseudocode unlocks the door to making it accessible to those with more limited skills, to build the products that depend on this functionality. This could well be where most human programmers will be employed; managing AI efficiently to create the applications needed by the world.

And let’s not forget the problem of maintenance. Having advanced applications coded in pseudocode that can be understood both by AI and by humans makes future maintenance far simpler than requiring every maintenance engineer to know the details of every programming language and its libraries.

Summary

As AI takes over the jobs of those doing simpler forms of coding, fewer people will come up through the system and progress to the higher skill levels. The adoption of a suitable pseudocode for describing a wide range of applications will reduce the need for highly skilled coders and create a demand instead for logical thinkers who understand problem domains and can translate requirements into a form that can run as a prototype. Once this has been tested thoroughly it can then be handed to AI to generate the final code for the target environment.

OTA Python updates with ESP-Now

Graham Trott — Sat, 27 Sep 2025 14:33:42 +0000

This is the third article in a series about using ESP-Now to implement an ESP32-based master-slave network. Part one describes the main network comms class and part two describes how to follow the system router when it channel-hops. In this final article I'll describe an implementation of over-the-air (OTA) updates to the Python and other files held on the devices.

In part one of this series, when a message is received it is handed to a message handler, which carries out the needed action(s) and returns a result. The following is that message handler. Most of it is directly applicable to any similar application, since there are only so many things a network slave is able to do.

import asyncio,machine,os
from binascii import unhexlify
from files import readFile,writeFile,renameFile,deleteFile,createDirectory

class Handler():

    def __init__(self,config):
        self.config=config
        config.setHandler(self)
        self.relay=config.getRelay()
        self.saveError=False

    def handleMessage(self,msg):
#        print('Handler:',msg)
        bleValues=self.config.getBLEValues()
        response=f'OK {self.config.getUptime()} :{bleValues}'
        if msg=='uptime':
            pass
        elif msg == 'on':
            response=f'{response} ON' if self.relay.on() else 'No relay'
        elif msg == 'off':
            response=f'{response} OFF' if self.relay.off() else 'No relay'
        elif msg == 'relay':
            try:
                response=f'OK {self.relay.getState()}'
            except:
                response='No relay'
        elif msg=='reset':
            self.config.reset()
            response='OK'
        elif msg == 'ipaddr':
            response=f'OK {self.config.getIPAddr()}'
        elif msg=='channel':
            response=f'OK {self.config.getChannel()}'
        elif msg[0:8]=='channel=':
            channel=msg[8:]
            self.config.setChannel(channel)
            response=f'OK {channel}'
        elif msg[0:7]=='environ':
            environ=msg[8:]
            self.config.setEnviron(environ)
            response=f'OK {environ}'
        elif msg=='temp':
            response=f'OK {self.config.getTemperature()}'
        elif msg=='pause':
            self.config.pause()
            response=f'OK paused'
        elif msg=='resume':
            self.config.resume()
            response=f'OK resumed'
        elif msg[0:6]=='delete':
            file=msg[7:]
            response='OK' if deleteFile(file) else 'Fail'
        elif msg[0:4]=='part':
        # Format is part:{n},text:{text}
            part=None
            text=None
            items=msg.split(',')
            for item in items:
                item=item.split(':')
                label=item[0]
                value=item[1]
                if label=='part': part=int(value)
                elif label=='text': text=value
            if part!=None and text!=None:
                text=text.encode('utf-8')
                text=unhexlify(text)
                text=text.decode('utf-8')
                if part==0:
                    self.buffer=[]
                    self.pp=0
                    self.saveError=False
                else:
                    if self.saveError:
                        return 'Save error'
                    else:
                        if part==self.pp+1:
                            self.pp=part
                        else:
                            self.saveError=True
                            return f'Sequence error: {part} {self.pp+1}'
                self.buffer.append(text)
                response=f'{part} {str(len(text))}'
        elif msg[0:4]=='save':
            if len(self.buffer[0])>0:
                fname=msg[5:]
                tname=f't-{fname}'
                print(f'Save {tname}')
                size=0
                f = open(tname,'w')
                for n in range(0, len(self.buffer)):
                    f.write(self.buffer[n])
                    size+= len(self.buffer[n])
                f.close()
                # Check the file against the buffer
                if self.checkFile(self.buffer, tname):
                    response=str(size)
                    print('File saved')
                else: response='Bad save'
            else: response='No data'
            text=None
        elif msg=='update':
            for tname in os.listdir('.'):
                if tname[0:2]=='t-':
                    fname=tname[2:]
                    print(f'Update {fname}')
                    deleteFile(fname)
                    renameFile(tname,fname)
            response='OK'
        elif msg[0:5]=='mkdir':
            path=msg[6:]
            print(f'mkdir {path}')
            response='OK' if createDirectory(path) else 'Fail'
        else: response=f'Unknown message: {msg}'
#        print('Handler response:',response)
        return response

    def checkFile(self, buf, file):
        try:
            with open(file, 'r') as f:
                i = 0  # index in lst
                pos = 0  # position in current list item
                while True:
                    c = f.read(1)
                    if not c:
                        # End of file: check if we've also finished the list
                        while i < len(buf) and pos == len(buf[i]):
                            i += 1
                            pos = 0
                        return i == len(buf)
                    if i >= len(buf) or pos >= len(buf[i]) or c != buf[i][pos]:
                        return False
                    pos += 1
                    if pos == len(buf[i]):
                        i += 1
                        pos = 0
        except OSError:
            return False

`init()`

The system makes extensive use of a Config class, which holds all the system-specific configuration data such as which I/O pins do what. It also acts as a central message interchange, with functions that either return an object that has the required functionality or call the objects themselves, avoiding the need for every class to know about all the others. So in this Handler class, the relay object supplied by Config is a class that has the ability to turn the relay on and off and to return the current relay state.

`handleMessage()`

This is where most of the work is done. The function handles a set of commands, most of which should be fairly obvious, though some will be more relevant than others. For example, elsewhere in the code there's a Bluetooth (BLE) module that collects occasional transmissions from specific devices and for some message types appends these values to the string that is returned to the caller. In my system these are temperature values, but not everyone will want this feature.

The most interesting part of this is the OTA file updater, which works like this. The maximum size of an ESP-Now data packet is something over 200 bytes, so few Python scripts can be transferred in a single message. I chose to send 100 bytes at a time, with the text data encoded as hex values. The file is sent as a series of messages, each containing 100 bytes, and the messages are prefixed by the command part and a part number starting with zero. The function splits up the packet to extract the part number and the data itself. If the part number matches what was expected, the text is added to a list, otherwise the function returns an error. For a successfully received part, the function returns the part number and the size of the data, which the sender can use to verify that it has been received successfully.

When all the parts have been sent, the list can be saved to a temporary file. This is done by the save command.

`checkFile(()`

Now we need to check the data has not been corrupted. This function reads the file back, one character at a time, and compares it with the contents of the list. If no errors are found, the new file can be assumed to be good.

However, a problem comes when several Python files all need to be updated together. If only some of the files are updated and then an error occurs or for some other reason the update stops, there could be a mismatch between the various parts and the program will not run after the next reboot. So rather than update files one by one they should instead be done as a batch. Only when all of the files have been saved and checked is it then safe to rename the entire batch. This is done by the update command.

In a practical implementation, when an error is discovered and reported back to the sender, the entire save can be retried. Although for the most part OTA updates proceed without a hitch, in some cases, particularly where the signal strength isn't great or suffers from interference, it can take several attempts to transfer a large file. I allow up to 10 retries before abandoning the save.

Channel-hopping on the ESP32-C3 with ESP-Now

Graham Trott — Sat, 27 Sep 2025 14:30:21 +0000

This is the second article in a series about using ESP-Now in Micropython to implement an ESP32-based master-slave network. Part one describes the main network comms class and part three describes how to do reliable OTA file updates.

The 2.4GHz wifi band is getting crowded. Much of the heaviest traffic now goes on 5GHz, but that leaves a lot of devices competing for bandwidth; in particular those in the world of IOT. Modern domestic routers deal with this by channel-hopping; they periodically monitor the network to see which channel is the least busy and switch to that channel. This can happen as frequently as every hour or two, but I would hope that routers are intelligent enough to realise that constantly ping-ponging back and forth would be mostly ineffective as well as annoying. For although computer operating systems can deal pretty seamlessly with channel-hopping, simpler devices don't have the firmware to do this, leaving you and me to do it in our applications.

The simplest and most recommended approach is of course to restrict the router to a single channel, but this is only feasible if you have control of the router. For my master-slave system I made no such assumption; I decided to grab the bull by the horns and work out a solution.

Before continuing, I should mention that this code has been developed for the ESP32-C3, which is a cut-down device with only a single radio subsystem. This means it can only handle a single wifi channel. Other variants have two subsystems and can operate two channels independently. The code here and in the rest of the series is built to handle the more constrained environment. During development I found that many published articles about ESP-Now on the ESP32 don't work on the C3 variant as they rely on having the dual-radio.

In fact, with dual radio devices there may well be no need to channel-hop at all. The entire ESP-Now subsystem can operate independently of the AP and STA interfaces. However, the C3 variant occupies a niche that was previously filled by the ESP8266; low cost and very small size, as exhibited by devices such as the ESP32-C3 Super Mini and the ESP01-C3, allowing a full implementation of Micropython to run many cost and space-critical applications. Here's a ESP32-C3 Super Mini (with antenna) and an ESP01-C3.

So if you're still with me, here's the code:

import asyncio,machine,time

class Channels():
    def __init__(self,espComms):
        print('Starting Channels')
        self.espComms=espComms
        self.config=espComms.config
        self.channels=[1,6,11]
        self.myMaster=self.config.getMyMaster()
        if self.config.isMaster():
            self.ssid=self.config.getSSID()
            self.password=self.config.getPassword()
            asyncio.create_task(self.checkRouterChannel())
        self.resetCounter()

    def setupSlaveTasks(self):
        print('Set up slave tasks')
        asyncio.create_task(self.findMyMaster())
        asyncio.create_task(self.countMissingMessages())

    def resetCounter(self):
        self.idleCount=0

    async def findMyMaster(self):
        if await self.ping(): return
        self.hopToNextChannel()
        asyncio.get_event_loop().stop()
        machine.reset()

    async def ping(self):
        peer=bytes.fromhex(self.myMaster)
        self.espComms.espSend(peer,'ping')
        _,msg=self.espComms.e.recv(1000)
        print('Ping response from',self.myMaster,':',msg)
        if msg:
            print('Found master on channel',self.espComms.channel)
            return True
        return False

    async def countMissingMessages(self):
        print('Count missing messages')
        espComms=self.espComms
        ap=espComms.ap
        e=espComms.e
        self.idleCount=0
        while True:
            await asyncio.sleep(1)
            self.idleCount+=1
            limit=30
            if self.idleCount>limit:
                print('No messages for 30 seconds')
                # Retry the current channel
                if await self.ping():
                    self.idleCount=0
                    continue
                print(f'Master not found on channel {self.espComms.channel}\n')
                self.hopToNextChannel()
                asyncio.get_event_loop().stop()
                machine.reset()

    def hopToNextChannel(self):
        index=-1
        for n,value in enumerate(self.channels):
            if value==self.espComms.channel:
                self.espComms.channel=self.channels[(n+1)%len(self.channels)]
                index=n
                break
        if index==-1: self.espComms.channel=self.channels[0]
        self.config.setChannel(self.espComms.channel)

    async def checkRouterChannel(self):
        print('Check the router channel')
        while True:
            await asyncio.sleep(300)
            sta=self.espComms.sta
            sta.disconnect()
            time.sleep(1)
            print('Reconnecting...',end='')
            sta.connect(self.ssid,self.password)
            while not sta.isconnected():
                time.sleep(1)
                print('.',end='')
            channel=sta.config('channel')
            if channel!=self.espComms.channel:
                print(' router changed channel from',self.espComms.channel,'to',channel)
                asyncio.get_event_loop().stop()
                machine.reset()
            print(' no channel change')
            self.espComms.restartESPNow()

In this system, the ESPComms class described previously manages all network functions except for the channel hopping, which is done by this Channels class. The system also makes heavy use of a Config class, which manages system data such as SSIDs and passwords, I/O pin usage and so on. It also acts as a central routing point for most communications between other modules, having getter and setter functions that just pass on calls to the appropriate class. Channels is something of an exception as it's tied quite closely to ESPComms, so most of the function calls in this class go there directly.

`init()`

This defines the channels to be used. Although there are about 14 distinct channels in the 2.4GHz band, they are close enough together that adjacent ones overlap quite seriously. In fact, there are only three completely non-overlapping channels: 1, 6 and 11, so it will come as no surprise that most routers hop between these three. The function defines these channels. Its other job is to start up an asynchronous job (checkRouterChannel(), see below) to monitor the system and detect when the channel has changed. This is needed when the system acts as the master device, as otherwise the change of channel might go undetected.

`setupSlaveTasks()`

This is typically called later during initialization, after other things have settled down.

`resetCounter()`

The class constantly increments a counter, and when it reaches a predefined limit, action must be taken. This function is called from ESPComms whenever a message is received, to prevent the action from being triggered.

`findMyMaster()`

When a slave starts up it has no way of knowing which channel its master device is using. So it calls this function, which issues a 'ping' message on the channel that was saved during the last run. If it gets a reply it knows it's picked the right channel. Otherwise, it hops to the next channel, thereby saving the new channel number, and resets itself. Only once it gets a reply to the ping can it break out of this endless cycle and start normal operations.

`countMissingMessages()`

This function runs continuously, incrementing the counter and checking if it has exceeded its set limit. If so, it first pings the master to see if this is still awake and on the same channel. There may be a perfectly valid reason for missing messages, such as the system being in maintenance mode where normal operations are suspended. If the ping fails, the device will reboot and re-enter the findMyMaster() cycle above.

`ping()`

This sends a 'ping' message to the master device and waits for a reply, returning True if it gets one within a second. To save you having to look it up, the function espSend() in ESPComms is:

    def espSend(self,peer,msg):
        if self.addPeer(peer):
            try: self.e.send(peer,msg)
            except Exception as ex: print('espSend:',ex)

`countMissingMessages()`

This continually increments idleCount and checks if it has reached its predefined limit. If so, messages have gone missing, so it hops to the next channel and resets.

`checkRouterChannel()`

This is called periodically - say every 5 minutes - by the system master to ensure it is still on the same channel as the router. To do this, the STA interface must be deactivated and reconnected. If the channel has changed, a reboot is needed, otherwise ESP-Now can be restarted.

ESP-Now master-slave in Micropython

Graham Trott — Sat, 27 Sep 2025 14:25:23 +0000

This article describes how to build a master-slave network of ESP32 devices using Micropython and ESP-Now.

ESP-Now is a simple low-level protocol for communicating between ESP32-based devices independently of HTTP. Many articles have been written on the subject, but the great majority cover simple scenarios, and once things get more complex these simple techniques start to reveal their limitations. This article is the result of several weeks of trial and error, with extensive help from DeepSeek, which patiently sought out the answers to a seemingly endless series of baffling questions.

[Note: When I asked DeepSeek how best to make code available to a wider audience, it encouraged me to write this article. Forgive me for anthropomorphising slightly, but AI engines appear to enoy reading articles that describe working code. They also examine code repositories, but it seems they give greater weight to those that are supported by articles aimed at a technical readership.]

The system in question comprises a single master (hub) device that acts as a gateway to an indefinite number of slaves arranged as a star network. In a previous article, I described how to extend such a network by arranging for slaves to run their own local networks, which overcomes the problem of wifi range and also of the number of peers that any one node can support. The code I presented was somewhat simplistic and proved to have a number of limitations, so here I describe a revised class that deals with the problems unearthed during the development process.

I'll start by presenting the code, then I'll describe its most important features:

import asyncio,network,time,random,machine
from espnow import ESPNow

class ESPComms():
    e=ESPNow()

    def __init__(self,config):
        self.config=config

        if config.isMaster():
            print('Starting as master')
            try:
                self.sta=network.WLAN(network.WLAN.IF_STA)
                self.sta.active(True)
                ssid=self.config.getSSID()
                password=self.config.getPassword()
                print(ssid,password)
                t=0
                print('Connecting...',end='')
                self.sta.connect(ssid,password)
                while not self.sta.isconnected():
                    time.sleep(1)
                    t+=1
                    if t>30:
                        print('Timeout')
                        config.clearAndReset()
                    print('.',end='')
                ipaddr=self.sta.ifconfig()[0]
                self.channel=self.sta.config('channel')
                self.config.setIPAddr(ipaddr)
                print(f'{ipaddr} ch {self.channel}')
            except: machine.reset()
        else:
            self.channel=config.getChannel()
            print('Starting as slave on channel',self.channel)
        ap=network.WLAN(network.AP_IF)
        ap.active(True)
        ap.config(channel=self.channel)
        mac=ap.config('mac').hex()
        config.setMAC(mac)
        self.ssid=f'RBR-Now-{mac}'
        ap.config(essid=self.ssid,password='00000000')
        ap.ifconfig(('192.168.9.1','255.255.255.0','192.168.9.1','8.8.8.8'))
        self.ap=ap
        config.setAP(ap)
        print(config.getName(),mac)
        if not config.isMaster():
            self.sta=network.WLAN(network.WLAN.IF_STA)
            self.sta.active(True)

        self.e.active(True)
        print('ESP-Now initialised')

        self.requestToSend=False
        self.sending=False

    def closeAP(self):
        password=str(random.randrange(100000,999999))
        print('Password:',password)
        self.ap.config(essid='-',password=password)

    def addPeer(self,peer):
        h=peer.hex()
        if not hasattr(self,'peers'):
            self.peers=[]
        if h in self.peers:
            return True
        try:
            self.e.add_peer(peer,channel=self.channel)
        except OSError as ex:
            print(f'Failed to add peer {h} to ESP-NOW: {ex}')
            return False
        self.peers.append(h)
        print('Added',h,'to peers on channel',self.channel)
        return True

    def espSend(self,peer,msg):
        if self.addPeer(peer):
            try: self.e.send(peer,msg)
            except Exception as ex: print('espSend:',ex)

    def send(self,mac,msg):
#        print(f'Send {msg[0:20]}... to {mac} on channel {self.channel}')
        self.requestToSend=True
        while not self.sending: await asyncio.sleep(.1)
        self.requestToSend=False
        peer=bytes.fromhex(mac)
        if self.addPeer(peer):
            try:
                result=self.e.send(peer,msg)
                if result:
                    result=None
                    counter=100
                    while counter>0:
                        while self.e.any():
                            _,reply=self.e.irecv()
                            if reply:
                                reply=reply.decode()
                                if reply=='ping': continue
#                                print(f"Received reply: {reply}")
                                result=reply
                                break
                        if result: break
                        await asyncio.sleep(.1)
                        counter-=1
                    if counter==0: result='Fail (no reply)'
                    else:
                        print(f'{msg[0:20]} to {mac}: {result}')
                        self.config.resetCounter()
                else: result='Fail (no result)'
            except Exception as ex:
                result=f'Fail ({ex})'
                print(result)
        else: result='Fail (adding peer)'
        self.sending=False
        return result

    async def receive(self):
        print('Starting ESPNow receiver')
        while True:
            if self.e.active():
                while self.e.any():
                    mac,msg=self.e.recv()
                    msg=msg.decode()
                    print('Received',msg)
                    if msg=='ping':
                        try:
                            self.addPeer(mac)
                            self.e.send(mac,'pong')
                        except Exception as ex: print('ping:',ex)
                    else:
                        if msg[0]=='!':
                            # It's a message to be relayed
                            comma=msg.index(',')
                            slave=msg[1:comma]
                            msg=msg[comma+1:]
#                            print(f'Slave: {slave}, msg: {msg}')
                            response=await self.send(slave,msg)
                        else:
                            # It's a message for me
                            response=self.config.getHandler().handleMessage(msg)
                            response=f'{response} {self.getRSS(mac)}'
                        self.addPeer(mac)
                        try:
                            self.addPeer(mac)
                            self.e.send(mac,response)
                            print(response)
                            self.config.resetCounter()
                            if not self.config.getMyMaster() and not self.config.isMaster():
                                self.config.setMyMaster(mac.hex())
                        except Exception as ex: print('Can\'t respond',ex)
                if self.requestToSend:
                    self.sending=True
                    while self.sending: await asyncio.sleep(.1)
            else: print('Not active')
            self.config.kickWatchdog()
            await asyncio.sleep(.1)

    def getRSS(self,mac):
        try: return self.e.peers_table[mac][0]
        except: return 0

    def restartESPNow(self):
        self.e=ESPNow()
        self.e.active(True)

The job of this class is to deliver small packets of data to specified MAC addresses and to return results back to the caller. As ESP-Now is a low-level protocol it does not deal with IP addresses, access point names or any of the other higher layers of the OSI model.

ESP-Now requires the device to be already set up in either Access Point or Station mode. Its communications go unnoticed by higher-level functions such as HTTP, so it can run concurrently with them.

In this class, the master (hub) device connects to the house router with its station (STA) interface but the slaves do not (they still however require STA to be active for ESP-Now to work). Both master and slaves use the same code modules but in different ways. In both cases, an access point (AP) is set up for the first 2 minutes following a reboot, so that an administrator can talk directly to the device over-the-air from a local browser. This feature is used for configuration, mostly avoiding the need for ESP32 modules to be physically extracted for code alterations.

In this implementation, all messages, commands and requests are initiated outside of the ESP-Now network and handed to an HTTP server maintained by the master for onward transmission. Slave devices do not initiate messages other than the occasional 'ping' (see below). Adding full two-way messaging would result in considerable extra complexity, and all the necessary features can usually be provided without it.

The code here was developed with a lot of trial-and-error and the assistance of AI to find relevant examples. There turned out to be none that directly considered the complexities of the scenario; most are far simpler. If anyone else has solved the same problem, they hadn't told the world about it at the time this code development was underway.

`init()`

The class makes use of a second module, Config, which handles system parameters such as router SSID/password and identifies which I/O pins are used by the device. This data is typically held in a JSON file that can be read and written as necessary. Wherever you see config.xxxxx(), a function in the config module is being called.

ESP-Now is extremely fussy about configuration. There may be differences between the variants of ESP32; this code is known to work on ESP32-C3, which is a fairly basic model, so it should be fine on most other variants. There is a strict sequence to follow, which differs between master and slave.

The master first requires STA to be active. Here it is connected to the house router, which governs the wifi channel the system will use. Then the AP is set up, and finally ESP-Now is activated. This ordering guarantees that ESP-Now will use the STA wifi channel.

The slave does things the other way round, first setting up AP and following that with a stub STA. This ensures that the AP channel will be used for ESP-Now communications. If you don't get this order right, some things will probably not work and strange errors will arise.

`closeAP()`

During the first 2 minutes following boot, the AP can be connected from a browser and will accept requests and commands. These include rewriting the JSON config file and rebooting the device. After this period, the SSID of the AP is changed to - and a random password is applied, making it almost impossible to discover and connect to. This is a security feature that prevents malicious interference unless the attacker can physically repower the device.

`addPeer()`

It's very easy to get ESP-Now peer management wrong. The published documentation gives few hints as to how fussy the protocol is about handling peers. For example, it fails to mention that a channel change not only invalidates the peers table but makes it impossible to recreate. Only a reset will do that.

Although ESP-Now will happily receive messages from anywhere, sending can only be done to a peer whose MAC address has been registered with e.add_peer(). This must be done just once for any given peer or an error will be thrown. So this function manages its own peers list to track what has already been added. Note that the device is likely to have 2 MAC addresses; one for each of its 2 interfaces. When you initiate a send it will usually be to one of these, but when replying to a message it will be the other. Both must be added to the peers table.

`send()` and `receive()`

The receive() function runs asynchronously in a loop, waiting for messages and replying to them. The original version of this class allowed messages to be sent without any consideration of what might happen if both activities try to take place at the same time. In a simple situation this will often work, but it's extremely vulnerable. One solution that appeared in the AI searches was to use a lock, so that send and receive can be exclusive, but this gives rise to some horrible issues that are hard to track down. The strategy here is one that avoids such issues and is much easier to debug.

In the receive loop, when a message arrives it is passed to the handleMessage() function in a companion Handler class, which returns a reply to send back to the caller. While this is happening, a sending flag is set False. Any request to send will cause the sender to raise requestToSend and block until sending goes True, which is done at the bottom of the receive loop. The receive function can tell from requestToSend that the sender is waiting. The function can now release sending to allow the message to be sent, and it then waits for that process to finish.

However, a received message might not have come from the device we just sent to. This system is designed to handle router channel changes, and when this happens the master quickly changes to the new channel, but all the slaves know is that messages have stopped arriving (because they're all on the wrong channel). So after a while they issue a 'ping' message to the master. If it replies they're on the right channel, otherwise they change channels and try the ping again.

The result is that one or more 'ping' messages can arrive while the sender is waiting for a reply to its own message, so the code watches out for them and discards them. Eventually it sees its own reply message and returns it to the caller, but before doing so it clears sending, which releases the receiver to look out for more incoming messages.

Note that this code cannot deal with anything but 'ping' messages from its slaves. Any scenario requiring a more general 2-way send and receive will probably need to use some form of addressing to identify message senders.

The code in receive() has one more feature. If the received message starts with ! it's not for this device, but instead is a request to forward a message to another device. The MAC address of the intended recipient follows the ! and is terminated by a comma. The code strips these away and is left with a MAC address and a (shorter) message, which it forwards, returning whatever returns back up to its own caller.

Channel changes

Because the 2.4Ghz wifi band is often quite crowded, most modern wifi routers run a background task that looks out for the least busy channel. Every so often, typically at intervals of several hours, the router will then change to this quieter channel. PC operating systems are aware of this behaviour and engage in a dialog with the router to get advance notice of an upcoming change, so the effect is barely noticeable by the users of these machines. However, simple IOT devices are not generally equipped to deal with channel changes; all they know is that messages stop arriving. To avoid the need to restrict a router to a single channel, the Channels class works with the code above to deal with this and other related issues, and this is described in part two of this series. Part three describes the message handler, which includes a strategy for reliably updating Python modules over the air.

Full code and documentation for the project can be found at https://github.com/easycoder/rbr. The Micropython code is in the RBRNow folder. I may be able to keep the code above in step with changes as they happen, but it would be better to use the repository version.

An AI virtual keyboard

Graham Trott — Mon, 12 May 2025 15:30:49 +0000

I only recently overcame my usual inertia and responded to the invitations by VSCode to start using Copilot. Since then I've spent quite some time refining a fairly complex prompt, so I've decided to share my experience with the community.

I operate in the world of Python. In this world there really is no excuse for not using a graphical interface wherever one is appropriate. It's really easy to get Copilot to create a UI to accept input and present output, though a little harder to produce something that's also aesthetically pleasing. But there's a point at which a prompt takes longer to write than does coding the entire job manually.

The project I'm working on is a home heating control system, the control code and the user interface for which are both written in Python, the latter with some help from Copilot. The controller hardware is a 7-inch industrial computer from Ali Express, which arrived with a dreadful Android implementation but which I successfully reflashed to Debian Linux. Now it's a proper computer, but it lacks an essential feature; there's no virtual keyboard. (There seem to be a couple of truly appalling ones out there, but I wouldn't inflict them on a customer.)

So the need was for a Python virtual keyboard. Quite a tedious manual programming task, but I figured it would be ideal for Copilot to get its teeth into. I decided to emulate an Android keyboard, which has 4 views that switch when shift or mode keys are tapped, and above the keys there's a text field for the characters being typed. Here's the first view of the keyboard:

'Prompt engineering' is the term that refers to the skill needed to get effective results from an AI tool. As with most things in life, the GIGO principle applies; Garbage In, Garbage Out. I found it necessary to apply a good deal of guidance or the results had severe limitations and were a poor starting point for further manual embellishment. But if you're careful with your prompts you can get quite decent Python code from an AI engine.

I say "can" because there's a massive caveat coming. A keyboard is not an exceptionally complex item, but without a lot of guidance all the AI engines I tried produced poorly structured code that didn't behave properly. So I tried again, this time leveraging my existing knowledge of Python and PySide (the Qt graphics library module) to structure my prompt. I requested a series of classes - a keyboard button, a row of buttons, a keyboard view comprising several rows, and finally the keyboard itself - giving for each class a set of function definitions and letting Copilot fill in the details. It turned out to be quite a substantial task. In fact, the prompt - even though it's incomplete - is far longer than the final working Python code - somewhere around 500 lines versus 320 lines of Python.

One problem is that human-computer interfaces are a lot more complex that might be imagined, and a lot of details have to be spelled out in, well, excruciating detail. That's a job for a computer program, of course, which begs the question of what kind of language we should be using to write AI prompts. It's early days for this technology, but I feel there's a Eureka! moment waiting to be unearthed.

Sadly, the rather long prompt below will not create a complete keyboard. For example, there's no text box to receive the typed characters. And if your experience is anything like mine, you won't get a complete set of keyboard views; Copilot just generates a comment that views 1, 2 and 3 are similar to view 0 and invites you to do the rest yourself. It seems to be influenced by previous versions of your prompt and decides to ignore bits of the new one. However, what you do get is probably enough to make completing the job by hand relatively easy.

And that's my main take-away from this exercise. There are a lot of details that are simply too tedious to cover in an all-encompassing prompt script. You're far better off generating the main outline then using smaller prompts to fill in the nit-picky little details.

The bottom line is, AI isn't going to take away your job and do it better. Not yet, anyway. But it can be a most valuable assistant, so please start using it. And your comments are welcome - I'm still at the start of the learning curve.

(Cover image by Gemini)

This prompt describes a virtual keyboard,
to be written in Python using Pyside6 graphics.
I provide Python-style class definitions,
with constructor and function prototypes.
Python comments are general notes and instructions
to help build the classes.

#---------------------------------------------------------------

class KeyboardButton(QPushButton):
    def __init__(self, width, height, callback, text, icon):

# The text to be shown on the button can be any string or None
# The icon to be shown on the button can be any graphic, or None
# The button has rounded corners of radius 20% of its height
# The font size is half the button height
# The background of the button is white
# Apply a light shadow to the right and bottom of the button
# When the button is clicked it moves 2 pixels down and right,
# waits 200ms then moves back to its original position.
# On click, call function 'callback' with the value of 'text'.

#---------------------------------------------------------------

class KeyboardRow(QHBoxLayout):
    def __init__(self, list):  # add the items in 'list'

#---------------------------------------------------------------

class KeyboardView(QVBoxLayout):
    def __init__(self, list):  # add the items in 'list'

#---------------------------------------------------------------
This class is a complete keyboard with 4 views,
which are added one by one and only one of which can be visible.

class VirtualKeyboard(QStackedWidget):
    def __init__(self, buttonHeight):
        super().__init__()
        self.buttonHeight = buttonHeight
    def addView(self, view): # Add 'view' to the VirtualKeyboard
    def getCurrentView(self): # return the current index
    def onClickChar(self, keycode): # Print the key code
    def onClickShift(self, keycode): # Print 'Shift'.
        # if currentView is 0 call AbstractKeyboard.setCurrentIndex(1)
        # elif currentView is 1 call AbstractKeyboard.setCurrentIndex(0)
    def onClickNumbers(self, keycode):  # Print `Numbers`
        # call AbstractKeyboard.setCurrentIndex(2)
    def onClickLetters(self, keycode):  # Print `Letters`
        # call AbstractKeyboard.setCurrentIndex(3)
    def onClickBack(self, keycode):  # Print 'Back'
    def onClickSpace(self, keycode):  # Print `Space`
    def onClickEnter(self, keycode):  # Print 'Enter'

# This class sets up a keyboard

# __init__() calls each of the addKeyboardLayout() functions

The 4 keyboard views are as follows.
All buttons have the height given by buttonHeight.
All buttons have the same width as their height
unless specified otherwise.

#---------------------------------------------------------------
# View 0

Create an empty row list

Create a list of KeyboardButton with the characters in 'qwertyuiop'
The 'callback' function is onClickChar
Create a KeyboardRow with this list
Add the KeyboardRow to the row list

Create an empty list of widgets
Add a horizontal stretch to the widget list
Create a list of KeyboardButton with the characters into 'asdfghjkl'
The 'callback' function is onClickChar
Append this list to the widget list
Add a horizontal stretch to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'up.png'
    the 'callback' function is onClickShift
Append the KeyboardButton to the widget list
Append a spacer whose width is 5% of the standard height
Create a list of KeyboardButton with the characters in 'zxcvbnm'
The 'callback' function is onClickChar
Append this list to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'back.png'
    the 'callback' function is onClickBack
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'numbers.png'
    the 'callback' function is onClickNumbers
Append the KeyboardButton to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with the character ','
The 'callback' function is onClickChar
Append this button to the widget list
Create a KeyboardButton with
    a width 500% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'space.png'
    the 'callback' function is onClickSpace
Create a KeyboardButton with the character '.'
The 'callback' function is onClickChar
Append this button to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'enter.png'
    the 'callback' function is onClickEnter
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create a KeyboardView with the row list
Add the KeyboardView to this VirtualKeyboard

#---------------------------------------------------------------
# View 1

Create an empty row list

Create a list of KeyboardButton  with the characters in 'QWERTYUIOP'
The 'callback' function is onClickChar
Create a KeyboardRow with this list
Add the KeyboardRow to the row list

Create an empty list of widgets
Add a horizontal stretch to the widget list
Create a list of KeyboardButton  with the characters in 'ASDFGHJKL'
The 'callback' function is onClickChar
Append this list to the widget list
Add a horizontal stretch to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'up.png'
    the 'callback' function is onClickShift
Append the KeyboardButton to the widgt list
Append a spacer whose width is 5% of the standard height
Create a list of KeyboardButton with the characters in 'ZXCVBNM'
The 'callback' function is onClickChar
Append this list to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'back.png'
    the 'callback' function is onClickBack
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'numbers.png'
    the 'callback' function is onClickNumbers
Append the KeyboardButton to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with the character ','
The 'callback' function is onClickChar
Append this button to the widget list
Create a KeyboardButton with
    a width 500% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'space.png'
    the 'callback' function is onClickSpace
Create a KeyboardButton with the character '.'
The 'callback' function is onClickChar
Append this button to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'enter.png'
    the 'callback' function is onClickEnter
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create a KeyboardView with the row list
Add the KeyboardView to this VirtualKeyboard

#---------------------------------------------------------------
# View 2

Create an empty row list

Create a list of KeyboardButton  with the characters in '1234567890'
The 'callback' function is onClickChar
Create a KeyboardRow with this list
Add the KeyboardRow to the row list

Create a list of KeyboardButton  with the characters in '@#£&_-()=%'
The 'callback' function is onClickChar
Create a KeyboardRow with this list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    the standard height
    a text value None
    an icon taken from the graphic 'symbols.png'
    the 'callback' function is onClickSymbols
Append the KeyboardButton to the widgt list
Append a spacer whose width is 5% of the standard height
Create a list of KeyboardButton with the characters in '"*'
Add "'" to the list
Add the characters in ':/!?+' to the list
The 'callback' function for everything in this list is onClickChar
Append this list to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    the standard height
    a text value None
    an icon taken from the graphic 'back.png'
    the 'callback' function is onClickBack
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'letters.png'
    the 'callback' function is onClickLetters
Append the KeyboardButton to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    a width 500% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'space.png'
    the 'callback' function is onClickSpace
Create a KeyboardButton with the character '.'
The 'callback' function is onClickChar
Append this button to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'enter.png'
    the 'callback' function is onClickEnter
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create a KeyboardView with the row list
Add the KeyboardView to this VirtualKeyboard

#---------------------------------------------------------------
# View 3

Create an empty row list

Create a list of KeyboardButton  with the characters in '$€¥¢©®µ~¿¡'
The 'callback' function is onClickChar
Create a KeyboardRow with this list
Add the KeyboardRow to the row list

Create a list of KeyboardButton  with the characters in '¼½¾[]{}<>^'
The 'callback' function is onClickChar
Create a KeyboardRow with this list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    the standard height
    a text value None
    an icon taken from the graphic 'numbers.png'
    the 'callback' function is onClickNumbers
Append the KeyboardButton to the widget list
Append a spacer whose width is 5% of the standard height
Create a list of KeyboardButton with the characters in '`;÷\∣|¬±'
The 'callback' function is onClickChar
Append this list to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    the standard height
    a text value None
    an icon taken from the graphic 'back.png'
    the 'callback' function is onClickBack
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    the standard height
    a text value None
    an icon taken from the graphic 'symbols.png'
    the 'callback' function is onClickSymbols
Append the KeyboardButton to the widgt list
Append a spacer whose width is 5% of the standard height
Create a list of KeyboardButton with the characters in '"*'
Add "'" to the list
Add the characters in ':/!?+' to the list
The 'callback' function for everything in this list is onClickChar
Append this list to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with
    the standard height
    a text value None
    an icon taken from the graphic 'back.png'
    the 'callback' function is onClickBack
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create an empty list of widgets
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'letters.png'
    the 'callback' function is onClickLetters
Append the KeyboardButton to the widget list
Append a spacer whose width is 5% of the standard height
Create a KeyboardButton with the character ','
The 'callback' function is onClickChar
Append this button to the widget list
Create a KeyboardButton with
    a width 300% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'space.png'
    the 'callback' function is onClickSpace
Create a KeyboardButton with the character '.'
The 'callback' function is onClickChar
Append this button to the widget list
Append a spacer whose width is 5% of the standard height
Create a list of KeyboardButton with the characters in '✕§¶°'
The 'callback' function is onClickChar
Append this list to the widget list
Create a KeyboardButton with
    a width 150% of the standard height
    the standard height
    a text value None
    an icon taken from the graphic 'enter.png'
    the 'callback' function is onClickEnter
Append the KeyboardButton to the widget list
Create a KeyboardRow with the widget list
Add the KeyboardRow to the row list

Create a KeyboardView with the row list
Add the KeyboardView to this VirtualKeyboard

#---------------------------------------------------------------

This is a test program:

Create a window 600 pixels wide and 350 high with background #ccc.
Compute a standard button height value, as 15% of the window height. 
Create a VirtualKeyboard, giving it the computed height
Add the VirtualKeyboard to the window
Show the window

Note: QGraphicsDropShadowEffect is part of QtWidgets, not QtCore

And here are the control key graphics:

Using subnets to extend ESP-Now

Graham Trott — Sat, 19 Apr 2025 16:03:45 +0000

ESP-Now is a lightweight networking protocol that uses only two layers (Physical and Data Link) of the OSI model. By doing so it offers high performance and simple programming.

In a typical configuration, a single hub device maintains an HTTP connection to the system controller, either as a client or a server, and uses ESP-Now to communicate directly with up to 20 other devices. Commands and queries arrive at the hub on the HTTP interface and are routed to the appropriate device. These messages are sent using a call to a library function that takes 2 arguments:

The MAC address of the intended recipient
The payload, which can be up to 200 bytes. (A new version increases this limit but I'll ignore that for the purposes of this article.)

For a small network, the system offers great performance, impressive reliability and easy programming, but it has two fundamental limitations, being the 20 devices as stated above, and the distance from the hub to the furthest device. As with all things wifi, range is an issue. It can be very hard to find a location for a wifi hub where all of the devices to be controlled can see it reliably. Here's view of a hub with 4 connected devices:

The red hatched area is outside the wifi range of the hub, so if new devices are added here they will suffer from poor connections and will consequently be unreliable:

One well-known solution to this is to implement a mesh, but this functionality is not offered as part of the ESP-Now spec. In any case, a mesh is often overkill, since what is usually needed is for the hub to drive more devices over a greater range, not for all the devices to talk to each other in arbitrary ways.

Fortunately, there's a quite simple solution to this problem, which is to allow any device to handle devices on a subnet of its own. The device count and the range of the network both immediately increase dramatically, albeit with a performance hit, but one that is likely to be insignificant for most small control systems:

The way it works is like this. Every device in the system is described by a packet of information, typically a JSON structure, with fields such as the following:

"name" - the unique name of the device in the system
"mac" - the MAC address of the device
"channel" - the wifi channel used
"led" - the pin that drives the device LED
"led-invert" - a Boolean that governs the ON and OFF polarity
"relay" - the pin that drives the device's primary relay
"relay-invert" - a Boolean that governs the ON and OFF polarity
... (etc for other hardware and software features of this device)
"path" - the path to this device

Each device has a JSON file containing its own information, and the central controller holds a file containing all the device structures, indexed by the device names.

The last item in the list is the key to the extended networking. For devices close to the hub, nothing is needed here as messages are sent using the MAC address of the device and the payload. But by placing the MAC address of an intermediate device into the field, that device then becomes the parent of the target device (the one the message is intended for). The message is now sent to the parent (the "path" as above), to be forwarded to our required target device. We prefix the payload with the MAC address of the actual target device and structure it so as to make it easy to decode.

As an example, let's suppose we want to send a command to turn on the device relay. Let's suppose the MAC address of the device is aa:aa:aa:aa:aa:aa and the payload to turn on the relay is the single word ON. If the network is small and the device is close to the hub we can just send it as is, using a function call such as

send(mac, payload)

but if the network is full or the device is physically remote we need to relay the message through another device already in the system, one that is closer to our new device. Let's say the MAC address of this parent device is bb:bb:bb:bb:bb:bb. So we send a message to that device, and we adjust the payload to be

!aa:aa:aa:aa:aa:aa,ON

The function call then becomes (micropython version)

send(parent, f'!{mac},{payload}')

The exclamation mark is there to signal that the payload is to be routed to the device whose MAC address is given, and the comma is just a separator. Needless to say, this strategy does not allow payloads to begin with an exclamation mark, so you might wish to use a different marker.

The code to handle this new message type is remarkably simple. You have to check for the marker, then extract the embedded MAC address and use it to construct a new message to send to the target device. The payload is everything that follows the comma - in this case the original ON - and the reply from the target is simply passed back to your caller. You can see this done in the receive function below.

This technique adds no extra devices to the list of "peers" that are held by any given device, and it can be extended almost indefinitely, by concatenating MAC addresses in the "path" field, such as

bb:bb:bb:bb:bb:bb,cc:cc:cc:cc:cc:cc

which causes the message to be routed through 2 intermediate devices.

I reproduce below the ESP comms class from a Micropython implementation of the above. It relies on a config parameter being supplied during initialization, to provide access to other functions in the system, but mostly it just deals with sending and receiving ESP-Now messages. You will notice that the system holds MAC addresses as human-readable strings, whereas ESP-Now requires formatted binaries, so there's a degree of back-and-forth between the two formats.

import asyncio
from binascii import hexlify,unhexlify
from espnow import ESPNow as E

class ESPComms():

    # Default initializer
    def __init__(self,config):
        self.config=config
        config.setESPComms(self)
        E().active(True)
        self.peers=[]
        print('ESP-Now initialised')

    # Check if a peer is already known
    def checkPeer(self,peer):
        if not peer in self.peers:
            self.peers.append(peer)
            E().add_peer(peer)

    # Send a message to a given MAC address
    async def send(self,mac,espmsg):
        peer=unhexlify(mac.encode())
        self.checkPeer(peer)
        try:
            print(f'Send {espmsg[0:20]}... to {mac}')
            result=E().send(peer,espmsg)
            print(f'Result: {result}')
            if result:
                counter=50
                while counter>0:
                    if E().any():
                        sender,response = E().irecv()
                        if response:
                            print(f'Received response: {response.decode()}')
                            result=response.decode()
                            break
                    await asyncio.sleep(.1)
                    counter-=1
                if counter==0:
                    result='Response timeout'
            else: result='Fail'
        except Exception as e:
            print(e)
            result='Fail'
        return result

    # Handle received messages as they arrive
    # Either deal with them or pass them on
    async def receive(self):
        print('Starting ESPNow receiver on channel',self.config.getChannel())
        while True:
            if E().any():
                peer,msg=E().recv()
                sender=hexlify(peer).decode()
                msg=msg.decode()
                print(f'Message from {sender}: {msg[0:20]}...')
                if msg[0]=='!':
                    comma=msg.index(',')
                    slave=msg[1:comma]
                    msg=msg[comma+1:]
#                    print('Forward to',slave,', msg:',msg)
                    response=await self.send(slave,msg)
                else:
                    response=self.config.getHandler().handleMessage(msg)
                    response=f'{response} {self.getRSS(sender)}'
#                print('Response',response)
                self.checkPeer(peer)
                E().send(peer,response)
            await asyncio.sleep(.1)
            self.config.kickWatchdog()

    # Return the RSS (signal strength) of the received message
    def getRSS(self,mac):
        peer=unhexlify(mac.encode())
        return E().peers_table[peer][0]

Chained wi-fi networking for control systems

Graham Trott — Sat, 24 Aug 2024 20:59:52 +0000

This describes a networking strategy for controlling simple wifi devices distributed over a large physical area, without the need for repeaters. The code is Micropython and the system was designed for a network of 4MB ESP8266 devices. It was originally developed for a home heating control system but is equally well suited to handling the needs of industrial heating, irrigation or similar systems, where several devices need to be controlled centrally.

Background

Most small-scale WiFi networking systems adopt a “star” topology:

with all the control and sensor nodes connected to a single hub; often a standard internet router. This will be located close to where the service provider's cable enters the property, often in a remote corner. Devices such as the ESP8266 have small antennae and operate poorly over anything more than quite a short distance. This shows up as delays in responding, frequent resets and even crashes, but these may not become noticeable until after the installation has been completed and signed off. The system would then require a time-consuming rework.

This problem can be overcome by the use of extra routers or network extenders, to form what is known as a 'mesh' system, but these may be visually intrusive and they add extra cost and complexity to the system.

In the case of the home heating system, there was an additional problem, that if the internet connection went down the local NAT network would also fail and the system controller could not access the relays and other devices. To avoid this, the controller was fitted with a second wifi interface to provide a private LAN, and the entire control network was placed on that. In the event of a WAN failure the system carries on happily by itself.

This works fine up to a point, but small computers like Raspberry/Orange pi can only support a limited number of wifi devices, and it didn't deal with devices being too far away to work reliably unless mesh routers were added.

A secondary issue concerned updates to the firmware inside the devices. When updates were required it was necessary to manually visit each device in turn and perform an update.

This article describes a simple, low-cost approach where the controlled devices themselves take care of all the extended networking. It can handle a large number of devices and can be deployed using very modest hardware such as an ESP8266. The system connects to its controller through a single wifi address and can be completely configured and tested before delivery for installation. Software updates propagate silently and automatically through the system.

Push or pull

In a "push" scenario, a central controller determines the states of each of the controlled devices and "pushes" messages to each of them via REST, MQTT or some other comms mechanism. This is fine while it's working but can cause major timing issues when devices go offline and the controller doesn't get the responses it expected.

This system adopts a "pull" strategy. As before, the controller decides the states of the devices, but it's then up to each of them to poll the controller whenever it wishes and to act on the response it receives back. After several months' testing there have been no signs of timing issues. Occasionally a device will lose contact, suffer a memory overflow or some other problem that causes the local watchdog to force a reset, but this has little to no effect on the behaviour of the overall system.

Outline of the strategy

This strategy employs some of the networked devices as message relays handing data to and from other nearby devices. The network then looks like this:

Implementation

The strategy centres on a packet of information constructed by the system controller. Encoded as JSON, this contains a section for each of the networked devices, keyed by the name of the device. It also contains the current timestamp and the version number of the system firmware. Let's call this packet the “map”. Here’s an example of a simple system with two relay nodes:

{“Room1“: {“relay“: “off“}, “Room2“: {“relay“: “on“}, “ts“: “1719094849“, “v“: “90“}

The content of each device section in the map depends on the device. A relay has its state, on or off. A thermometer usually requires no information since its job is to return data, not consume it.

One device is designated as the "entry point" to the network. It connects to the access point (wifi hotspot) published by the hub, which either is the system controller itself or an interface device that connects to it. The entry device knows nothing of its special role; it behaves just like every other node. The reason for having a single entry node is so the hub doesn’t have to support more than one wifi client or combine messages coming from multiple sources. Once connected, the entry device polls the hub at regular intervals such as every 2 seconds, and the hub responds by sending it the map. The device looks in the map for its own name and extracts the attached data to perform whatever tasks are needed. For a relay this will be to turn on or off. For a thermometer, no actions are needed at this stage.

When a device starts up it constructs an empty 'return packet', which it maintains for the lifetime of the running program. The device adds to this packet the timestamp contained in the last outgoing map. For a relay, that's all there is; a thermometer also adds the current temperature. The return packet looks like this:

{“Room1“: {“ts”: “1719094849”}}
or
{“Thermo1“: {“temp”: “25.3”, “ts”: “1719094849”}}

When a device polls its parent it supplies its current return packet, which includes those of its own child devices as explained below. If polling stops or for any other reason the timestamps don't get updated, the controller can tell there's something wrong in the system by checking the ages of the returning timestamps.

Adding more devices

Devices like the ESP8266 and ESP32 can operate simultaneously in Station and in Access Point mode. That is, they can connect to a wifi hotspot while at the same time publishing one of their own. These local hotspots have limited functionality but can usually support up to 4 connections, which allows us to expand the system by connecting devices to each other in a chain rather than all to a central hub. In this system, each device sets up a hotspot with an SSID based on its own MAC address and a network IP address that's different to the one it’s connected to.

When a device receives a polling request from a child device it extracts the contained return packet and adds it to its own return packet, for onward transmission to its own parent on the next poll. For example, if Room2 is a child of Room1, then following a poll of Room1 by Room2 the return packet from Room1 will expand to look like this:

{“Room1“: {“ts”: “1719094849”}, “Room2“: {“ts”: “1719093995”}}

As we move back along the chain towards the entry point device, the return packet grows to include all the nodes before finally being delivered to the hub on the next poll.

After dealing with the return packet, the device responds to the poll request by returning the current outgoing map (that came from the controller) to the child device that polled it. In this way, the outgoing map propagates to all devices.

When devices are added to the system they do not connect directly to the system hub as they would in a star topology. Instead, they connect to the nearest already configured device that has remaining capacity for another connection. Since each device publishes a hotspot, it's easy to check on a smartphone which one has the strongest signal, to guarantee reliable system performance.

The behaviour of every device is identical so it’s very easy to configure the system; the device name and its parent SSID are the only items that differ from one device to another. And so the system grows, with devices being connected to any other device that can offer a good wifi signal.

The system also has the welcome ability to operate without the need to manually track IP addresses. Devices are identified by unique names and SSIDs, where the latter are created automatically from the device MAC address. Each device chooses for its own hotspot an IP address range that's different to the one used by its parent. In the prototype systems, these ranges alternate between 172.24.100.x and 172.24.101.x. Each device uses one or the other.

It will often be useful to create a simple spreadsheet to record the essential details of the system, as a guide while installing and when doing maintenance. Here's the table for a home heating system:

In this example, Request is the entry device. (Here it's the relay that controls whether the boiler - or in this case heat pump - should run. It's ON when any radiator is ON.)

It's quite simple to check signal strengths at a given location using any computer or smartphone, to decide which devices can connect to which parents. When your computer connects to a device hotspot, the gateway address (whose 4th octet is 1, e.g. 172.24.100.1) runs a small HTTP server that returns basic information about the device; its name, SSID, parent and how long it's been up and running.

Over The Air (OTA) updates

The version number handed out by the hub is used to keep the system firmware up to date. Each device keeps a note of its current version. When a message packet (map) arrives with a higher version number, the device requests a list of files from its parent, and then requests each of the files listed, one by one. Once it has finished updating it saves the new version number.

While updating is taking place, normal operation is suspended. Clients of a device will not get responses to their polling and will eventually time out and reset themselves. Once updating has finished, a device will restart normal operation and when it receives the map from its parent will pass it on. Since this now contains an updated version number, each of its clients will start its own update. So the latest version ripples through the system and everyone ends up with the same code.

If the system is a mix of different device types, the code for all of them must be contained in each. So a relay contains thermometer code and vice versa.

Upsides

Ability to handle a large number of devices spread over a large physical area
No wifi black spots
All IP addresses are inferred by the devices themselves
Very low cost
The messaging protocol is flexible. It can be extended to add more complex information exchanges, without any compatibility issues.

Downsides

The main downside of this system is the time it takes for messages to propagate through the network. It's not suitable for use where a rapid response is needed, as response times are measured in seconds or even tens of seconds. There are however many applications for which this is not a problem, such as home or commercial heating systems and large-scale irrigation.

The code

The device on which this code runs, typically an ESP8266, must have at least 4M bytes of flash memory in order to support the full set of Micropython modules required - most importantly, asynchronous functions. If you are using the ESP-01 be sure to get the 4MB version, not the older 1MB.

Since this is an ongoing project, any code I present here will be out of date very rapidly. So instead, here is a link to the GitHub repository, which includes all the source files referred to in the following paragraphs:
(https://github.com/easycoder/rbr/blob/main/roombyroom/Controller/home/orangepi/firmware/XR)

As with all Micropython projects there is a boot file and a main program. First, boot.py:

from machine import Pin

led = Pin(2, Pin.OUT)
led.on()

import esp
esp.osdebug(None)

import gc
gc.collect()

This is pretty standard. Next, main.py, which will always run automatically following boot.py:

import os

def fileExists(filename):
    try:
        os.stat(filename)
        return True
    except OSError:
        return False

if fileExists('config.json'):
    if fileExists('update'):
        import updater
        f = open('update','r')
        value=f.read()
        f.close()
        updater.run(value)
    else:
        import configured
        configured.run()
else:
    import unconfigured
    unconfigured.run()

The operating mode is determined by a couple of files kept in the device flash. The file config.json contains information specific to this device. If this file does not exist the device is unconfigured; if it exists then the device is considered to be configured and ready to run. In the latter case, the file update is checked; if this one also exists the device enters update mode, which I’ll cover last.

Note that the main function modules are only called in if they are needed for the chosen mode. This avoids having to load everything up front when much of it will remain unused.

Unconfigured mode

This mode is handled by unconfigured.py. The mode is optional; if you provide a config.json file manually the device will never run unconfigured. It exists so that a batch of devices can be flashed with identical firmware and then each one configured individually using a browser.

The job of the module is to run a web server that offers a configuration form for the user to supply configuration data. At the top are a couple of HTML pages, then some utility functions and finally the main code.

The key feature is the use of asynchronous functions to permit multitasking. The program runs 2 jobs; one waits for a client to connect and the other flashes the LED. Since neither of these can be allowed to block, asynchronous code is needed. The first thing is to create the two tasks, using asyncio, then enter an endless loop while they perform their duties.

The code sets up a webserver whose SSID is of the form RBR-xr-xxxxxx, where RBR-xr- is a product descriptor and xxxxxx are the last 6 digits of the device MAC address. The IP address of the server is 192.168.66.1 (or anything else you prefer). Connecting to it at http://192.168.66.1/config brings up the following screen:

The fields are as follows:

Relay Name: Any name that uniquely identifies this relay in the network. Spaces are not allowed.
Host SSID: The SSID of the device this one should connect to.
Host Password: The Wifi password of the host.
My Password: The Wifi password that will allow access to the server on this device once configured.

When Setup is clicked the fields are combined into a JSON structure and saved into the device flash area as config.json. Then the device resets itself. If all is well it should restart in Configured mode.

Configured mode

This mode is handled by configured.py. As with unconfigured mode, the code runs concurrent tasks using the asyncio library. One of these tasks is the main program; the other is a watchdog that monitors activity to ensure polling runs as expected. If for any reason it stops (such as the device’s host going offline), the watchdog forces a reset after a decent interval. This avoids un-needed resets when momentary failures occur.

Most of the functions required by the code - other than standard libraries - are kept in two local modules; functions.py and hardware.py. These will be described later.

The main progam starts by doing some initialization. Two special dictionary objects - outgoingMap and incomingMap - hold the data passing around the network. Both of these, once created, last for as long as the program is running.

The code also sets up a local access point (hotspot) with an SSID that is similar but not identical to the one used in configured mode. In this example it will be RBR-XR-xxxxxx, where xxxxxx is as before the last 6 digits of the MAC address. Since two of the characters in the product descriptor are now uppercase, the device is easy to find in a list of local access points.

In the main loop, the code polls its parent device at regular intervals, sending it the incomingMap. The reply from the parent - the outgoingMap - is parsed to find if the local relay should be on or off. The version number in outgoingMap is compared with that currently saved, and if there’s been an advance the device creates an update file and forces a reset.

When a client device polls this one, the incoming request is parsed to find the command and its data (if any). The format of the request is one of

http://(my ip address)/command
http://(my ip address)/command?data

The commands recognised are as follows:

reboot forces a restart of the code.
reset erases the configuration file then forces a reboot, causing the device to restart in unconfigured mode.
getFile?{filename} is a request for the contents of the named file to be returned. This is used by the updater.
poll?data={incomingMap} Poll the parent of this device, supplying the incoming map (or the part of it known to this device). The outgoing map is returned to the caller.

Any other command will return a line of general information about the device and its status.

The 'functions' modules

A set of helper functions is kept in functions.py and hardware.py. These are provided as two files to help avoid running out of memory during updates.

A single third-party library, uaiohttpclient, which handles asynchronous HTTP requests, is placed in the lib directory.

Update mode

This mode is handled by updater.py. The updater runs its main task and a watchdog. The main task starts by requesting from its parent device a list of all the files that comprise the code set. This list is saved as files.txt, then each of the files named is requested. When all have successfully been dealt with, the new version number is saved.

The mechanism for processing each file has to be robust and able to recover from errors. The procedure is as follows:

1 Download the requested file and save it as a temporary file.
1 Read the saved file and check it’s the same as the data downloaded. If not, force a reset.
1 If there is no existing file with the name requested, rename the temporary file and return.
1 If there is an existing file, read it and compare it with the downloaded file. If they are the same, no update is needed so delete the temporary file and return.
1 If they are different, delete the current (old) version, rename the temporary file and return.

Future work

The system is agnostic about the environment in which it operates. In the current prototypes the hub is a small computer such as a Raspberry or Orange pi, where control is done by a PHP module on an Apache web server. A useful next step would be to create an integration for Home Assistant. A hub module - probably itself an ESP8266 - would handle a complete chain of devices and present them to HA as a single item via MQTT. A suitable API would allow any named device in the chain to be addressed and controlled independently via the hub.

Dynamic arrays in C++

Graham Trott — Thu, 15 Feb 2024 21:43:58 +0000

I am an engineer turned self-taught programmer, with no formal computer science training. (That's by way of an excuse for what follows.) For about 10 years up to about 1995 I used C but never really got fully to grips with it. Then I moved on, first to Java then later JavaScript and Python. Recently however I started some ESP8266 projects. I tried Micropython but found it lacking in “bare metal” capabilities and it takes up too much space in a 512KB device.

So it was time to return to C - or more specifically C++. It came as something of a shock to rediscover how complex it is, how easy it is to make simple programming errors that cause programs to crash, and how lacking C++ is in features we take for granted in other languages. One of these is dynamic arrays, which allow elements to be freely added without the need to consider memory management.

There are of course libraries out there that could serve my needs, but I usually find it's a better use of my time (and more enjoyable) to spend a couple of days writing code rather than hunting down potential candidate libraries, getting to understand them and often as not rejecting them as unsuitable. So if the code that follows is similar to something already available then it's a complete coincidence.

My dynamic arrays start with a linked list class. This has a no-argument constructor and 3 main methods:

void add(element) adds the supplied element to the list. It throws an exception if the request fails

int getSize() returns the number of elements in the list

void* get(index) returns the element at the specified position

void clear() clears the list

The list is doubly-linked (forwards and backwards) and needs no malloc() commands as the elements added are all pointers to the original items. All elements are stored as void* pointers so the list can handle any kind of data. An inner LinkedListElement class handles each element.

Here's the code:

#ifndef LINKED_LIST
#define LINKED_LIST

#include <stdio.h>

/*
    This is a doubly linked list element able to store any kind of data.
    The content is immutable within this class.
*/
class LinkedListElement {

    private:

       const void* content;
       LinkedListElement* previous = nullptr;
       LinkedListElement* next = nullptr;

    public:

       // Get the content of this element
       void* get() {
           return (void*)content;
       }

       // Get the previous element pointer
       LinkedListElement* getPrevious() {
           return previous;
       }


       // Set the previous element pointer
       void setPrevious(LinkedListElement* data) {
           previous = data;
       }


       // Get the next element pointer
       LinkedListElement* getNext() {
           return next;
       }


       // Set the next element pointer
       void setNext(LinkedListElement* data) {
           next = data;
       }


        // Constructor
        LinkedListElement(const void* data) {
            content = data;
        }

        // Destructor
        ~LinkedListElement() {}
};

class LinkedList {

    private:
        int size = 0;                          // the number of items
        LinkedListElement* head = nullptr;
        LinkedListElement* tail = nullptr;

    public:

        ///////////////////////////////////////////////////////////////////////
        // Get the size (the number of elements in the list)
        int getSize() {
            return size;
        };

        ///////////////////////////////////////////////////////////////////////
        void add(const void* data) {
            LinkedListElement* element = new LinkedListElement(data);
            if (size == 0) {
                head = element;
                tail = head;
            } else {
                tail->setNext(element);
                element->setPrevious(tail);
                tail = element;
            }
            ++size;
        }

        ///////////////////////////////////////////////////////////////////////
        void* get(int index) {
            if (index < size) {
                LinkedListElement* element = head;
                while (index > 0) {
                    element = element->getNext();
                    --index;
                }
                return element->get();
            }
            return nullptr;
        }

        ///////////////////////////////////////////////////////////////////////
        // Clear the list. Remove everything except the element data.
        void clear() {
            LinkedListElement* walker = head;
            if (walker != nullptr) {
                while (walker->getNext() != nullptr) {
                    LinkedListElement* next = walker->getNext();
                    delete walker;
                    walker = next;
                }
                delete walker;
            }
            size = 0;
        }

        // Default constructor
        LinkedList() {}

        // Destructor
        ~LinkedList() {
            clear();
         }
};

#endif

Linked lists are simple and effective, but to retrieve an element means “walking” the list, which has a performance overhead that gets significant for large lists. By contrast, an array provides instant access to its elements but requires knowledge of its size to allocate memory for its list of pointers before any elements can be added. To get around both these limitations we need to combine an array with a linked list.

The next class is StringArray, which is an array of character strings referenced by char* pointers. As well as this array it also contains a LinkedList. The significant methods of the class are

void add(element) adds the supplied element to the StringArray. It will be added to the internal linked list.

int getSize() returns the number of elements held by this class instance. This is the sum of those in the array and those in the linked list.

char* get(index) returns the element at the specified position. The element might be taken from either the array or the linked list, depending on the index and how many elements there are in each part.

void flatten() “flattens” the array. It creates a new array big enough to hold the contents of the existing array plus those in the list, copies all of their pointers to the new array then clears the list and frees the old array. Note that the data itself is untouched and unaffected. This method is “idempotent”; that is, it can be called as often as desired. If the list is empty it does nothing.

Here's the code:

#ifndef STRINGARRAY
#define STRINGARRAY

/*
    StringArray is a memory-efficient class for managing arrays of strings.
    A typical usage is to break a piece of text into lines and make them
    available by line number.
*/
class StringArray {

    private:
        int size = 0;                       // the number of items
        char** array = (char**)malloc(1);   // the array of items
        LinkedList* list;                   // a list to hold new data items

    public:

        ///////////////////////////////////////////////////////////////////////
        // Get the size (the number of elements in the array)
        int getSize() {
            return this->size + list->getSize();
        };

        ///////////////////////////////////////////////////////////////////////
        // Get a specified item.
        // If the index is greater than the array size, return the item from the list.
        char* get(int n) {
            if (n < size) {
                return array[n];
            }
            else if (n < size + list->getSize()) {
                return (char*)list->get(n - size);
            }
            return nullptr;
        };

        ///////////////////////////////////////////////////////////////////////
        // Add an item. This goes into the linked list.
        void add(const char* item) {
            list->add(item);
        }

        ///////////////////////////////////////////////////////////////////////
        // Flatten this item by creating a single array to hold all the data.
        void flatten() {
            char** oldArray = array;
            int oldSize = size;
            // Create a new array big enough for the old array and the list
            int total = oldSize + list->getSize();
            if (total > 0) {
                array = (char**)malloc(sizeof(StringArray*) * (total));
                // Copy the old array to the new
                size = 0;
                while (size < oldSize) {
                    array[size] = oldArray[size];
                    size++;
                }
                free(oldArray);
                // Copy the list to the new array
                int n = 0;
                while (n < list->getSize()) {
                    array[size++] = (char*)list->get(n++);
                }
                list->clear();
            }
        }

        ///////////////////////////////////////////////////////////////////////
        // Default constructor
        StringArray() {
            list = new LinkedList();
        }

        ///////////////////////////////////////////////////////////////////////
        // Destructor
        ~StringArray() {
            free(array);
            delete list;
         }
};

#endif

Not shown here are useful methods such as indexOf(char* item), which searches for and returns the index of an item in the array. My own implementation also has a parse(char* data) method that takes a single string containing a number of items separated by a common delimiter (such as lines of a text document or a CSV file), breaks them up into individual strings and adds them all to the array.

The final class is ObjectArray, which is an array of arbitrary objects referenced by void* pointers. This class is useful for holding data of varying types. It differs only very slightly from the previous StringArray. Here’s the code:

#ifndef OBJECTARRAY
#define OBJECTARRAY

/*
    ObjectArray is a memory-efficient class for managing arrays of arbitrary objects.
*/
class ObjectArray {

    private:
        int size = 0;                         // the number of items
        void** array = (void**)malloc(1);     // the array of items
        LinkedList* list;                     // a list to hold new data items

    public:

        ///////////////////////////////////////////////////////////////////////
        // Get the size (the number of elements in the array)
        int getSize() {
            return this->size + list->getSize();
        };

        ///////////////////////////////////////////////////////////////////////
        // Get a specified item.
        // If the index is greater than the array size, return the item from the list.
        void* get(int n) {
            if (n < size) {
                return array[n];
            }
            else if (n < size + list->getSize()) {
                return (void*)list->get(n - size);
            }
            return nullptr;
        };

        ///////////////////////////////////////////////////////////////////////
        // Add an item. This goes into the linked list.
        void add(const void* item) {
            list->add(item);
        }

        ///////////////////////////////////////////////////////////////////////
        // Flatten this item by creating a single array to hold all the data.
        void flatten() {
            void** oldArray = array;
            int oldSize = size;
            // Create a new array big enough for the old array and the list
            int total = oldSize + list->getSize();
            if (total > 0) {
                array = (void**)malloc(sizeof(ObjectArray*) * (total));
                // Copy the old array to the new
                size = 0;
                while (size < oldSize) {
                    array[size] = oldArray[size];
                    size++;
                }
                free(oldArray);
                // Copy the list to the new array
                int n = 0;
                while (n < list->getSize()) {
                    array[size++] = list->get(n++);
                }
                list->clear();
            }
        }

        ///////////////////////////////////////////////////////////////////////
        // Default constructor
        ObjectArray() {
            list = new LinkedList();
        }

        ///////////////////////////////////////////////////////////////////////
        // Destructor
        ~ObjectArray() {
            free(array);
            delete list;
         }
};

#endif

Here is a simple test program to illustrate its usage. (Yes, I know the classes defined here are all animals and could share a common superclass, but that’s not the point of the exercise.)

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "linkedlist.cpp"
#include "objectarray.cpp"

class Cat {
  private:
    const char* name;
  public:
    char* getName() { return (char*)this->name; }
    Cat(const char* name) { this->name = name; }
};

class Dog {
  private:
    const char* name;
  public:
    char* getName() { return (char*)this->name; }
    Dog(const char* name) { this->name = name; }
};

class Rabbit {
  private:
    const char* name;
  public:
    char* getName() { return (char*)this->name; }
    Rabbit(const char* name) { this->name = name; }
};

class Canary {
  private:
    const char* name;
  public:
    char* getName() { return (char*)this->name; }
    Canary(const char* name) { this->name = name; }
};

// Main program
int main(void) {

  ObjectArray* pets = new ObjectArray();
  pets->add(new Cat("Garfield"));
  pets->add(new Dog("Scooby"));
  pets->add(new Rabbit("Bugs"));
  pets->add(new Canary("Tweety Pie"));

  pets->flatten();    // optional

  printf("I have a cat called %s,\n", ((Cat*)pets->get(0))->getName());
  printf("a dog called %s,\n", ((Dog*)pets->get(1))->getName());
  printf("a rabbit called %s,\n", ((Rabbit*)pets->get(2))->getName());
  printf("and a canary called %s.\n", ((Canary*)pets->get(3))->getName());
};

The output when run is

I have a cat called Garfield,
a dog called Scooby,
a rabbit called Bugs,
and a canary called Tweety Pie.

Having written these simple classes I am amazed by how much the rest of my code now depends on them. I hope others may find them just as useful.

Photo by Pierre Bamin on Unsplash

It's in there somewhere

Graham Trott — Tue, 26 Dec 2023 15:12:47 +0000

My partner takes a lot of photos. Since smartphones replaced compact cameras for taking photos, everything has been shovelled into Google Photos, but previously she would take a pocket camera everywhere with her. This left her with thousands of images distributed in a messy filing system comprising hundreds of folders and no real organization. She owned several computers over that period, and each time one packed up, most of its hard drive got copied to the (much larger one) on the replacement computer. So duplication is rife. Everything is in there somewhere, but it's all hard to find and it's taking up way too much space. So what should we do about it?

There are two ways to approach the problem. One is to find a tool that will do the job; the other is to write one. The problem with the first of these is that it will take time to find such a tool, largely because I can't figure out a suitable question to ask Google. Having found a tool I then have to learn how to use it. Unless it's top-drawer commercial software its documentation - if there is any - will likely be cryptic if not incomprehensible, so this part could take hours. Then there's finally the good chance that it won't actually do the job after all, so the whole exercise is wasted. And if the original search came up with several products, everything has to be repeated for each one.

So that leaves writing my own. At the time of writing it's the Christmas break, so what better time to do some easy coding with none of the usual work pressures? I decided to write the thing in Python as a command-line tool; you'll find the source in the EasyCoder repository at https://github.com/easycoder/unpick. By the time you read this, the repo version may well have moved on, so the documentation below could be out of date.

(Note: Writing a tool is one thing. Finding a good name for it - that hasn't already been used - is quite another.)

Running the tool with no arguments outputs some basic help, as follows:

A recursive tool to list extensions, copy/move all files
of a given type, remove duplicates etc.
Syntax:
unpick [options] [path]
Options (note that many of these are mutually exclusive):
   -types, -extensions, -ext -- list all file extensions
   -list  -- list all files with the given extension
   -duplicates, -dup  -- list all duplicates with the given extension
   -delete, -del -- delete all files with the given extension
   -remove, -rem -- delete all duplicates with the given extension
   -copy -- copy files to a target directory
   -move -- move files to a target directory
   -type, -extension, -ext -- specify the extension
   -target [path] -- copy/move to this target directory
   -flatten -- don't maintain directory structure in copy/move
   -ignore -- ignore duplicates in copy/move
   -help -- this help message

The tool walks any given directory recursively, taking appropriate actions as it goes. Here are some examples of how to use each option (at the time of writing). In each case, <path> is the path (absolute or relative) to the directory being scanned:

unpick -types <path>

Output a list of all the file types (extensions) found in the directory and its subdirectories, with the number of files of that type, sorted most to least.

unpick -copy -type doc -target <target folder> <path>

Copy all files with the given extension, putting them into target folder. The same folder hierarchy will be maintained, or you can add the -flatten option to force everything to go into a single directory. You can also add the -ignore option to ignore duplicates.

unpick -move -type pdf -target <target folder> <path>

Move all files with the given extension, putting them into target folder. This is the same as copy but the copied files will be deleted as they are moved.

unpick -duplicates -type jpg <path>

List all duplicate file names. The first one encountered is taken to be the original and all others duplicates. The fact that the names are the same doesn't necessarily mean they are the same file, of course, so if there's any doubt you'll need to examine the output carefully. Maybe a later version of this tool will examine the content as well as the name.

unpick -remove -type png <path>

Delete all duplicate files (files with the same name but in different directories). Use duplicates first to check these really are the same file. Also be aware that the first instance of any file discovered by unpick is assumed to be the master copy, which may cause the wrong copy to be deleted.

So did it work? Well, after I ran copy once for each wanted file type, the extracted files took up just over 100GB, compared to more than 500GB on the original backup disk. Most of the rest is old emails, ancient Windows executables and so on, none of which need to be kept, and it should now be a lot easier for my partner to find photos, videos or documents should the need arise. The results are far from perfect, as it can still take a while to burrow through the folder hierarchy or to pick the right file from a huge number in one directory, but at least it's an improvement and it does save a lot of space.

I hope others here may also find it useful in pruning the immense piles of clutter we fill our computers with. I could do with spending some time adding extra goodies such as a check that a duplicate file name really means an identical file, but if someone else gets there before me I'm happy to use their code.

Setting up a low-cost server

Graham Trott — Mon, 12 Sep 2022 20:19:25 +0000

This article is really about choosing an alternative to a Raspberry Pi and using Network Manager to set up wifi on a headless install, but I couldn't get all that into the title. In fact, networking causes more headaches than pretty well anything else so I think the title is actually quite useful, as once that part is done the rest is plain sailing.

For several years the Raspberry Pi has been the preferred choice for small servers, as well as for a host of hobbyist projects. It has decent performance, small size, an attractive price and is well supported by both official and community help channels.

However, the recent chip shortages have hit availability and as a result the price has multiplied several times. At $25 it's excellent value; at $150, meh. So many people have been looking for decent alternatives.

There are many rivals to the RPi, but none enjoy anything like the same breadth of support. There is a distinct lack of good articles out there on how to set up one of these alternatives, so here goes.

The Orange Pi Zero 2 is equivalent in performance to a Pi Model 3, if not slightly better. It's considerably smaller and has everything needed for duty as a small webserver. It's a 4-core, 64-bit model with 1GB of RAM, an RJ-45 Ethernet port, wifi, an SD socket, a single USB port, a micro-HDMI port and a USB-C power socket. And it's available at a very attractive price from Ali Express. I chose a package that bundles the computer with a rather neat transparent case, fired off an order for two and they arrived in the UK in a couple of weeks.

So, now being the proud owner of a pair of rather dinky little computers, the big challenge is what software to run on them. There are a selection of alternatives, notably Armbian, Debian and Ubuntu. I've tried all three and my preference is for Debian.

Debian is available via the Orange Pi website. I choose the bullseye server version since I have little need for a desktop; my application is a local controller for smart radiator valves. I flash a 16GB SD card using Balena Etcher, which takes a couple minutes, then put the card into the OPi, connect an Ethernet cable to my router and power up. A red light comes on right away and few seconds later it goes out and a green one appears. After a minute or two the device is visible on my router's connected devices screen, and a short while after that the SSH server is ready to go. So I log in as root with the password orangepi.

The big job is to set up wifi because I don't want to be tethered to the router all the time. A Google search for help on setting up wifi on Linux turns up a good number of pages, but... nearly all of them are out of date. Since they were written there have been some big happenings in the way networking is managed. You can wave goodbye to dhcpcd, networks/interfaces, ifconfig, ifup and ifdown, because they're all history!

The new kid on the block is Network Manager, a superbly-crafted toolkit that sweeps away all the clutter and makes setting up networking a breeze, just as it ought to be. It's included in new OS builds and specifically in Debian bullseye for the Orange Pi Zero 2.

I'm going to add wifi to my OPi and give it a static IP address. To do this I run the strangely-named nmtui (could that be Network Manager Text UI?) and the user interface pops up:

I move the highlight to Activate a connection, find my own router and click Activate:

I have to give the network password:

which shows the connection as Activated.

I click Back to return to the home screen, then Edit a connection and select the connection I just activated:

I click Edit and the main editing screen appears:

I move down to IPv4 CONFIGURATION and click Automatic to change it to Manual:

then I fill in all the relevant fields. The result looks like this:

Then I go to the bottom of the form, click OK to return to the previous screen, Back to return to the home screen and Activate a connection. I move to my chosen connection and click Deactivate. Then I select it again and click Activate. Finally, I back out of nmtui and close my ssh session by typing exit.

Now I can ssh back in again, either at the previous address using Ethernet or at the new one (192.168.0.95 in my example) using wifi. As well as root, Debian also sets up an orangepi user account, also with the same orangepi password. Now I can disconnect the Ethernet cable and use just wifi. From here on, things depend on the application. The hardest part is in getting networking to work, and the creators of Network Manager have done a superb job that takes away all the frustration of setting up wifi on a Linux headless system.

A Smart Heating System

Graham Trott — Fri, 28 Jan 2022 16:26:19 +0000

Photo by Aleksandar Cvetanovic on Unsplash

Introduction

The cost of domestic heating is currently going through the roof (so to speak). Here in the UK in winter/spring of 2022 we're facing a 50% or more increase in the cost of town gas and electricity, so energy-saving measures are a hot topic (ha!). This piece is about a private project: a DIY hardware/software system for home heating control.

I'd been thinking for a while about building a system to control radiators or other heating types, with electric actuators to replace the thermostatic ones I had at present. Something using Arduinos or Pi Picos, perhaps. Then, a couple of months ago, by lucky chance I discovered the Bulgarian company Shelly™, who sell a range of smart, low-cost relays and sensors, each one having a wifi interface. Since my main interest was in building software, not hardware, this was too good to pass up, so I bought a small kit of parts and set to work.

The aims of the project

The primary aim was to build an energy-saving system that would regulate the temperature in each room independently and turn off the heating in rooms that are not being used, either with timing rules, movement detectors or just manually. Everything would be controlled from a smartphone; not just from inside the house but from anywhere with an Internet connection. And the project would respect Occam's Razor, avoiding complexity wherever possible.

Some requirements and constraints soon became apparent:

The user interface is a browser-based mobile webapp. I don't have the time, experience or inclination to program native apps for Android and iPhone.
The system is set up as a DIY project requiring little more expertise than the ability to wire up a plug. Full documentation is (or will be) provided as part of the UI. It should offer commercial opportunities for contractors to deploy systems to their own customers.
For keen coders, all the software is Open Source except for the Shelly items, which for the more dedicated can be replaced by home-builds if so desired.
The software should be accessible to the widest possible range of interested parties, which means no complex build or deploy toolchains and the absolute minimum of dependencies. This should avoid being held hostage to the future, but also make the software accessible to amateur and part-time programmers as well as full-time professionals.

System description

So with the above aims in mind, this is how I structured the system:

The control hardware is based on Shelly 1 relays (see their on-line store) that control standard 30mm electric radiator valve actuators. They can also be used with direct resistance heating as they are able to switch up to 16A. A Shelly H&T (Humidity and Temperature) unit is provided in each room to report the temperature at regular intervals.

Here's a photo of a "room kit" - all the parts needed to equip a single room with a single radiator, using a standard UK pattress box. The relay is the small blue item inside the box, which also has a neon indicator to show when power is applied to the actuator (right of photo). It plugs into a standard wall outlet and communicates by wifi with the system controller. On the upper left is a Shelly H&T thermometer unit.

The relays and sensors are controlled by a small computer such as a Raspberry Pi. It doesn't have to be the latest Model 4; I've successfully used a Model 1B, which although very slow is able to do the job and is available at a low cost. It runs a local HTTP server into which the thermometer modules post messages, and a control program that runs through a set of rules for the system to decide when and where heating is needed and turn relays on and off as appropriate. All the Pi software is written in Python (indirectly, as I'll explain later).

The controller reports changes of temperature to an external web server using REST requests. I use standard public shared hosting and that pretty well mandates PHP, but (for those who hate the language) the REST server is only a few hundred lines of PHP and once built can mostly be forgotten.

The user interface is a browser-based webapp written in JavaScript (indirectly - see later) and built as a single-page application (SPA). It gives the user a view of the current state of the system and the ability to add and remove rooms, change the target temperature and set up timing schedules of any complexity. It includes a full help stack (or will eventually) and a demo module running behind the scenes that gives a full simulation of a system without the need to buy any hardware.

How it works

The user's interaction with the webapp creates a system map - a JSON structure with details of the rooms, the sensors and the relays. When any changes occur the map is posted to the server with a flag set to request confirmation. The system controller picks up the map every few seconds and confirms, passing back the current states of the sensors. These are then retrieved by the webapp and displayed to the user.

All the software is held in a GitHub repository. Anyone wishing to contribute to the project will be welcomed, whether it's to work on the software, the user interface or the documentation. The project is currently in a fairly early state of development, with a growing list of things to fix and things to add.

Here are some notes on the component parts of the software:

The system controller

This is Python code running on a headless Raspberry Pi. Any other computer would do just as well but few can match it on price or compactness. The local HTTP server uses Bottle, which can be regarded as a leaner version of Flask, and it has very little to do. Each thermometer unit regularly reports its temperature to the local HTTP server, which writes it to a file whose name is the IP address of the thermometer.

The control program is run as a cron task, once every minute. It downloads the system map from the web server, collects the current temperature values from the appropriate local thermometer files and posts them back to the server. It also uses this information to decide if heating should be turned on or off in any room, and sends commands to the relays, each of which also has a local IP address and is listening on port 80.

As hinted earlier, the controller code is not written directly in Python but in an English-like scripting language that is itself written in Python. The compiler for this language runs on the Pi itself and is supplied with the project as Python source files. High-level source scripts compile on the fly and the compiler output is passed directly to a built-in runtime engine. Here's a snippet to give the general flavour; here, all capitalized words are script variables:

...
open File Path cat `/sensors/` cat Sensor cat `.txt` for reading
read SensorValues from File
close File
put json SensorValues into SensorValues
put float property `temperature` of SensorValues into Temperature
put property `relays` of Room into Relays
put property `events` of Room into Events
take 1 from the length of Events giving L

! Deal with mode requests
put property `mode` of Room into Mode
if Mode is `off` go to TurnOff

if Mode is `on`
begin
    put float property `target` of Room into Target
    if Temperature is less than Target go to TurnOn
    go to TurnOff
end
...

The reason for doing this is for accessibility and maintainability. Changes are simply a matter of editing a text file and writing it back to the Pi. Using English-like script means the code - to a considerable extent - resembles the way you would describe the operation of the system in words. Software professionals commonly assume that everyone shares their ability to read code and understand complex frameworks and build tools, but that's far from the case. The majority - including some quite competent programmers - find it very hard to read conventional code. Our brains are wired for words, not mathematical expressions.

There's a performance price to pay, of course. The control program is little more than 200 lines of script but takes nearly half a second to compile on a Pi Model 1B, though later models slash this down to well under 20ms. However, even on the old model this is quite acceptable; a couple of seconds to get the program up and running is of little importance in a one minute cron cycle.

The user interface

The UI home page is intended for mobile use and currently looks like this:

As with the system controller, an unusual approach is used here too, in the form of another high-level script compiler and runtime, this one written in JavaScript. It's syntactically very similar to the Python version and designed to facilitate the creation of complex interactive user interfaces without the need to learn any complex programming techniques. In terms of accessability and long-term maintainability there is nothing to beat English. For a native English speaker the learning curve is very short as scripts tend to follow the natural way of describing what's happening. Which is that a DOM is created and the user interacts with it. The compiler/runtime is loaded directly from the repository; all relevant URLs can be found at the end of this article.

Some might argue that all this represents an unnecessary additional layer of complexity. I would counter that most large coding projects today use a complex framework and require an additional build/deploy stage with its own learning requirements. The compiler used here - EasyCoder - is far simpler than products such as React or Angular and is extensively documented in its own website and repository. The ability to run directly from a high-level script greatly speeds development and maintenance, and runtime performance is scarcely an issue; on a modern smartphone Chrome will compile a thousand-line script in well under 50ms.

Additionally, a high-level scripting language makes the underlying JavaScript invisible, except perhaps when tracking down the more obscure bugs. This is great for people who don't live and breathe JavaScript but who would like to know what's happening.

DOM construction

This project also uses an unconventional approach to DOM construction; one that I described in a recent Dev.to article. Webson is a JSON format; a language for representing DOM trees. In this project, no HTML is present. Instead, all of the screen elements are defined in a series of Webson files, which combine layout and style in a format not dissimilar to CSS. Webson permits - but neither encourages nor discourages - the use of external CSS classes but none are used so far in this project. With screens of this kind the layout is quite closely associated with its appearance so I personally prefer to keep the styles in with the layout elements they affect.

Here's the code for the icon and text at the left-hand end of each room in the UI photo above when the operating mode is set to "on". As you can see, much of it is CSS:

{
    "#doc": "The icon and text for 'on' mode",
    "#element": "button",
    "@id" : "room-/ROOM/-mode-icon",
    "display": "flex",
    "flex-direction": "row",
    "width": "10em",
    "#": ["$Button", "$Padding", "$Text"],

    "$Button": {
        "#element": "div",
        "margin-right": "0.5em",
        "pointer-events": "none",
        "#": "$Icon",

        "$Icon": {
            "#element": "img",
            "@src": "resources/icon/temperature.png",
            "height": "2em",
            "padding": "0.5em"
        }
    },

    "$Padding": {
        "#element": "div",
        "width": "0.2em",
        "pointer-events": "none"
    },

    "$Text": {
        "#element": "div",
        "display": "flex",
        "flex-direction": "column",
        "flex": 1,
        "height": "100%",
        "text-align": "left",
        "pointer-events": "none",
        "#": "$Inner",

        "$Inner": {
            "#element": "div",
            "display": "inline-block",
            "flex": 1,
            "text-align": "center",
            "padding": "0.2em",
            "#": ["$Content1", "$Content2"],

            "$Content1": {
                "#element": "div",
                "#content": "On",
                "font-size": "1.4em",
                "font-weight": "bold"
            },

            "$Content2": {
                "#element": "div",
                "@id" : "room-/ROOM/-target",
                "font-size": "0.9em",
                "font-weight": "bold"
            }
        }
    }
}

Using Webson in an EasyCoder script is simple. It's generally 2 lines of code:

rest get MainMenuScript from `/resources/webson/mainmenu.json`
render MainMenuScript in Container

where Container is a script variable that acts for the specific element - usually a div - that's already in the DOM and is due to receive the new content.

Demo mode

The hardware needed for a typical installation costs a couple of hundred pounds/dollars/euros, so I added a "demo mode" for system evaluation. This runs a background script that mimics the effects of heating being turned on and off, and runs in accelerated real time. Users can set up entire systems and watch them perform according to the built-in rules.

Documentation

Every software product needs good documentation; for system installers, users and maintenance people. There's no better place to put it than in the user interface, where it can't get lost. To make the writing job as simple as possible I chose Markdown, enhanced to support some extra tags and to fit in with the single-page design. This is a modified version of another past project from the same overall repository and is delivered as a single high-level script.

Every screen and every popup dialog has a Help button that leads to a page of help in a cross-referenced stack.

Resources

The mobile webapp is at https://rbrheating.com/home. Its documentation can be viewed by navigating from the screen that appears the first time the site is visited, or from the Help button that's present on every screen.

The project repository is at https://github.com/easycoder/rbr.

EasyCoder is at https://easycoder.github.io with its repository at https://github.com/easycoder/easycoder.github.io.

The Webson repository is at https://github.com/easycoder/webson.

The Storyteller (enhanced Markdown) repository is at https://github.com/easycoder/storyteller. This project uses a modified version whose source script is included in the main repository.

Webson: a new DOM markup

Graham Trott — Mon, 25 Oct 2021 17:50:27 +0000

Photo by Hal Gatewood on Unsplash

Synopsis

This article introduces Webson, an easy-to-use syntax with its own run-time rendering engine, that turns JSON into DOM markup and adds features way beyond those of HTML.

Introduction

A web page is the visual representation of a Document Object Model, or DOM, the data structure maintained internally by all browsers. Traditionally, the DOM is constructed by the browser from HTML scripts, but as pages get bigger and more complex HTML becomes ever more cumbersome. In recent years it has become increasingly common to create the DOM using JavaScript, with no HTML ever being seen, but while this suits programmers well it requires a different skill set from that needed to build pages the traditional way.

Today's Web pages may have hundreds or thousands of elements, all carefully positioned to create the desired result. There's no way to hide this complexity, whether it's done with HTML/CSS, JavaScript or some kind of no-code visual builder. In the end it's a human brain that's doing the real work of translating the customer's requirements - a mental picture - into something the browser can use to create the DOM.

HTML is not program code; it's a form of "markup", the ultimate expression of which came in the form of XML, able to represent not only visual structures but a wide range of other data too. Unfortunately, XML is wordy and hard to read and is not greatly loved. In 2001, Douglas Crockford invented (he would say "discovered") a simpler syntax for representing data structures, as a means of transferring data in and out of JavaScript programs in the form of plain text. The syntax is JavaScript Object Notation, or JSON, and in the past 2 decades it has widely supplanted XML. Virtually every programming language has the ability to read and write JSON and it's now the most common way to transfer data across the Web.

Since HTML shares many of the disadvantages of XML, the question might be asked, Can JSON also replace HTML?. If the answer is "yes", a couple of supplementary questions might be

Can we have user-defined variables and reusable blocks?
How about conditional structures?

which would greatly reduce the amount of markup needed to describe a complex web page, where items are commonly repeated with only minor differences.

Webson

Webson is a markup syntax that allows JSON to be used to describe a DOM, together with a JavaScript rendering engine that can be embedded in any web page to process scripts at runtime. The system is immediately usable by HTML/CSS coders and no JavaScript experience is required. It's aimed at simplifying the design and implementation of highly complex layouts, where precise positioning of large numbers of elements is hard to achieve manually, and it achieves this with JSON markup rather than with code.

Getting started

Let's start with a simple example; a layout commonly found in online magazines and social media. At the top there's a full-width header; under this a central panel with 2 sidebars and at the bottom a footer. As this is only an example I've given each of the component divs its own background color so it stands out clearly. It looks like this:

Here's the HTML that will create this screen. It uses inline styles to avoid the need to present a separate CSS file:

    <div style="width:50%;height:50%;display:flex;flex-direction:column">
        <div id="top" style="height:20%;background:cyan">
        </div>
        <div style="width:100%;flex:1;display:flex;flex-direction:row">
            <div id="left" style="display:inline-block;width:25%;height:100%;background:green">
            </div>
            <div id="center" style="display:inline-block;height:100%;background:yellow;flex:1">
            </div>
            <div id="right" style="display:inline-block;width:15%;height:100%;background:blue">
            </div>
        </div>
        <div id="bottom" style="height:10%;background:magenta">
        </div>
    </div>

This is a total of 655 characters. The corresponding Webson script to create the same screen is 1172 characters, nearly twice as many, and occupies 61 lines rather than 14, but before you dismiss Webson as being too wordy I must say in its defence that this is a very basic example which doesn't make use of any of the more advanced features of the system. More complex scripts tend to be far smaller than their HTML equivalents, as we'll see later.

The reason for the extra size in this example is partly that every item is named and partly because JSON itself is fairly bulky (lots of double-quotes), while the increase in lines is mainly because it's a lot more spaced out. This helps readability; high information density makes code hard to read at a glance as the eye has to pick out specific details from a dense surrounding mass. With Webson, the CSS properties are separated out, one per line, rather than all being crammed onto a single line. This can of course be done with HTML too, but because there's no agreed way to present it the result is usually an unstructured mess, so most coders just put everything on the same line.

Here's the script. It just uses a basic feature set; I'll get on to some of the advanced features later.

{
    "width": "50%",
    "height": "50%",
    "display": "flex",
    "flex-direction": "column",
    "#": ["$Top", "$Middle", "$Bottom"],

    "$Top": {
        "#element": "div",
        "height": "20%",
        "background": "cyan"
    },

    "$Middle": {
        "#element": "div",
        "width": "100%",
        "flex": 1,
        "display": "flex",
        "flex-direction": "row",
        "#": ["$Left", "$Center", "$Right"]
    },

    "$Bottom": {
        "#element": "div",
        "height": "10%",
        "background": "magenta"
    },

    "$Left": {
        "#element": "div",
        "display": "inline-block",
        "width": "25%",
        "height": "100%",
        "background": "green"
    },

    "$Center": {
        "#element": "div",
        "display": "inline-block",
        "flex": 1,
        "height": "100%",
        "background": "yellow"
    },

    "$Right": {
        "#element": "div",
        "display": "inline-block",
        "width": "15%",
        "height": "100%",
        "background": "blue"
    }
}

How it works

Running through the script, you will see that every DOM element has its own named block of JSON data. User-defined names all start with $. There are also directives and other system items; the names of these start with #. Everything else in the script above is a CSS style to be applied to the current element.

In the above, most of the blocks include a #element directive, which names the DOM element type. If this is missing, everything in the block applies to the current element (the one defined in the block that calls this one). Here the only block that lacks an #element is the very first one, so its styles all apply to the parent container that was created outside Webson and passed to its renderer as a parameter.

The symbol # by itself signals that child elements are to be added. This directive takes either a single name or an array of names.

Attributes

The structure we've built here isn't much use unless we can add further items to the various divs. Some of this can be done with further Webson code but ultimately you'll either use an onClick="<something>" callout or a JavaScript function that populates or interacts with the DOM. For the latter to work, elements must have unique ids to allow JavaScript to find them. Here's the $Left block again, with an id and a couple of other additions:

    "$Left": {
        "#debug": 2,
        "#doc": "The left-hand sidebar",
        "#element": "div",
        "@id": "left",
        "display": "inline-block",
        "width": "25%",
        "height": "100%",
        "background": "green"
    },

Here we have another new symbol, @, which (appropriately) signifies an attribute. Various HTML elements require special attributes such as @id, @class, @type, @href, @src, etc. In each case the name is that of the HTML attribute prefixed by @.

Another feature above reveals a built-in debugging capability. When hand-building HTML, errors are common, often resulting in strange layouts that are not at all as intended. Webson allows you to specify 3 different debug levels:

"#debug": 0 - no debugging output
"#debug": 1 - Show all #doc properties
"#debug": 2 - Show every item

which enables you to see what is happening. The output for the above is

Build $Left
The left-hand sidebar
#element: div
Attribute id: "left" -> left
Style display: "inline-block" -> inline-block
Style width: "25%" -> 25%
Style height: "100%" -> 100%
Style background: "green" -> green

This is a simple example where all values are constants. The values appear to be repeated but this will not always be the case. In more complex scripts you will often see the results of expressions being evaluated.

#doc items can be either single lines of text or arrays of lines. They are just there for the benefit of the programmer and have no effect on the screen being constructed.

A #debug directive affects its own block and those below it (defined using #).

Nested bocks

Webson implements nesting, whereby items declared at one level apply to all those in lower (contained) levels. Changing a value at one level only affects those at that level and beneath it; those above are unaffected.

For example, let's suppose the two sidebars share a common feature; they each have an inner div and padding to produce a border. Here's what it should look like:

To achieve this we can rewrite the last part of the script as follows:

    "$Left": {
        "#doc": "The left column",
        "$ID": "left",
        "$Width": "25%",
        "$Color": "green",
        "#": "$LRPanel"
    },

    "$Right": {
        "#doc": "The right column",
        "$ID": "right",
        "$Width": "15%",
        "$Color": "blue",
        "#": "$LRPanel"
    },

    "$LRPanel": {
        "#element": "div",
        "display": "inline-block",
        "width": "calc($Width - 2em)",
        "height": "calc(100% - 2em)",
        "padding": "1em",
        "#": "$LRSubPanel"
    },

    "$LRSubPanel": {
        "#element": "div",
        "@id": "$ID",
        "width": "100%",
        "height": "100%",
        "background": "$Color"
    }

Here I've left out the block for $Center as it's unchanged. Both $Left and $Right now no longer declare their own #element; instead they set up user-defined variables $ID, $Width and $Color and invoke $LRPanel to construct the element. I suggest using an initial capital letter for each user-defined name, to make them easier to spot, but it's not mandatory. Any variable declared or modified at a given level in the structure will be visible at all points beneath that one, but changes do not propagate upwards.

$LRPanel creates a div, applies padding to it and creates an inner div called $LRSubPanel. Note how the $Color variable is passed down and used here, resulting in a colored panel with a white border. Note also the use of calc() in $LRPanel to allow for the padding, which in a conformant browser adds to the width or height of the element.

How to run it

To view this demo on a PC, place the following HTML file on your server:

<html>
  <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Webson demo</title>
    <script type='text/javascript' src='resources/plugins/webson.js'></script>
  </head>

  <body>
    <div id="main" style="width:640px"></div>
    <script>
        window.onload = function() {
            render(`resources/json/simple.json`);

            async function render(file) {
                const response = await fetch(file);
                const script = await response.text();
                Webson.render(document.getElementById("main"), `keyboard`, script);
            }
        }
    </script>
  </body>
</html>

For mobile, the width can be set to `100%`. The JSON script is assumed to be in a folder on your server at

(your domain)/resources/json/simple.json

The code above uses the relatively-new standard function fetch() to get the named script from a file on the server. It then calls render() in the Webson package (webson.js in the repository) to create the DOM tree that corresponds to the JSON script.

From here on in

This has been a necessarily brief introduction to Webson, since to cover every feature in detail would result in a very lengthy article. A more in-depth treatment can be found in the Webson repository. The example used is the following on-screen virtual keyboard:

The repository documentation starts with the page you are reading now, then goes on to describe how to make the virtual keyboard depicted above. It then shows how to make the keyboard respond to its Shift and ?123 keys being tapped, to change the key legends appropriately. This is all done with simple JSON commands and no conventional coding at all. You can see and test-drive the virtual keyboard with the above functionality here.

Comments are welcome, as are suggestions on how to improve Webson.

DEV Community: Graham Trott

The Catch-22 of programming

Becoming a programmer

Programming with an LLM

The crux of the matter

What are our options?

An example

How does this work with AI?

Compilable pseudocode reduces complexity

How does this change the way we work?

New opportunities

Summary

OTA Python updates with ESP-Now

__init__()

handleMessage()

checkFile(()

Channel-hopping on the ESP32-C3 with ESP-Now

__init__()

setupSlaveTasks()

resetCounter()

findMyMaster()

countMissingMessages()

ping()

countMissingMessages()

checkRouterChannel()

ESP-Now master-slave in Micropython

__init__()

closeAP()

addPeer()

send() and receive()

Channel changes

An AI virtual keyboard

Using subnets to extend ESP-Now

Chained wi-fi networking for control systems

Background

Push or pull

Outline of the strategy

Implementation

Adding more devices

Over The Air (OTA) updates

Upsides

Downsides

The code

Unconfigured mode

Configured mode

The 'functions' modules

Update mode

Future work

Dynamic arrays in C++

It's in there somewhere

Setting up a low-cost server

A Smart Heating System

Introduction

The aims of the project

System description

How it works

The system controller

The user interface

DOM construction

Demo mode

Documentation

Resources

Webson: a new DOM markup

Synopsis

Introduction

Webson

Getting started

How it works

Attributes

Nested bocks

How to run it

From here on in

`init()`

`handleMessage()`

`checkFile(()`

`init()`

`setupSlaveTasks()`

`resetCounter()`

`findMyMaster()`

`countMissingMessages()`

`ping()`

`countMissingMessages()`

`checkRouterChannel()`

`init()`

`closeAP()`

`addPeer()`

`send()` and `receive()`