DEV Community: Chris Riddick

Golang - Unmarshal a JSON message with a multi-type property

Chris Riddick — Fri, 14 Jun 2024 19:04:12 +0000

Go provides a convenient library that makes parsing a JSON message simple. Just call the json.Unmarshal(msg, &parsedStructure). It takes the JSON msg and parses out the individual properties defined by the parsedStructure.

It works perfectly unless you have a property that is not well-behaved in the message. For example, in my case, the message from a weather station can be parsed into a structure called WeatherData. The one glitch in the parsing is that a property in the message does not conform to a single data type. Specifically, the Channel value can appear as a letter, e.g. 'A', or as a numeral. e.g., '1'. Since json.Unmarshal() uses the data types of the properties of the destination structure to determine how to parse the message, it can only handle one data type.

So, if I want the final result of the parsed message to include a string value for the Channel, it works fine until it encounters a message with a field like this: 'channel: 1'. Since it is expecting a string value for 'channel' in the WeatherData structure, it fails when it sees the numeral '1' instead of "A".

How do we deal with exceptions like that? The Go JSON library includes an interface for UnmarshalJSON() that allows you to create a dedicated function to handle the case of a special type. Unfortunately, it can only be applied to a structure as a method.

To make it work, I created a special structure called CustomChannel that has only one property, Channel as a string. Then, I wrote a new UnmarshalJSON() function per the interface that will handle instances of Channel as string and Channel as int. The json.Unmarshal() function invokes the interface CustomChannel function when it gets to the Channel property rather than trying to parse it simply as a string. When my custom UnmarshalJSON function returns, it has placed an appropriately converted integer to a string in the case of an int value, or passes back the string, if that was in the original message.

Since I want to work with a string value for Channel, I created a separate structure, WeatherDataRaw, for the raw parsed message with the CustomChannel structure, and a final structure WeatherData that I will work with in the program to write to a file or a database.

Code snippets are shown below of the structures and message handling code. You can see that the function handling an individual message calls json.Unmarshal(), but then the interface function is activated to handle the CustomChannel property. A helper function retrieves the string value of Channel from the WeatherDataRaw structure once it is processed so it can be stored in the final WeatherData structure.

    incoming WeatherDataRaw
    outgoing WeatherData
)

type WeatherDataRaw struct {
    Time          string        `json:"time"`          //"2024-06-11 10:33:52"
    Model         string        `json:"model"`         //"Acurite-5n1"
    Message_type  int           `json:"message_type"`  //56
    Id            int           `json:"id"`            //1997
    Channel       CustomChannel `json:"channel"`       //"A" or 1
    Sequence_num  int           `json:"sequence_num"`  //0
    Battery_ok    int           `json:"battery_ok"`    //1
    Wind_avg_mi_h float64       `json:"wind_avg_mi_h"` //4.73634
    Temperature_F float64       `json:"temperature_F"` //69.4
    Humidity      float64       `json:"humidity"`      // Can appear as integer or a decimal value
    Mic           string        `json:"mic"`           //"CHECKSUM"
}

type CustomChannel struct {
    Channel string
}

func (cc *CustomChannel) channel() string {
    return cc.Channel
}

type WeatherData struct {
    Time          string  `json:"time"`          //"2024-06-11 10:33:52"
    Model         string  `json:"model"`         //"Acurite-5n1"
    Message_type  int     `json:"message_type"`  //56
    Id            int     `json:"id"`            //1997
    Channel       string  `json:"channel"`       //"A" or 1
    Sequence_num  int     `json:"sequence_num"`  //0
    Battery_ok    int     `json:"battery_ok"`    //1
    Wind_avg_mi_h float64 `json:"wind_avg_mi_h"` //4.73634
    Temperature_F float64 `json:"temperature_F"` //69.4
    Humidity      float64 `json:"humidity"`      // Can appear as integer or a decimal value
    Mic           string  `json:"mic"`           //"CHECKSUM"
}

var messageHandler1 mqtt.MessageHandler = func(client mqtt.Client, msg mqtt.Message) {
    log.Printf("Received message: %s from topic: %s\n", msg.Payload(), msg.Topic())
    // Sometimes, JSON for channel returns an integer instead of a letter. Check and convert to string.
    err := json.Unmarshal(msg.Payload(), &incoming)
    if err != nil {
        log.Fatalf("Unable to unmarshal JSON due to %s", err)
    }
    copyWDRtoWD()
    printWeatherData(outgoing, "home")
}

I'm sure there are a dozen other ways to achieve this, but after spending hours reading the library descriptions and postings online, this was the method that made the most sense to me.

The code is posted on github for a weather dashboard project I'm working on. Feel free to check it out and comment. It is still in the early stages and no GUI has been implemented yet, but this is a side project that will progress over time.

Learning Go, Building a File Picker using Fyne.io

Chris Riddick — Thu, 30 May 2024 15:47:18 +0000

Part 3 - Using fyne.io to select files

Go has an io library that enables a developer to access the host file system. Building a GUI application that interacts with the native file system requires the developer to try to make the user experience the same, or similar, across platforms. We want a user to be able to work with the application without having to learn multiple ways to respond to application prompts to open files. Fortunately, fyne.io provides a fairly robust cross-platform toolset with which to accomplish this task.

Background and Review

In my journey learning Go, I’ve structured my efforts around an actual programming challenge. My longtime interest in CPU design and instruction sets led me to base my learning around a CPU simulator. Specifically, I set myself the problem of adding a GUI front end to an existing terminal-based CPU simulator.

I started my project with an imaginary CPU I found on GitHub. That simple instruction set helped me get familiar with the basics of Go and fyne.io. Once I had the base dashboard developed, I searched for a simulator for a more complex CPU. Fortunately, there was a comprehensive simulator for the 6502 8-bit processor with associated assembler and disassembler that has been actively maintained. The 6502 was the processor on which the original Apple computer was based. I forked a copy of that and continued my learning. A summary of that effort is described in my second post here.

Using the new fork of a 6502 simulator, I copied over my dashboard package and began studying the code. I don't know how other developers study and learn someone else's code, but my approach has been to first scan the entire repository and get a brief idea of the structure and components of the application. I review the README, if there is one, and then I try to compile and run the application. I've found that getting the environment properly configured and correcting missing dependencies, through often frustrating, forces me to learn a lot about the application and how it is built.

If you are lucky, the original developer will have left a few test cases to run. These definitely help in understanding how to use the various packages in the application. If you aren't that lucky, then start at the top and find the main.go package and follow how it sets up and executes the rest of the application.

In the case of the 6502 simulator, I found the CPU and Host structures, the methods that operate on them, and focused on how the elements of the CPU were specified and stored. After some study, I learned that the Host object and associated methods were called upon from a terminal session to invoke individual instructions and return results. I just had to simulate the commands to the CPU that would be sent from the terminal and generate them using GUI buttons. Output was redirected from stdout to my GUI scrolling component. The well-designed 6502 simulator made it relatively easy to integrate my dashboard.

The only thing missing from the updated GUI-based application was a means for the user to select files to compile and load rather than have to enter them on the command line. That is where the file picker project starts.

The File Picker

Fyne provides a Dialog package for use in creating various dialogs a user may need to conduct with an application. One specific to our needs is the FileDialog. It opens a dialog with the user allowing navigation through the local file system to select a specific file. On pressing Save, that file info is provided to the callback function that the developer specifies in the call to the dialog. If there is no error, then the application can use the resulting file URI in further processing.

Let's look at the details of what happens. I wrote a function in my dashboard I called showFilePicker(w Fyne.Window). It takes as an argument the Fyne.Window object used by the dashboard. I created a File button on the dashboard and associated showFilePicker(w Fyne.Window) as the callback function the window invokes whenever the File button is pressed. I also created a Label widget to display and contain the selected file and path. I also save the URI returned by the picker for later use.

var fileButton            *widget.Button
var selectedFile          *widget.Label
var fileURI               fyne.URI

fileButton = widget.NewButton("File", func() { showFilePicker(w) })

When the user presses the File button, showFilePicker is activated and immediately invokes the Fyne dialog.ShowFileOpen method. You must import the dialog package at the head of your file.

import (
    "fyne.io/fyne/v2/dialog"
)

//...

// Show file picker and return selected file
func showFilePicker(w fyne.Window) {
    dialog.ShowFileOpen(func(f fyne.URIReadCloser, err error) {
        saveFile := "NoFileYet"
        if err != nil {
            dialog.ShowError(err, w)
            return
        }
        if f == nil {
            return
        }
        saveFile = f.URI().Path()
        fileURI = f.URI()
        selectedFile.SetText(saveFile)
    }, w)
}

The Fyne window invokes showFilePicker() when the File button is pressed. When showFilePicker is running, the rest of the GUI is blocked until it returns. showFilePicker starts a modal dialog by calling dialog.ShowFileOpen. As a modal dialog, the dashboard is blocked until it returns. ShowFileOpen returns a Fyne.URIReadCloser, which has numerous useful methods, and an error object. If error is nil, then we can process the selected file object.

I place the file path into the label widget by calling selectedFile.SetText(saveFile). I also preserve the returned URI in a variable I prepared called fileURI. I will use that URI to determine what type of file has been selected. Is it a .asm, assembly language file, or a .bin, assembled 6502 binary file? The Assemble and Load button callback functions use the URI to check the extension before trying to complete their actions.

Conclusions

Sure, this is a primitive file selector. It doesn't maintain state to know where we are in a specific directory and to start there instead of the user's home. However, because it is modal and blocking the GUI, we want to handle the file selection as efficiently as possible and return control to the window so it can respond to other buttons and actions.

So, that is all for now. I'd like to enhance the picker to enable selection of multiple files so we can assemble a batch at once. Likewise, loading multiple binaries into different areas of memory could be quite handy.

Please feel free to comment and suggest improvements or other topics of interest to a learner of Go and GUI development. Thanks for reading.

My Experience Learning Go, next steps

Chris Riddick — Sun, 26 May 2024 15:16:52 +0000

I recently posted about my efforts to learn Go and GUI development in Go here. This post is about the next steps in my learning process.

I was able to get a basic dashboard implemented and functioning using Fyne.io and a simple CPU simulator. You can see that in my git repo.

My next step was to find an existing CPU simulator implemented in Go and integrate my dashboard with it. Reading and understanding someone else's code is one of the best ways to learn a new language. Not only must you grasp the syntax and semantics of the language, but you also must decipher and learn the idioms and patterns used by the other programmer. Programming is as much an art as it is engineering, and reading and following someone else's coding is, to me, like walking through an art gallery and appreciating the nuances and techniques applied by each artist.

Much to my delight, I discovered a complete implementation of a 6502 CPU simulator written in Go. Not only did the project include a CPU simulator, but it included an assembler, disassembler, and utilities! The project I found, go6502, is so comprehensive that after only a few days working my way through the code, I was able to understand how almost everything worked together. The project is so well-done that I was able to integrate my dashboard without breaking any of the existing simulator code.

When a software engineer encounters code that is so well-written that it makes adding features a breeze, you have to tip your hat to the engineer for doing such a fine job. That is how I found this project. It was complex, but well-designed and built for flexibility. I was able to learn many new things about Go that were not obvious from reading the manuals and tutorials.

Details

I created a fork of the go6502 project. Then, I added a folder for my dashboard package. I worked through updates to the go.mod files to make sure that I was pulling from my repo and not the original repo. That is one of the beauties of Go. Dependencies are much easier to work with than when I was doing Java development.

The primary hooks into the existing go6502 simulator were the bufio.Writer for terminal output and the Host object with a structure and associated functions to take commands from the GUI buttons. Using the Fyne widgets and callback functions, I was able to insert calls to the Host object, passing in command strings instead of sending them to the terminal. Output is redirected to a scrolling widget in the dashboard. New commands, replacing a command line function, are submitted using a data entry widget and submit button. The user determines whether or not to use the GUI at time of program startup by including a flag (-g) in the command. If the flag is absent, the program starts up in its usual terminal mode. If the flag is included, the dashboard is started and the program control is handed off to the GUI. Termination of the program is by an Exit button instead of a command line quit statement.

Sometimes it is hard to know if a program is still running or if it is hung up in an error loop. So, I added a goroutine that updates a clock display every second. That way a user can see the program is still running and not frozen. It also helps someone understand how goroutines and channels make concurrent processing in Go easy.

There is still much to learn from this. I need to add a file picker to allow a user to select a file from the file system to assemble and load. Currently, the program looks for a sample file in the current directory to load. I want to learn if I can use the terminal themes that Vickers incorporated in the terminal code to customize the look of the GUI dashboard. In many ways, Fyne.io is comprehensive and cross-platform, but it is new enough that there isn't a lot of examples to use on which to build your own code. Next steps will explore this further.

Finally, I plan to experiment with the go6502 code to see if I can create my own instruction set and replace the 6502 with a different CPU. The Vickers code seems designed for that purpose. Learning Go this way has been a blast!

How One Experienced Software Engineer Learns a New Programming Language

Chris Riddick — Thu, 16 May 2024 11:25:24 +0000

I retired two years ago from a software engineering career that started in 1979, but have never lost the joy of programming and building useful applications. I recently decided to challenge myself to learn a new programming language just for the fun of it. My most recent experience had been with Java and Linux doing big data processing.

Java is a cool language. It can do most anything. it is also large and complex. It can take years to master even the most basic features. It is also fully object-oriented with all of the baggage that goes with OO. My first job years ago was writing assembly language programs for Z80 on a custom data collection system. We were doing FFTs and other digital signal processing with only the Z80 and a multiplier chip from Texas Instruments. I loved it!

I want to relive that experience with a new language. I've used C, Objective-C, FORTRAN, Java, Python, and dozens of other specialized languages. What I really enjoyed about assembler and the C language was the simplicity and purity of the instruction set and features. Was there something that I could learn that got me back to my roots and closer to the basics?

I learn best when I'm building something useful, not just working tutorials or examples. I wanted to find a new language to learn that would enable me to build something other people might actually find useful or fun -- maybe something to help others to learn, too. Working my way down the list of popular languages, I narrowed my choices down to two: Rust and Go. These are languages people are using to build applications today. Rust is even making inroads into the Linux core and kernel, showing off its strength at building complex machine-level features. Go, or Golang as it is often known, is very C-like in that is doesn't have a lot of complex language features. Instead, it gives programmers enough tools to build anything, without overwhelming them with complexity. Unlike C, Go relieves us of memory management for garbage collection and helps avoid other issues related to data types and error handling. Surprisingly, Go has built-in testing and logging, too!

I decided on Go as my new language. I felt it was simple enough that I could learn the basics and begin building useful applications in a short amount of time. I also liked that it has a well-developed toolset for compiling and managing dependencies. If you've ever struggled with Maven or Gradle for Java, you know what I'm talking about.

I wanted to learn Go by designing and building an application that others may find useful. I've always had an interest in CPU (Central Processing Unit) design and instruction sets. I'm dating myself, but during my college days, I built an RCA Cosmac Elf from a Popular Electronics article. It was designed around the RCA 1802 8-bit microprocessor and 256 bytes of RAM. I was able to program it using eight switches and a button to input a machine language program that turned an LED on and off. You can't get any more basic than that.

Learning the RCA 1802, Z80, and other processors, I felt that I had a solid understanding of what was happening under the cover of any computer. It enabled me to be more successful in designing and building applications. It also helped me to debug and troubleshoot because I knew it was not "magic" behind the code, but very basic instructions you could follow. After working on a project that used a Motorola 68000 16-bit processor to simulate a Zilog Z80 8-bit processor, I realized way back then that you can simulate anything in a computer if you understand it and have the right tools.

I decided my learning project would be to develop a processor simulator in Go that could be used to teach the basics of CPU architecture and instruction sets. As an experience engineer, I wasn't trying to reinvent the wheel. There are thousands of open-source processor simulators out there. I decided to find an existing project in the public domain that was simple enough that I could take it and recode it to run in Go. I also wanted to add a graphical user interface (GUI) that would make it easy for a user to see the operation of the CPU instructions and to write simple machine-language programs to test ideas.

My approach to learning Go was first to read The Go Programming Language by Donovan and Kernighan and then peruse numerous resources like the website go.dev. Once I had a basic familiarity with the syntax and features of the language, I searched for code examples to see how other developers used the language. Since my interest was in CPU simulation, I searched for those examples.

Google and github have a wealth of information, and I quickly found a project that met my needs. The one I found was already written in Go, so I could see how a Go program is actually constructed. It was ritten by an engineer (Wojciech S. Gac) for a company competition . He posted his code on github here go-cpu-simulator. It had no GUI, so it gave me an opportunity to take someone else's work and add on a GUI written in Go. The project I selected was a simple CPU simulator with just eight instructions. It was a very basic CPU, but didn't require digging too deeply into someone else's code to figure out how to use it. I just needed to figure out how to add a useful GUI front-end to drive the simulator as an educational tool.

Reading someone else's code is not always easy. It helps if you can find a complete application to build and run while studying the code. Using Wojtek's simulator, I was able to see many of the features of Go in action.

Go is very C-like in the syntax. Like C, Go is simple and clean. There is not a lot of fluff in the language. Scope tends to be one area that causes confusion and trouble when learning a new language. Go is fairly straightforward in this regard. Modules and packages are clear concepts to grasp. If you want a variable or function to be visible outside a package, you capitalize the name. The Go language tools help you avoid scope errors by immediately warning you of names out of scope and unused variables and functions. Like C, Go uses structures. Go adds some basic object-oriented (OO) features by allowing methods on structures. Although not fully OO, Go's use of structures and methods expands the options of a programmer when designing application code.

Goroutines and channels give you simple, built-in concurrent processing without having to drop down into the operating system threading and processes.

Once I gained some familiarity with the simple CPU simulator by Wojtek, I began researching cross-platform graphical user interface frameworks for Go. My objective was to strap on a GUI front-end to his simulator so I could watch the simulator operate in real-time.

The GUI challenge

With a little research, I found a GUI framework that runs under Linux, macOS, and Windows. I selected the open source fyne.io framework, itself written in Go. It is cross-platform and has interfaces with Linux, Mac, and Windows so that any code I develop can be compiled for all three platforms for similar GUI results. It has enough capability that I felt I could create the GUI dashboard for running the simulator.

I wanted to build the dashboard in a way that could be generalized and reused for other CPU simulators I might want to build. Therefore, I wanted the dashboard to be its own module and package, separate from the simulator code. I used a Model-View-Controller pattern for my project. The Wojtek simulator was a single Go file. I created separate packages for the CPU (model), the dashboard (view), and the main program (controller). Using Go's module feature to manage external dependencies, I set the file structure with the main package and its go.mod dependency file and two folders, one for the dashboard package and the other for the cpu model package. Each folder represents its own module and has its own go.mod file. These can all be seen in my github repo.

The process of separating the program into three packages of main, dashboard, and simplecpu forced me to understand the fundamentals of Go scope and naming. It also clarified for me how the dependency system worked. My most recent experience has been with Java and Maven. I found the Go modules and dependency system simpler to work with compared to the complexity of Maven or Gradle.

The CPU monitor dashboard layout was fairly straightforward using the fyne.io framework. Like most GUIs, you create all your display objects and widgets, add containers for structuring the objects in columns, rows, and grids, and then place the containers into a window. I set up some control buttons with associated functions that get invoked when they are pressed. I also set up some label widgets to display specific CPU fields and data. I decided that it would be simpler for displaying memory if I pre-formatted memory in blocks of strings before placing them in containers. I created an UpdateAll() function that the controller called whenever it had new data to display.

The main package calls a dashboard.New() function from the dashboard package to create a new dashboard with all of its widgets and containers. The dashboard.New() function returns a pointer to a Fyne window object which the main() function owns. The call to the dashboard.New() function passes into the dashboard package a pointer to the CPU structure and pointers to all of the callback functions that the dashboard should run when buttons are pressed. Thus, the main package can control how the GUI responds to user input.

One thing I quickly learned about Go and threads is that as soon as the window object is created and displayed, it is operating independently from the main package. The button callback functions are how the GUI handles user input. A user presses the Run button and the CPU simulator begins executing instructions in the callback function. The callback function does not return to the dashboard window until the function ends. If a user tries pressing another button, like Pause, there is no response from the dashboard because it is blocked until the callback function for the Run button returns. I needed some way to allow user input while a CPU simulation is in progress. That's where Goroutines and channels come into play.

To prevent the dashboard from blocking all user input when a function is running, I need to give the button a way to run concurrently with the dashboard. The solution is to invoke the actual CPU execution using a Goroutine call in the callback function. As soon as the go Run() call is made, the code kicks off a separate Go thread running that function, returning control back to the main program and its dashboard.

Now, how do I let the program know when the Run() function is finished? I use a channel. Channels are simple, program-level, message-passing services that support concurrency. I set up a channel for each Goroutine, e.g., the Run() callback function, to send a message to a queue when it finishes. I set up a channel monitor function that runs as a Goroutine on the main program thread that watches for a message on that channel. When the message is received, the monitor function completes the processing necessary for the main thread to handle the results of the simulation. I end up with monitors for each button that could be pressed by a user so that the button can take as long as it needs to complete its task without blocking the dashboard from further user input.

Here is a screenshot of the dashboard running a simple program loaded in memory at address x0000.

Expanding the CPU functionality and learning Go types and operations

Since the basic simulator was created to execute a limited set of seven instructions as part of a programming contest, it didn't spend a lot of time worrying about internal data formats and manipulation. The original CPU had a set of 17 integer (int) registers and a separate integer (int) stack array. Programs were byte (uint8) arrays. To run a program, the controller had to send a pointer to the code array to the processor and tell it how long the program was (length of the array). All operations were in native integer format.

When I built the dashboard and began monitoring execution of the sample programs, I realized that the simulator was not handling arithmetic the way a hardware CPU would have handled it. We had a mix of bytes and native integer format. Integer on some machines is signed, 64-bit. That's not what we really want to be simulating. We are simulating 16-bit registers operating on 8-bit bytes from a memory array.

I refactored the simplecpu package to use uint8 and uint16 variables to restrict it to what a simple hardware CPU might be doing. I moved the programs into memory as a sequence of bytes, and memory itself is simply an array of bytes (Big-Endian and Little-Endian storage is an important element in processor design). Rather than being a separate integer array, the stack is now a part of the memory, located by the 16-bit stack pointer. Programs execute beginning at wherever the program counter is pointing. Programs either run over the end of the memory array generating a memory fault, or they execute a HALT instruction to stop CPU processing. Now, the CPU simulator looks and acts like a basic 8-bit microprocessor from the 1970's. The dashboard displays the contents of memory and registers as the CPU executions instructions, and the user can observe how machine code runs. [Go arrays and slices are very important!]

The fyne.io framework provides an API for data binding so that data associated with a widget is automatically updated in the container without requiring the programmer to watch for changes and call update routines. Unfortunately, this application uses some more limited data types like uint8 and uint16 which are not currently supported by fyne.io. Thus, I had to use my own update function whenever a data field changed.

Another change I made was to the CPU package. As originally designed, a CPU instruction was completely contained in an 8-bit byte with the upper three bits defining the operation and the lower five bits as the operand. I wanted to be able to simulate a full 8-bit memory array with corresponding addressing, subroutines, and comparisons. To do this, I added an extended instruction set to the basic CPU, giving the simulator a more advanced programming capability while maintaining backward compatibility with the original seven instruction CPU. This is similar to how the original microprocessors like the 8008 and 8080 implemented backward instruction compatibility as technology progressed.

Simple CPU Simulator is far from finished. There are still some quirks in the GUI. I'm not sure I've mastered the concurrency feature using channels. I broke the go tests when implementing a clock-based execution loop. Programs have to be loaded by storing them in the code itself. It needs a feature to read an executable binary from a file.

This simple CPU simulator can be the basis for learning how real microprocessors operate and can help programmers better understand the internals of a CPU, the basics of machine-level arithmetic, and the use of memory in computers. Understanding the internals of the machine you program can give you better insights into its capabilities and limitations. it can also help you build more efficient applications.

Stay tuned for more as I integrate RCA 1802 and Z80 CPUs with the dashboard in future projects while continuing to learn Go and Fyne.io.

Project Summary

My code is maintained using github and is located here https://github.com/cjr29/go-cpu-simulator.git. This is a fork of Wojtek's repo. Wojtek is aware of my GUI and other enhancements. My project will be moving in other directions as I continue my learning, but I want to be sure he gets credit for providing a solid foundation for starting my learning.

Thanks for reading my post. Please let me know if this was helpful in the comments section. I'm happy to answer any questions.