DEV Community: Jake Fenton

Codemash 2019 Reading List

Jake Fenton — Tue, 15 Jan 2019 13:39:52 +0000

Last updated: 2019-01-14 4pm EST

Hello! This is a compiled list of all the books mentioned during talks given at Codemash 2019.

This list has been compiled from slide decks linked from this google sheet and from my own notes on given talks.

I apologize if I missed books from your talk or made an error in copying them down--Please let me know and I'll update the list or make corrections!

The list should be in alphabetical order by talk title. Nearly all the links are for purchasing the books off of Amazon, unless they weren't available or were free through other sources.

I did not link additional resources--papers, news articles, YouTube videos, etc.

Also, fair warning, a couple of the books in talks I went to were mentioned in jest. I tried to note where this was the case, but I didn't attend other talks, so it's possible the same happened elsewhere.

A Better Faster Pipeline for Software Delivery

"Lean Thinking" - James P. Womack, Daniel T. Jones amazon link

Expanded, updated, and more relevant than ever, this bestselling business classic by two internationally renowned management analysts describes a business system for the twenty-first century that supersedes the mass production system of Ford, the financial control system of Sloan, and the strategic system of Welch and GE. It is based on the Toyota (lean) model, which combines operational excellence with value-based strategies to produce steady growth through a wide range of economic conditions.

"The Phoenix Project: A Novel about IT, DevOps, and Helping your Business Win" - Gene Kim, Kevin Behr, George Spafford amazon link

Note: fiction, but a good example of how to turn a team (or company) onto using lean principles

Avoiding Landmines: A Tech LEader's guide for the Critical Decisions

"Thinking Fast and Slow" -- Daniel Kahneman amazon link

"...a groundbreaking tour of the mind and explains the two systems that drive the way we think. System 1 is fast, intuitive, and emotional; System 2 is slower, more deliberative, and more logical."

"Drive: The Surprising Truth About What Motivates Us" - Daniel H. Pink amazon link

"...the secret to high performance and satisfaction-at work, at school, and at home—is the deeply human need to direct our own lives, to learn and create new things, and to do better by ourselves and our world."

"Cultues and Organizations: Software of the Mind" - Geert Hofstede, Gert Jan Hofstede, Michael Minkov amazon link

"...Cultures and Organizations examines what drives people apart―when cooperation is so clearly in everyone’s interest. "

"The Advantage: Why Organizational Health Trumps Everything Else in Business" - Patric M. Lencioni amazon link

"...Patrick Lencioni, argues that the seminal difference between successful companies and mediocre ones has little to do with what they know and how smart they are and more to do with how healthy they are."

Better Names for Better Code

"Clean Code: A Handbook of Agile Software Craftsmanship" -- Robert C. Martin amazon link

Even bad code can function. But if code isn’t clean, it can bring a development organization to its knees. Every year, countless hours and significant resources are lost because of poorly written code. But it doesn’t have to be that way.

"Structure and Interpretation of Computer Programs" - Harold Abelson, Gerald Jay Sussman, Julie Sussman amazon link

This long-awaited revision contains changes throughout the text. There are new implementations of most of the major programming systems in the book, including the interpreters and compilers, and the authors have incorporated many small changes that reflect their experience teaching the course at MIT since the first edition was published. A new theme has been introduced that emphasizes the central role played by different approaches to dealing with time in computational models: objects with state, concurrent programming, functional programming and lazy evaluation, and nondeterministic programming.

Code Reviews?! That's a great idea!

"Code Complete: A practical handbook of Software Construction" -- Steve McConnel amazon link

quoted in a talk:* "effectiveness of code review for determining faults in software is between 30% and 35% more effective than standard unit testing"
"Widely considered one of the best practical guides to programming, Steve McConnell’s original CODE COMPLETE has been helping developers write better software for more than a decade. Now this classic book has been fully updated and revised with leading-edge practices—and hundreds of new code samples—illustrating the art and science of software construction."

Don't Rewrite, React!

"Learning React" -- Kirupa Chinnathambi amazon link

"A hands-on guide to building maintainable, high-performing web application user interfaces using the React JavaScript library"

"React and React Native" -- Adam Boduch amazon link

"Build React and React Native applications using familiar component concepts"
"Dive deep into each platform, from routing in React to creating native mobile applications that can run offline"
"Use Facebook's Relay, React and GraphQL technologies, to create a unified architecture that powers both web and native applications"

Embrace your Legacy... Code!

"Working Effectively with Legacy Code" -- Michael C. Feathers amazon link

"In this book, Michael Feathers offers start-to-finish strategies for working more effectively with large, untested legacy code bases. This book draws on material Michael created for his own renowned Object Mentor seminars: techniques Michael has used in mentoring to help hundreds of developers, technical managers, and testers bring their legacy systems under control. This book also includes a catalog of twenty-four dependency-breaking techniques that help you work with program elements in isolation and make safer changes."

"Refactoring: Improving the Design of Existing Code" -- Martin Fowler, Kent Beck, John Brant, William Opdyke, Don Roberts, Erich Gamma amazon link

For several years, expert-level object programmers have employed a growing collection of techniques to improve the structural integrity and performance of such existing software programs. Referred to as refactoring, these practices have remained in the domain of experts because no attempt has been made to transcribe the lore into a form that all developers could use. . .until now. In Refactoring: Improving the Design of Existing Software, renowned object technology mentor Martin Fowler breaks new ground, demystifying these master practices and demonstrating how software practitioners can realize the significant benefits of this new process.

Getting Unstuck

"Creativity: Flow and the Psychology of Discovery and Intervention" - Mihaly Csikszentmihalyi amazon link

"Drawing on nearly one hundred interviews with exceptional people, from biologists and physicists, to politicians and business leaders, to poets and artists, as well as his thirty years of research on the subject, Csikszentmihalyi uses his famous flow theory to explore the creative process."
Note: At least three talks have referenced this book

"The Art of Thought" -- Grahm Wallas amazon link

Amazon Note: Available at a lower price from other sellers that may not offer free Prime shipping.

Keeping Up With Technology

"Thinking in Systems" -- Donella H. Meadows amazon link

Thinking in Systems, is a concise and crucial book offering insight for problem solving on scales ranging from the personal to the global. Edited by the Sustainability Institute’s Diana Wright, this essential primer brings systems thinking out of the realm of computers and equations and into the tangible world, showing readers how to develop the systems-thinking skills that thought leaders across the globe consider critical for 21st-century life.

"Win Friends and Influence People" -- Dale Carnegie amazon link

Amazon Note: Available at a lower price from other sellers that may not offer free Prime shipping.
Dale Carnegie’s rock-solid, time-tested advice has carried countless people up the ladder of success in their business and personal lives. One of the most groundbreaking and timeless bestsellers of all time, How to Win Friends & Influence People will teach you:
- Six ways to make people like you
- Twelve ways to win people to your way of thinking
- Nine ways to change people without arousing resentment

"How to Have Confidence and Power in Dealing With People" -- Les Giblin amazon link

Les Giblin, a recognized expert in the field of human relations, has devised a method for dealing with people that can be used when relating with anyone – parents, teachers, bosses, employees, friends, acquaintances, even strangers. Giblin shows step by step how to get what you want at any time and in ways that leave you feeling good about yourself. Moreover, the people who have given you want you want wind up feeling good about themselves, too. The result? Nobody gets shortchanged. It’s a win-win situation.

"Crucial Conversations: Tools for talking when the stakes are high" -- Kerry Patterson, Joseph Grenny, Ron McMillan, Al Switzler amazon link

The first edition of Crucial Conversations exploded onto the scene and revolutionized the way millions of people communicate when stakes are high. This new edition gives you the tools to:
- Prepare for high-stakes situations
- Transform anger and hurt feelings into powerful dialogue
- Make it safe to talk about almost anything
- Be persuasive, not abrasive

Leadership Guide for the Reluctant Leader

"The Ideal Team Player: How to Recognize and Cultivate the Three Essential Virtues" -- Patrick M Lencioni amazon link

In his classic book, The Five Dysfunctions of a Team, Patrick Lencioni laid out a groundbreaking approach for tackling the perilous group behaviors that destroy teamwork. Here he turns his focus to the individual, revealing the three indispensable virtues of an ideal team player.

"StrengthsFinder 2.0" -- Tom Rath amazon link

In its latest national bestseller, StrengthsFinder 2.0, Gallup unveils the new and improved version of its popular assessment, language of 34 themes, and much more (see below for details). While you can read this book in one sitting, you'll use it as a reference for decades.

Learning the Three Types of Microservices

"Release It: Design and Deploy Production Ready Software" - Michael T. Nygard amazon link

If you're a developer and don't want to be on call for 3AM for the rest of your life, this book will help. In Release It!, Michael T. Nygard shows you how to design and architect your application for the harsh realities it will face. You'll learn how to design your application for maximum uptime, performance, and return on investment.

Management 101

"Rules of the Red Rubber Ball" -- Kevin Carroll amazon link

With simple but delightful storytelling, Kevin Carroll channels his childhood passion for sport and play into a universally appealing blueprint for life. Drawing wisdom from the playgrounds of his youth, where he spent hour upon hour sharpening his body and his mind, Carroll shares with readers his Rules of the Red Rubber Ball - how to achieve maximum human potential through the power of passion and creativity. Finding your own -red rubber ball+ and chasing it to your heart+s content, he argues, is the surest route to peace, prosperity, and happiness. Over the years as an athletic trainer and public speaker, Carroll has transformed his philosophy into seven simple rules that any successful leader will endorse: 1) Commit to it 2) Seek out encouragers 3) Work out your creative muscle 4) Prepare to shine 5) Speak up 6) Expect the unexpected 7) Maximize the day With an award-winning design and color photos throughout
Rules of the Red Rubber Ball will inspire the child in everyone for generations to come.

Modern Identity Management

"JWT Handbook" -- Sebastian Peyrott free ebook link

Ever wondered how JWT came to be and what problems it was designed to tackle? Are you curious about the plethora of algorithms available for signing and encrypting JWTs? Or are you interested in getting up-to-speed with JWTs as soon as possible? Then this handbook is for you.

The Agile DBA

"The Dev Ops Handbook" -- Gene Kim, John Willis, Jez Humble, John Allspaw amazon link

Following in the footsteps of The Phoenix Project, The DevOps Handbook shows leaders how to replicate these incredible outcomes, by showing how to integrate Product Management, Development, QA, IT Operations, and Information Security to elevate your company and win in the marketplace.

"The Phoenix Project: A Novel about IT, DevOps, and Helping your Business Win" - Gene Kim, Kevin Behr, George Spafford amazon link

Note: fiction, but a good example of how to turn a team (or company) onto using lean principles ## The Two Question Programming Interview

"How would you move mount fuji?" -- William Poundstone amazon link

note: this book was a tongue-in-cheek mention for ways to not ask programming questions. Direct quote: "[the problems from this book] are great for dinner parties, but not really for programming interviews. Instead of teaching to think outside-the-box, they teach you to think inside of a slightly-larger-but-very-specific box."
"William Poundstone reveals the toughest questions used at Microsoft and other Fortune 500 companies -- and supplies the answers."

"Never Split the Difference: Negotiating As If Your Life Depended On It" - Chris Voss amazon link

" Reaching the pinnacle of his profession, he became the FBI’s lead international kidnapping negotiator. Never Split the Difference takes you inside the world of high-stakes negotiations and into Voss’s head, revealing the skills that helped him and his colleagues succeed where it mattered most: saving lives. In this practical guide, he shares the nine effective principles—counterintuitive tactics and strategies—you too can use to become more persuasive in both your professional and personal life."

"Smart and Gets Things Done: Joel Spolsky's concise Guide to Finding The Best Technical Talent" -- Avram Joel Spolsky amazon link

"The secret to success for any software company then is to hire the good programmers. But how to do that? In Joel on Hiring, Joel Spolsky draws from his experience both at Microsoft and running his own successful software company based in New York City. He writes humorously, but seriously about his methods for sorting resumes, for finding great candidates, and for interviewing, in person and by phone."

User-story driven thread modeling

"Threat Modeling: Designing for Security" -- Adam Shostack amazon link

Adam Shostack is responsible for security development lifecycle threat modeling at Microsoft and is one of a handful of threat modeling experts in the world. Now, he is sharing his considerable expertise into this unique book. With pages of specific actionable advice, he details how to build better security into the design of systems, software, or services from the outset. You'll explore various threat modeling approaches, find out how to test your designs against threats, and learn effective ways to address threats that have been validated at Microsoft and other top companies.

"Securing Systems: Applied Architecture and Threat Models" -- Brook S.E. Schoenfield amazon link

Internet attack on computer systems is pervasive. It can take from less than a minute to as much as eight hours for an unprotected machine connected to the Internet to be completely compromised. It is the information security architect’s job to prevent attacks by securing computer systems. This book describes both the process and the practice of assessing a computer system’s existing information security posture. Detailing the time-tested practices of experienced security architects, it explains how to deliver the right security at the right time in the implementation lifecycle.

"Risk Centric Threat Modeling: Process for Attack Simulation and Threat Analysis" -- Marco Morana and Tony UcadeVelez amazon link

This book introduces the Process for Attack Simulation & Threat Analysis (PASTA) threat modeling methodology. It provides an introduction to various types of application threat modeling and introduces a risk-centric methodology aimed at applying security countermeasures that are commensurate to the possible impact that could be sustained from defined threat models, vulnerabilities, weaknesses, and attack patterns.

"Measuring and managing Information Risk: A FAIR Approach" -- Jack Jones and Jack Freund amazon link

Using the factor analysis of information risk (FAIR) methodology developed over ten years and adopted by corporations worldwide, Measuring and Managing Information Risk provides a proven and credible framework for understanding, measuring, and analyzing information risk of any size or complexity. Intended for organizations that need to either build a risk management program from the ground up or strengthen an existing one, this book provides a unique and fresh perspective on how to do a basic quantitative risk analysis. Covering such key areas as risk theory, risk calculation, scenario modeling, and communicating risk within the organization, Measuring and Managing Information Risk helps managers make better business decisions by understanding their organizational risk.

What are Observables and why should I care?

"Build Reactive Websites with RxJS" -- Randall Koutnik the pragmatic bookshelf link

Upgrade your skill set, succeed at work, and above all, avoid the many headaches that come with modern front-end development. Simplify your codebase with hands-on examples pulled from real-life applications. Master the mysteries of asynchronous state management, detangle puzzling race conditions, and send spaceships soaring through the cosmos. When you finish this book, you’ll be able to tame the wild codebeasts before they ever get a chance to wreck your day.

What Every Beginning Developer Should Know

"Code Complete: A practical handbook of Software Construction" -- Steve McConnel amazon link

quoted in a talk:* "effectiveness of code review for determining faults in software is between 30% and 35% more effective than standard unit testing"
"Widely considered one of the best practical guides to programming, Steve McConnell’s original CODE COMPLETE has been helping developers write better software for more than a decade. Now this classic book has been fully updated and revised with leading-edge practices—and hundreds of new code samples—illustrating the art and science of software construction."

"The Pragmatic Programmer" -- Andrew Hunt, David Thomas amazon link

The Pragmatic Programmer cuts through the increasing specialization and technicalities of modern software development to examine the core process--taking a requirement and producing working, maintainable code that delights its users. It covers topics ranging from personal responsibility and career development to architectural techniques for keeping your code flexible and easy to adapt and reuse.

Receiving data from ESP8266 sensors

Jake Fenton — Wed, 25 Apr 2018 05:00:00 +0000

Right, so last time I walked through how I have programmed and wired some ESP8266 microcontrollers to collect temperature, humidity and soil moisture data and beam that data to a specified address on the network.

In this post, I'll walk through the code to run a server on something like a laptop or a raspberry pi to collect the incoming data and write it to a sqlite3 database for later analysis.

This post will cover topics like threading, more advanced socket management, and interfacing with a database from Python.

The full, versioned code is available on Github. Note that link is fixed and will not update as I improve--for that, go Here and check out the Greenhouses project.

I'll be pulling snippets from that code into this post, but to see it all in context you should look at the link.

Quick Rehash

So, remember, my goal for this project is to have a fleet of environmental sensors deployed throughout a greenhouse to sample data and send it to a central location where I'll make control decisions (e.g., turn on the heater, open a vent, water tray A, etc). We need our server architecture to support multiple sensors simultaneously, as well as collating the every-two-second readings into an average over a minute or so.

Step One: Get The Data Off the Network

For a moment, let's pretend we're only expecting a single client sensor to talk to us. We'll get to dealing with multiple sensors later, but then we have to deal with threads and locking and it's a gigantic mess.

So for now, one sensor.

Aside: Server socket programming

When you're a server, you generally have one socket/port open to listen for incoming clients--like a web server, listening on port 80 for new clients. The tricky thing is you don't want to do your primary communication with your clients on that socket, or you won't be able to use it to listen for more incoming clients.

Imagine a system named Bob goes to Google and asks to download some gigantic file, and Google responds with that file on the same socket it uses to accept new client connections. Then you go to Google, but can't get through because the 'line is busy' as Google offloads the file onto Bob, which takes forever--and that entire time, you can't talk to Google.

So, you need one socket on the 'public' port to initialize the connection, and then another socket to actually handle the communication on--kind of like an operator at a business taking incoming calls and then routing them to different lines. Kind of.

Back to the network...

So, we have a bunch of ESP8226s trying to open connections to a fixed IP and port--I'm going to say, arbitrarily, it's IP 192.168.1.5 and port 5555. It doesn't really matter, so long as the server and the microcontrollers agree and the port number is somewhere between 2048 and 60666 or so. (I'm guessing on those numbers, you may want to look them up).

Your server needs to listen on that address--the IP/port combination--for incoming connections. This is fairly easy:

serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(("192.168.1.5", 5555))
serversocket.listen()

Walking through this:

First we're creating a socket with some attributes. This is a socket of the AF_INET family, which basically just means it's going to talk to the Internet (instead of another process on your system or some weird other networking protocol). This mostly just determines how the socket works under the hood, as well as the type of address we're going to give it.

This is also a socket of the SOCK_STREAM type, which really just means the socket will have an internal buffer so we can 'stream' the data at a useful rate to ourselves.

Then, we bind the socket to an address. Note: If you run into difficulty with that line, make sure the IP you have there matches the IP of the system you're using. If they mismatch, you'll get an exception here (since you're trying to bind to an address you don't actually have).

After that, we set the socket to 'listen' mode.

With me so far?

(clientsocket, address) = serversocket.accept()

Alright--so now, we're saying "Hey, socket, listen for a new connection and when you get one, hand me a socket to talk to that new connection and let me know what the address is." This is a blocking call, which means our program will now stop until the new connection comes in. Obviously, this makes it rather complicated to do any sort of data processing while we just sit here and wait--but we'll fix that later.

If you're playing with this alongside a running microcontroller what you should see is the program launch and then hang as it waits for the incoming socket connection--if you're using my ESP8266 code it'll blink steadily as it acquires the wifi connection, then it'll blink more angrily as it makes the socket connection and then it should be solid blue with an intermittent blink as it writes to the socket.

Of course, we aren't actually reading any data from the socket yet! Let's fix that:

buffer = ""

while 1:
    data = clientsocket.recv(128)
    datastr = buffer + data.decode('utf-8')
    split = datastr.split("\n")
    buffer = split[-1]
    print(split[:-1])

So, the workhouse here is the clientsocket.recv(128) call. That asks for 128 bytes of data from the socket--this is also a blocking call, meaning execution will halt until we get the 128 bytes of data. When we get the data, we decode it from just a bunch of bytes into a string and slap it onto a buffer.

The buffer is necessary because 128 bytes might not be exactly the length of a single 'record' sent from the ESP8266--you may have all of record A and half of record B. In our 'protocol', newlines are used as boundaries between records, which means we have an easy way of telling when we have an entire record.

So, we add the data we've received to the buffer (in case we just got the second half of record B) and then we split the buffer on newlines, to separate the records. We put the last record from the split, which may be a partial, and shove it back in the buffer. Then we print the rest of the records (excluding the last one).

With this, you should be able to see the incoming data from a single ESP8266 sensor! Woo!

Now we need to work on scaling to multiple connections.

Step Two: Threading the Connections

As I've mentioned a couple times we're making use of some calls that block execution, making our program halt until some outside thing happens. That makes it hard to scale to large numbers of sensors, so we're going to have to introduce some threading.

If you aren't familiar, threading is a way for a single program to do lots of things at the same time--multiple 'threads of execution.' A single process can have multiple threads where the threads all share the same memory, which makes them faster than if you had multiple processes just trying to talk to each other. (It also makes it a lot easier to write.)

The tricky thing about threads is you can have race conditions , where two threads are trying to access the same resource simultaneously. This typically leads to strange--and difficult to reproduce--behavior, so you have to be very, very careful to manage your access to shared resources when using threads to prevent this, or debugging it will be a living hell.

The general architecure we're going to build will look like most simple server models. We'll have a server thread that'll listen on port 5555 and accept incoming connections. Those connections will get moved to a different socket on a different port on a different thread, so the server thread can continue waiting for things.

Python provides a helpful threading library, thankfully. There's a couple ways you can write threads--You can either make a new Thread.Thread object and pass in the function that thread will run as an argument, or you can make a class that derives from Thread.Thread and just override the run method. I chose the latter, since I wanted to have some saved state and that seemed a good way to do it. You can do whatever you like.

Accepting incoming connections

class Overseer(threading.Thread):
    def __init__(self, threads):
        super(Overseer, self).__init__()
        print("Starting overseer thread...")

    def run(self):
        global threads_run
        global threads

        serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        serversocket.bind(("192.168.1.149", 5555))
        serversocket.listen()

        while threads_run:
            (clientsocket, address) = serversocket.accept()
            l = Listener(clientsocket, address)
            threads.append(l)
            l.start()

        serversocket.close()

Most of this should look familiar, really. The important things here are the global threads_run variable and the l = Listener(clientsocket, address) with l.start() at the bottom.

The Listener class is another thread class I made, which we'll get to in a moment.

Walking through:

The init method should be much what you expect. One catch, you must make sure to call the superclass init, or you'll get an exception when you try to .start() the thread.

The while loop is keyed off of a variable shared between all the threads--and only modified in the original main thread. This makes it possible for us to cleanly stop all the threads from the original one and clean up our resources when we're done.

With the Overseer in it's own thread we can constantly block and wait for incoming connections without preventing our program from doing other things. This is the main idea and main value of threading.

Now, one flaw in this design is that if we toggle threads_run while we're waiting for an incoming connection, we could just be waiting forever. I've been meaning to write an interrupt for that, but... later.

Also note how we're appending the new Listener thread to a global threads list--We'll come to that.

Listening to those incoming connections

class Listener(threading.Thread):
    def __init__(self, clientsocket: socket.socket, address) -> None:
        super(Listener, self).__init__()
        print("Starting listener thread...")
        self.socket = clientsocket
        self.address = address
        self.buffer = ""

    def run(self):
        global lock
        global shared_list
        global threads_run

        while threads_run:
            data = self.socket.recv(128)
            datastr = self.buffer + data.decode('utf-8')
            self.buffer = ""
            split = datastr.split("\n")
            self.buffer = split[-1]
            split = split[:-1]

            with lock:
                shared_list.extend(split)
                lock.notify()

        self.socket.close()

This is where things start to get a little more complicated, but it's still pretty close to what we had before. Remember that the Listener threads get built and started in the Overseer thread, where they get their socket and their address.

Within the listener thread, while the global threads_run is true, we constantly block and receive data from the socket. Again, since this is occuring in its own thread, the blocking here does not prevent us from executing in other threads.

We do the same processing on the received data that we did before... and then we have this weird with lock thing.

So, remember how I was talking about how you have to be super careful when accessing shared resources when you're working with threading? This is one of those instances. Here, shared_list is shared between all the running threads, and it's where they write the data they obtain from the network. To avoid collisions and weird behavior we use this lock thing, which is a threading.Condition class. In essence, the lock is a way for threads to ensure only one of them gets to touch a thing at a time. A thread has to acquire a lock, which only one thread can have at a time, and then it can do things, and then it has to release the lock for other threads, which block when they try to acquire the lock. This is handled nicely with the with syntax in Python; when you with with a lock, Python handles the acquiring and releasing of the lock. Do note that you have to manually call lock.notify() though, to let other threads know that the lock is available. Note also that I only acquire the lock when I'm about to interact with the shared data, and I release it immediately after I'm done messing with the shared data--There's no point in making other threads wait for me while I process strings that they don't share, they could be doing something useful in that time, so I try to hold onto the lock for as short a time as possible.

Now, this is a super high level view of locks. Locks can get very complicated, especially if you end up with circular waits (where thread A is waiting on thread B, but thread B is waiting on thread A). Threading is a super complicated concept that I'm not going to go into in this post, but rest assured there's plenty of other reading on it if you're so interested.

Anyway, once you have the lock, you add stuff to the list and then release the lock, ensuring nothing weird happens with shared access. Cool? Cool.

Interacting with a Database

If you haven't used them before, databases are systems designed for the easy storage and retrieval of--you guessed it--data. They're used in virtually every aspect of software, from managing the users of blogs to the content of twitter to the access logs of sensitive physical areas. They're fairly complex structures that often require a server in their own right to run on.

For our purposes, we can make do with sqlite, a super lightweight database engine that just stores all the data in a file near your program. It's trivially easy to set up and interface with in Python.

Now, I'll explain what I'm doing with the SQL commands I use here, but this isn't really a database usage tutorial so much as it is an interfacing-with-them tutorial.

Cool? Cool. Onwards!


class DBWriter(threading.Thread):
    def __init__(self):
        super(DBWriter, self).__init__()
        print("Starting DB writer thread...")

    def run(self):
        global threads_run
        global lock
        global shared_list

        connection = sqlite3.connect("greenhouse.db")
        connection.execute("CREATE TABLE if not exists readings "
                           "(source text, "
                           "airtemp real, "
                           "humidity real, "
                           "soil_moisture int, "
                           "timestamp text);")

        while threads_run:
            time.sleep(60)

            to_load_into_database = []
            with lock:
                if len(shared_list) > 0:
                    to_load_into_database = list(shared_list)
                    shared_list.clear()

            print(f"Condensing {len(to_load_into_database)} records...")
            by_source = defaultdict(list)
            record = namedtuple('SensorRecord', ['air_temp', 'air_hum', 'soil_moisture'])
            for line in to_load_into_database:
                line = line.split(';')
                mac = line[3].strip().split(" ")[1]
                by_source[mac].append(
                    record(
                        float(line[1].strip().split(" ")[1]),
                        float(line[2].strip().split(" ")[1]),
                        int(line[0].strip().split(" ")[1]),
                    )
                )

            for key in by_source:
                list_len = float(len(by_source[key]))

                avg_temp = 0
                avg_hum = 0
                avg_soil = 0

                for record in by_source[key]:
                    avg_temp += record.air_temp
                    avg_hum += record.air_hum
                    avg_soil += record.soil_moisture

                avg_temp /= list_len
                avg_hum /= list_len
                avg_soil /= list_len
                timestamp = datetime.datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")

                values = f" ('{key}', {avg_temp}, {avg_hum}, {avg_soil}, '{timestamp}')"
                print(f"Writing {values} to DB")
                connection.execute(f"INSERT INTO readings VALUES {values};")
            print("Sleeping...")

        print("DB writer thread exiting.")
        connection.commit()
        connection.close()

"Aaah! That's a lot of code! What the hell, Jake, it was going so smoothly?!"

It's fine! I promise! Really!

We'll break it down into pieces. Let's look at the setup:

class DBWriter(threading.Thread):
    def __init__(self):
        super(DBWriter, self).__init__()
        print("Starting DB writer thread...")

    def run(self):
        global threads_run
        global lock
        global shared_list

        connection = sqlite3.connect("greenhouse.db")
        connection.execute("CREATE TABLE if not exists readings "
                           "(source text, "
                           "airtemp real, "
                           "humidity real, "
                           "soil_moisture int, "
                           "timestamp text);")

        while threads_run:
            time.sleep(60)

So, the __init__ does... pretty much nothing useful.

At the start of run, we global some things that we share, and we open a connection to the database, giving it the filename we want the database to live in.

"Wait, Jake," you say. "Why wouldn't you move that connection up into __init__? Isn't it... You know... Initialization?"

You would be very correct! And I tried that. Turns out, sqlite3 objects can only be used in the thread they're created in, and, technically speaking, the __init__ function is executed from a different thread than the run function is, since __init__ is called when the thread object is built, but run is only called after you .start() the thread. So, you have to make the object after the threads have diverged.

Once we have the connection to the database we make sure the table we're going to use exists. If we don't have the if not exists phrase there, we'll get a SQL error when we try to create an already-existing table. We then provide the name and type pairs of all the columns we want in the database:

source, text: this will be the MAC address of the sending sensor. I'll use this later to figure out which sensor is where in the greenhouse so I can actually understand the incoming data, and not have to guess where it's coming from.
airtemp, real: temperature is a continuous value, not an integer. Especially once we average it over the period a sensor has been sending, and since the temps will be in celcius, an integral value is so very wide and imprecise compared to a real one.
humidity, real: See previous
soil_moisture: So this is a bit strange, since it's a percentage. Technically, the average could also be a real, but I got lazy and made this an int. Sue me.
timestamp, text: We're going to timestamp all the data, otherwise we won't know when a reading was taken, but there isn't really a consensus on how timestamps should be stored in databases. I'm doing text since it's the most flexible format.

With all that set up, we can actually enter our while loop. This thread naps for a while, since we need to wait for the data we're going to aggregate to come in.

Now for the meat of the aggregation:

            to_load_into_database = []
            with lock:
                if len(shared_list) > 0:
                    to_load_into_database = list(shared_list)
                    shared_list.clear()

            print(f"Condensing {len(to_load_into_database)} records...")
            by_source = defaultdict(list)
            record = namedtuple('SensorRecord', ['air_temp', 'air_hum', 'soil_moisture'])
            for line in to_load_into_database:
                line = line.split(';')
                mac = line[3].strip().split(" ")[1]
                by_source[mac].append(
                    record(
                        float(line[1].strip().split(" ")[1]),
                        float(line[2].strip().split(" ")[1]),
                        int(line[0].strip().split(" ")[1]),
                    )
                )

So we make an empty list that we'll copy the shared data into, then we grab the lock and empty the shared list into our local one. Then we release the lock so the other threads can write to the shared lit.

Once that's done, we start pulling the data out of the string and loading it into a namedtupe, which is really just a super efficient (and immutable) dictionary. We also group the data by the sensor that sent it.

Now we have data, grouped by sending sensor, that we need to aggregate.

  for key in by_source:
                list_len = float(len(by_source[key]))

                avg_temp = 0
                avg_hum = 0
                avg_soil = 0

                for record in by_source[key]:
                    avg_temp += record.air_temp
                    avg_hum += record.air_hum
                    avg_soil += record.soil_moisture

                avg_temp /= list_len
                avg_hum /= list_len
                avg_soil /= list_len
                timestamp = datetime.datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")

                values = f" ('{key}', {avg_temp}, {avg_hum}, {avg_soil}, '{timestamp}')"
                print(f"Writing {values} to DB")
                connection.execute(f"INSERT INTO readings VALUES {values};")
                connection.commit()

This is pretty much just math. We aggregate all the readings by the sensor that sent them and build a timestamp in what's essentially ISO format, but without milliseconds.

Once that's done, I build a string that the 'values' component of an SQL insert, using Python's wonderful format string syntax ( in Python 3.6 and above, yay!). Then we execute the insert statement and commit the change to the database.

Make sure, once the while in the run of the DatabaseWriter exits, that you close the connection! Weird things can happen if you don't.

That's great, but what does the main look like?

def main():
    global threads_run
    global shared_list
    global threads
    threads = []

    threads_run = True

    threads.append(Overseer(threads_run))
    threads.append(DBWriter())
    threads[0].start()
    threads[1].start()

    while threads_run:

        instr = ""
        while instr != "stop":
            instr = input("enter 'stop' to stop: ")

        threads_run = False

    print("Waiting for threads...")

    for thread in threads:
        thread.join()

We start the Overseer and the DBWriter, and we make sure those are appended to the threads list. Then we start both of them. The Overseer starts accepting connections and spawning Listeners, and the DBWriter thread starts aggregating data.

Then we start a loop where we wait for input, so the user can gracefully stop the program. When they do, we wait for all of the threads to halt with thead.join and then we exit our program.

There you go! That's the meat of my server! Keep in mind that this isn't by any means the "best" way to do this, and it is absolutely flawed. Keep an eye on the git repo to get updates!

Intro to working with with ESP8266

Jake Fenton — Wed, 18 Apr 2018 05:00:00 +0000

originally posted here

tl;dr

I'm building environmental sensors using the ESP8266, a super cheap wifi-capable microcontroller that can run Micropython. This guide walks through writing driver code for the 8266 to collect data from a DHT and soil moisture sensor and beam it back over the network to a server applicaiton that collects the data for use later. The project code is available here.

So, I live in the Midwest, where things get exceptionally cold during the winter. I also enjoy gardening and fresh vegetables. These facts don't mix too well. Thankfully, places like the University of Minnesota have extension programs working with deep winter greenhouses, designed to grow plants in the dead of winter.

My (long term) goal is to build and automate one of these with microcontrollers to monitor and control the environment within the greenhouse. Since I didn't want to be learning how to program microcontrollers alongside building a greenhouse I decided to get a jump start on it.

This post will detail my journey of making an environmental (temperature, humidity, soil moisture) sensor that reports back to a central server over the network.

This guide/devblog/tutorial will condense the information available from the Micropython ESP8266 tutorial, which is a phenomenally good reference, along with some documentation about the sensors I'm using.

Parts

1 ESP8266 ESP12
1 DHT11 sensor / 1 DHT 22 Sensor
1 soil moisture sensor
1 USB wall wart + micro cable

I'll assume you have various kinds of jumper wires lying around, along with a laptop.

Software

All the utilities I used were Python based; so I'm assuming you have a working Python 3 install alongside pip. You'll need some pip packages:

pip install adafruit-ampy esptool

You'll also need a copy of the most recent stable micropython firmware for the ESP8266 from the micropython page.

I use the serial tool picocom to talk to the board over USB; you're welcome to use whatever your favorite serial connection tool is, so long as it, y'know, works.

Why the ESP8266?

So, having just started working with microcontrollers I don't know crazy much about them but I'll tell you what I do know about the 8266.

Originally for this project I was going to use an arduino connected via USB to a raspi. The arduino would collect the sensor readings via it's GPIO port and report them back to the raspi via serial on USB. The pi would parse the incoming data, coallate it and send it to a controlling server. There were a couple problems with this deisgn:

Required a raspi to power and communicate with the arduino, with a physical connection. This would make it hard to place many monitoring sensors around the greenhouse, since now I need to run cable. Also increases cost, since for every 4 sensor arduinos I'd need one raspi to power and control them.
I had to write the program for the arduino in C. This I kind of expected, since C is definetely the most well-suited language for microcontrollers, but also sucks, since C is a pain in the ass to use, especially for a beginner.

I could mitigate Problem 1 by connecting a wifi chip to the arduino, and beaming the data across the network. I didn't relish having to do that with C, but it was technically possible; quick research pointed me in the direction of a chip called the ESP8266, which was apparently really popular with people wanting to use microcontrollers with wifi. What I missed during my first pass was that the ESP8266 is a microcontroller in it's own right. It doesn't need the arduino, and has GPIO capabilities of its own. In fact, the 8266 outclasses the arduino in available memory.

So, I switched tacks from building an arduino connected to a raspi over USB and decided to use a fleet of 8266s beaming data back to a single, central raspi via wifi.

Getting Started

The ESP8266s I ordered off of Ebay claimed to ship with the AT firmwhere, where you can issue commands over serial to the ESP8266 for things like connecting to a network, opening a TCP/IP socket, beaming data, accepting incoming data, etc. Unfortunately for me, either they didn't have included firmware, the firmware wasn't the AT I was expecting I or I was simply too incompetent to correctly use it, so I spent about three hours trying to convince the damn things to do what I wanted and then I ran into the Micropython project.

Following the instructions from their super helpful ESP8266 tutorial, I went hunting for the device in my /dev; running ls /dev/tty* and looking for things that end in USB0 or ACM0 will point you in the right direction.

Once I figured out what port it was on, I copied the Micropython firmware onto the chip:

$ esptool.py --port /dev/ttyUSB0 --baud 460800 write\_flash --flash\_size=detect -fm dio 0 esp8266-20171101-v1.9.3.bin

Then I connected to the chip with a serial terminal:

$ picocom /dev/ttyUSB0 -b115200

And I got greeted with the MicroPython REPL! Fantastic!

Hooking up the sensors

Most of my sensor platforms will be a soil moisture sensor running with a LM393 comparator chip and a DHT 11 or DHT 22 sensor.

The DHTs emit a digital sensor over a single wire interface, so you hook their positive lead to a 3.3v supply from the 8266, hook the negative lead to the ground on the 8266 and the middle data lead to one of the digital GPIO pins on the ESP8266.

The moisture sensor unfortunately only gives an analog signal depending on how moist the soil is, between 0 and 1 volts. This will read as a value between 0 and 1024 using the ADC module within micropython--we'll come back to that later.

As with the DHT sensor, the positive lead goes to 3.3v on the ESP8266, the negative lead goes to Ground and the analog out of the chip goes to the analog in pin of the 8266--there's only one analog input pin, in the upper left of the pinout diagram. It's labeled as "ADC0" or "A0" on the board I'm using.

One thing to note about the LM393 chip, it contains a potentometer that sends a digital signal when the analog value crosses a threshold--a "0" for below, and "1" for above the threshold. The potentometer can be adjusted with a screwdriver and some patience. For my uses, I care more about having the analog-ish value.

Now you'll note by now we've used two of the available three 3.3 rails of the ESP8266. It may be possible to split those lines to multiple directions, but I am not an electrical engineer and have no idea if that would twok, so proceed with caution.

I'm also planning to supply power for the 8266 via it's USB port. Your plans may be different, but I'm a simple fellow.

Writing the program

So, as a bonus of using Micropython, I don't have to write C code. Since I'll be dealing with network interactions and data collection, I'm very happy with this.

Now, the downside is I don't get fine-grained control over memory, and Micrpython inhabits more space on the device's program storage than an equivalent C binary would. I'm not planning to squeeze every little bit of performance out of the chip, so I'm okay with sacrificing a bit of space and management in favor of simpler code.

Micropython offers several out-of-the-box modules to do what I need for sensor data collection and network communication. The binary for the 8266 also includes functions to hook into the network fairly easily. I recommend spending some time in the REPL via a serial link when you first get your binary flashed to get comfortable with the available functionality.

Connecting to the Network

This is perhaps the meat and potatos of the ESP8266, and why you're using it in the first place: the damn thing is 2.4ghz wifi capable. Micropython is aware of this and incudes quick instructions for connecting in the help() function available from the repl, and their docs include this function:

def do_connect():
    import network
    sta_if = network.WLAN(network.STA_IF)
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect('<essid>', '<password>')
        while not sta_if.isconnected():
            pass
    print('network config:', sta_if.ifconfig())

This function, when called from your program, will block execution until the 8266 connects to the wifi network with the specified password.

I've modified this a bit to blink the built-in LED on my board while it connects, so I can get an approximate idea of the state of the device. ( Weird Note: for some reason, led.on() will turn the led off, and led.off() will turn it on. Dunno why.) You'll also note I pull the network name and password out to variables that I load from a config file:

def do_connect():
    global led_state
    sta_if = network.WLAN(network.STA_IF)
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect(NETNAME, NETPASS)
        while not sta_if.isconnected():
            if led_state:
                status_led.on()
                led_state = False
            else:
                status_led.off()
                led_state = True
            time.sleep(0.25)
    print('network config:', sta_if.ifconfig())
    status_led.off()

See? Nice and painless. (Aren't you glad this isn't C?)

Connecting to the Sensors

DHT Sensors

Right, so you're connected to wifi. You're not talking to anything on the network yet, but that's okay, we'll come to that later.

The micropython binary we're using includes a library for using DHT sensors, since they're apparently crazy popular. I have a DHT11, which is slightly cheaper and has disgustingly large errors in reads (2 degrees C!). The DHT22 is a slightly more expensive (DHT11 is ~$2/per, DHT22 is ~$3.5/per), but more accurate, at the cost of how often you can read from the sensor. We'll come to that too.

So, hooking up the DHT sensors is stupid easy. We pull in the machine module, to access the hardware pin, and the DHT module to communicate with whatever's on that pin:

d = dht.DHT11(machine.Pin(5))

Wow, that was easy.

Now, Important note: The number you hand to machine.Pin is the number of the 8266 pin , not of whatever handy board you're using. Pay attention to your pinout diagrams if you're having trouble connecting the sensor.

Anyway, now that we have this DHT11 object we can tell it to measure:

d.measure()

Which updates these values:

d.temperature(); d.humidity();

Another note, on the DHT11, you can only call measure once every second or so. On the DHT22, you can only read once every two seconds. I don't know why this is specifically, or what the consequences are of not obeying that. Again, use caution if you're coloring outside the lines.

Soil Moisture Sensor

As I mentioned before, the soil moisture sensor--as we have it hooked up--will give back an analog signal. We're going to have the ESP8266 interpret that signal as a digital value.

Here, we're making an ADC object off of that pin:
python

adc = machine.ADC(0)adc.read()

adc.read() will yield a value between 0 and 1024. Apparently 1024 means "Dry" and "0" means "wetter than water", since I stuck the sensor in a cup of water and only got a reading of something like 300. I dunno why it's this way, but it is. Use as you will.

Network communication

Okay, so we've got our data, we're connected to the wifi, now we need to take the data from the microcontroller we're on and shove it somewhere useful. Micropython includes a socket module that behaves just like the normal module in [Macro]?Python. In my code, I have the IP and port of the server application recorded in a config file, and so we open the socket and start hammering data out to it:

socket_connected = False

while True:

    while not socket_connected:
        try:
            s = socket.socket()
            s.connect((SERVIP, SERVPORT))
            s.send("Initializing link...\n")
            socket_connected = True
        except OSError:
            print("Socket connection failed... waiting.")
            # pulse LED to indicate problem
            status_led.on()
            time.sleep(0.1)
            status_led.off()
            time.sleep(0.1)
            status_led.on()
            time.sleep(0.1)
            status_led.off()
            time.sleep(0.1)
            status_led.on()
            time.sleep(0.1)
            status_led.off()

    status_led.on()
    d.measure()
    msg = "Soil: " + str(adc.read()) + "; temp: " + str(d.temperature()) + "; hum: " + str(d.humidity()) + "; from " + mac + "\n"
    print(msg)
    try:
        q = s.send(msg)
    except OSError:
        socket_connected = False
    print("Sent ", q, " bytes.")
    status_led.off()
    time.sleep(2)

Walking through this:

We drop into a forever loop, so we'll collect and transmit data until the end of time.

While we don't have an active socket connection, try to make one. If the server isn't listening, the attempt to open a socket will timeout and we'll flash the LED angrily and try again.

(I need to update this code to also check and make sure we still have a wifi link--no point in trying to transmit data if we aren't connected anyway.)

If we have a socket connection, we aren't trapped in that loop and we continue on to the meat of the loop.

We update the temperature sensor, then construct a message to transmit. This message can take whatever form you really want, I'm pretty sure it gets converted to bytes, so if you really wanted you could send raw binary data. Right now I'm just sending strings to make the dbeugging easier for me.

I also append the mac address of this board so I can identify which sensor the readings are coming from, when I'm on the server. I obtain the mac address with this line. which I stole from here:

mac = ubinascii.hexlify(network.WLAN().config('mac'),':').decode()

Once we construct the message I print it, which dumps it to serial output, which makes debugging a cinch, and I write it to the socket.

The sleep at the end gives the temperature sensor a moment to recuperate, and gives us a reasonable rate of data flow.

So, with this, you have a microcontroller environmental sensor that can send data over the network!

You can view all of my project code here.