Developer Service

Posted on Sep 11 • Originally published at developer-service.blog

How I Log ESPHome Device Data to CSV with Python (and Why You Should Too)

#python #homeassistant #esp #esphome

If you’ve ever relied on ESPHome devices inside Home Assistant, you might have run into the same frustration I did: sometimes the values just don’t update when you expect them to. Maybe it’s a sensor that reports sporadically, or a switch state that seems out of sync.

I recently started getting into home automation, and I’ve been loving the process of exploring both the software and the hardware side. For hardware, I chose to work with ESP32 boards running ESPHome, they’re cheap, powerful, and flexible enough for just about any DIY automation project.

Still, even with that setup, I wanted a way to see the raw updates directly from my ESP devices, without relying solely on Home Assistant. That’s how I ended up building a little Python logger, one that connects to my devices, listens for updates, and saves everything into daily CSV files. It’s been a game changer for troubleshooting and for keeping long-term historical logs.

Why not just use Home Assistant logs?

Home Assistant is amazing, but it isn’t really designed to give you a per-device raw data log. It’s fantastic at automation and dashboards, you can see the current state of your devices, set up history graphs, and even use the recorder integration to store data in a database.

But there are a few limitations I ran into:

The history database is optimized for Home Assistant dashboards, not for raw data analysis. It prunes older entries and doesn’t always give you every state change.
If a sensor only reports sporadically, it’s hard to tell whether that’s because the sensor didn’t send data or because Home Assistant just didn’t capture it.
Exporting data from Home Assistant can be clunky if all you want is a simple CSV file you can open in Excel, Pandas, or Grafana.

On top of that, I use ESPHome’s deep sleep mode on some of my ESP32 devices to save battery. That means the device only wakes up occasionally, sends its readings, and then goes back to sleep. While this is perfect for power efficiency, it makes it even harder to rely on Home Assistant’s database to know exactly when the updates happened and whether something was missed during the wake/sleep cycles.

I wanted something simpler: a “black box recorder” for my ESP devices that would capture every single update as it happens, with timestamps, and store it in a CSV file.

That’s where ESPHome’s native API comes in. With Python, you can connect directly to the device (the same way Home Assistant does behind the scenes) and log every update in real time. Instead of depending on what Home Assistant decides to store, you get a first-hand record straight from the source, including those infrequent updates from deep-sleeping devices.

Step 1: Talking to ESPHome from Python

ESPHome exposes a simple TCP API (default port 6053). It’s the same mechanism that Home Assistant uses under the hood when you add an ESPHome device through the integration. The nice part is that we don’t need to reinvent the wheel, there’s already a fantastic Python library called aioesphomeapi that handles authentication, device discovery, and streaming updates.

What this API gives us is direct access to raw sensor data, switches, binary sensors, and even services on the device. Instead of waiting for Home Assistant to decide how and when to store data, we can connect straight to the source and capture every update as it happens.

Here’s the minimal script I started with, just to print out values:

import asyncio
from aioesphomeapi import APIClient
from dotenv import load_dotenv
import os

# Load environment variables
load_dotenv()
API_HOST = os.getenv("API_HOST")
API_PASSWORD = os.getenv("API_PASSWORD")

async def main():
    # Connect to your ESPHome device by IP or hostname
    client = APIClient(API_HOST, 6053, password=API_PASSWORD, noise_psk=API_PASSWORD)
    await client.connect(login=True)

    # Define a callback function that will be triggered every time
    # the device sends a new state update (sensor reading, switch state, etc.)
    def state_callback(state):
        print("State update:", state)

    # Subscribe to all state updates
    client.subscribe_states(state_callback)

    # Keep the script running so we continue receiving updates
    while True:
        await asyncio.sleep(10)

asyncio.run(main())

When I ran this for the first time, it was a real “aha!” moment. Suddenly, I could watch every single update flowing in:

A temperature sensor reporting every 30 seconds
A binary sensor flipping between ON and OFF
Even deep-sleep devices popping online briefly, sending their data, and then disappearing again

Here is the output of one of my sensors:

State update: SensorState(key=3640870636, device_id=0, state=26.569412231445312, missing_state=False)
State update: SensorState(key=1365279611, device_id=0, state=1016.6199951171875, missing_state=False)
State update: SensorState(key=1277562889, device_id=0, state=63.120174407958984, missing_state=False)
State update: SensorState(key=426116527, device_id=0, state=156542.0, missing_state=False)
State update: SensorState(key=2104753181, device_id=0, state=84.01597595214844, missing_state=False)
State update: SensorState(key=1883882361, device_id=0, state=840.1597900390625, missing_state=False)
State update: SensorState(key=3764174908, device_id=0, state=1.2201377153396606, missing_state=False)
State update: SensorState(key=2195749573, device_id=0, state=2.855468988418579, missing_state=False)
State update: TextSensorState(key=1338075055, device_id=0, state='Stabilizing', missing_state=False)
State update: TextSensorState(key=688020976, device_id=0, state='Good', missing_state=False)

What’s powerful here is that this stream is unfiltered. It’s not averaged, not pruned, not optimized for dashboards. It’s the ground truth of what the device is actually sending. That’s exactly what I needed for troubleshooting and long-term logging.

Step 2: Making it Useful (Logging to CSV)

Printing to the console is fun, but it’s not very practical. If you really want to do something useful with the data, you need it in a format that’s easy to store, search, and analyze.

That’s why I decided to log everything into CSV files. CSV is one of those formats that never goes out of style:

You can double-click it and open it in Excel or Google Sheets
You can load it straight into Pandas with two lines of Python
It’s plain text, so you don’t need any special tools or databases to work with it
It’s simple enough that you can even inspect it with a text editor

But just dumping everything into a single file forever didn’t feel right. Over time, it would become massive and unmanageable. That’s why I also added daily rotation, each day gets its own file, named like esphome_2025-09-06.csv. This makes it easy to archive old data, analyze trends by day, or even compress older logs without touching the new ones.

So I wrote a reusable ESPHomeLogger class. It:

Connects to a device using the ESPHome native API
Subscribes to all updates (sensors, switches, binary sensors, etc.)
Writes each update into a CSV row with:
- a timestamp (ISO 8601, e.g. 2025-09-06T14:23:01)
- the entity ID (sensor.living_room_temp)
- the friendly name (Living Room Temperature)
- the raw state value (23.5)
Rotates automatically at midnight, starting a new file with headers

This way, I don’t just see updates in real time, but I also have a permanent historical record of every value my ESPHome device reported, neatly organized into daily slices.

Here’s the core of it:

import asyncio
import csv
from math import isnan
import os
from datetime import datetime, date, timedelta
from aioesphomeapi import APIClient
from aioesphomeapi.core import APIConnectionError

# ESPHomeLogger class
class ESPHomeLogger:
    def __init__(self, host, password=None, csv_dir="logs", retry_interval=15):
        self.host = host
        self.client = APIClient(host, 6053, password=password, noise_psk=password)
        self.csv_dir = csv_dir
        self.current_date = None
        self.entity_map = {}
        self.retry_interval = retry_interval
        self.connected = False
        self.connected_time = None

        os.makedirs(csv_dir, exist_ok=True)

    # Get the CSV file for the current date
    def _get_csv_file(self):
        today = date.today().isoformat()
        if today != self.current_date:
            self.current_date = today
            self.csv_file = os.path.join(self.csv_dir, f"esphome_{today}.csv")
            if not os.path.exists(self.csv_file):
                with open(self.csv_file, "w", newline="") as f:
                    csv.writer(f).writerow(["timestamp", "entity_id", "friendly_name", "state"])
        return self.csv_file

    # Connect to the ESPHome device
    async def connect(self):
        try:
            await self.client.connect(login=True)
            entities, _ = await self.client.list_entities_services()
            self.entity_map = {ent.key: ent.name for ent in entities}
            self.client.subscribe_states(self._state_callback)
            self.connected = True
            self.connected_time = datetime.now()
            print(f"Successfully connected to {self.host}")
        except APIConnectionError as e:
            self.connected = False
            print(f"Connection failed to {self.host}: {e}")
            raise
        except Exception as e:
            self.connected = False
            print(f"Unexpected error connecting to {self.host}: {e}")
            raise

    # State callback
    def _state_callback(self, state):
        entity_id = state.key
        friendly = self.entity_map.get(entity_id, "unknown")
        value = getattr(state, "state", None)
        # Skip if the value is None or nan
        if value is None:
            return

        # Check if value is a number and if it's NaN
        try:
            if isinstance(value, (int, float)) and isnan(value):
                return
        except (TypeError, ValueError):
            # If it's not a number, continue processing
            pass

        # Write to the CSV file
        with open(self._get_csv_file(), "a", newline="") as f:
            csv.writer(f).writerow([datetime.now().isoformat(), entity_id, friendly, value])
            print(f"State update: {datetime.now().isoformat()} - {self.host} - {entity_id} - {friendly} - {value}")

    # Run the logger with retry logic
    async def run(self):
        while True:
            try:
                if not self.connected:
                    print(f"Connecting to {self.host}")
                    await self.connect()

                # Keep the connection alive
                await asyncio.sleep(10)

                # Assume disconnected if connected time is more than 5 minute ago
                if self.connected_time and datetime.now() - self.connected_time > timedelta(minutes=5):
                    print(f"Last connection was more than 5 minutes ago, assuming disconnected")
                    self.connected = False
                    self.connected_time = None

            except APIConnectionError as e:
                print(f"Connection lost to {self.host}: {e}")
                self.connected = False
                print(f"Retrying connection to {self.host} in {self.retry_interval} seconds...")
                await asyncio.sleep(self.retry_interval)

            except Exception as e:
                print(f"Unexpected error with {self.host}: {e}")
                self.connected = False
                print(f"Retrying connection to {self.host} in {self.retry_interval} seconds...")
                await asyncio.sleep(self.retry_interval)

With this class running, I could finally trust my logs. Even my deep-sleeping ESP32 devices that only wake up every 10 minutes would faithfully drop a line in the CSV each time they reported.

Step 3: Scaling It Up (Multiple Devices)

Of course, once you have one device logging nicely, the natural next step is… why not all of them?

Most of us running ESPHome aren’t dealing with just one sensor. In my case, I’ve got:

An ESP32 in the living room tracking temperature and humidity
Another one in the kitchen monitoring CO₂ levels
A battery-powered sensor in the bedroom that uses deep sleep to report every 15 minutes
Plus a few switches and relays sprinkled around the house

If you only log one device, you’ll get insight into a single corner of your setup. But when you log all of them in parallel, you suddenly get a complete historical picture of how your home is behaving: room-by-room climate data, wake/sleep cycles, and even subtle glitches you’d never catch otherwise.

Thanks to Python’s asyncio, running multiple loggers at once is surprisingly easy. Each ESPHomeLogger instance runs in its own task, connecting to its device and writing to its own CSV file.

Here’s what that looks like:

import asyncio
from esphome_logger import ESPHomeLogger
from dotenv import load_dotenv
import os

# Load environment variables
load_dotenv()

# Bedroom
API_HOST_BEDROOM=os.getenv("API_HOST_BEDROOM")
API_BEDROOM_PASSWORD=os.getenv("API_BEDROOM_PASSWORD")
# Living room
API_HOST_LIVINGROOM=os.getenv("API_HOST_LIVINGROOM")
API_LIVINGROOM_PASSWORD=os.getenv("API_LIVINGROOM_PASSWORD")
# Attic
API_HOST_ATTIC=os.getenv("API_HOST_ATTIC")
API_ATTIC_PASSWORD=os.getenv("API_ATTIC_PASSWORD")

async def main():
    devices = [
        {"host": API_HOST_LIVINGROOM, "password": API_LIVINGROOM_PASSWORD, "csv_dir": "logs/living_room"},
        {"host": API_HOST_BEDROOM, "password": API_BEDROOM_PASSWORD, "csv_dir": "logs/bedroom"},
        {"host": API_HOST_ATTIC, "password": API_ATTIC_PASSWORD, "csv_dir": "logs/attic"},
    ]
    loggers = [ESPHomeLogger(**dev) for dev in devices]
    await asyncio.gather(*(logger.run() for logger in loggers))

if __name__ == "__main__":
    asyncio.run(main())

Each device writes its logs into a separate folder (logs/living_room, logs/kitchen, logs/bedroom), and within each folder you’ll find daily CSV files like:

logs/
├── living_room/
│   ├── esphome_2025-09-06.csv
│   ├── esphome_2025-09-07.csv
│   └── ...
├── bedroom/
│   ├── esphome_2025-09-06.csv
│   ├── esphome_2025-09-07.csv
│   └── ...
└── attic/
    ├── esphome_2025-09-06.csv
    ├── esphome_2025-09-07.csv
    └── ...

And a CSV example:

timestamp,entity_id,friendly_name,state
2025-09-09T09:02:14.362082,1797020032,Illuminance,2.263779401779175
2025-09-09T09:02:16.803419,305332570,(BMP180) Temperature,23.515762329101562
2025-09-09T09:02:16.804758,3839816525,(BMP180) Pressure,1018.670654296875
2025-09-09T09:02:27.640247,1554900236,PM <2.5�m,2.200000047683716
2025-09-09T09:02:27.642056,476366160,PM <10.0�m,9.0
2025-09-09T09:02:29.376025,40205146,(DHT11) Temperature,23.799999237060547
2025-09-09T09:02:29.377491,1570302627,(DHT11) Humidity,64.0
2025-09-09T09:02:29.378466,1797020032,Illuminance,2.263779401779175
2025-09-09T09:02:31.796109,305332570,(BMP180) Temperature,23.522075653076172
2025-09-09T09:02:31.797392,3839816525,(BMP180) Pressure,1018.62451171875

This structure keeps things neat and makes it super easy to:

Archive logs by device
Pull data for a single room when troubleshooting
Load multiple CSVs into Pandas and compare trends across devices

Once I had this running, I realized I finally had a transparent, reliable history of my ESPHome devices, something that I was struggling with Home Assistant.

You can find all the source code at: https://github.com/nunombispo/ESPHome-Logger

What I Learned

Working through this little side project taught me a few valuable lessons:

Home Assistant is great, but sometimes raw device data is the real truth: Don’t get me wrong, I love Home Assistant. It’s the brain of my smart home, and I wouldn’t trade its automation power or dashboards for anything. But when it comes to debugging flaky devices (especially those using deep sleep or running on battery), Home Assistant’s history view doesn’t always give you the full story. Raw logs let you see whether the device actually sent the data in the first place.

ESPHome’s native API is surprisingly easy to work with in Python: Thanks to aioesphomeapi, it was basically plug-and-play. Within a couple of lines of Python, I was watching updates stream in from my ESP32 devices. If you’re comfortable with asyncio, the whole thing feels very natural.

Having daily CSV logs is a game changer for analysis: CSV might not sound exciting, but it’s the perfect format for this. It let me quickly:

Spot missed updates (e.g., if a deep-sleep device failed to wake up on schedule).
Compare trends across rooms (temperature differences between living room and bedroom at night).
Archive old data without bloating my Home Assistant database.
Feed the logs into tools like Pandas or Grafana for deeper analysis.

Sometimes you need your own tools, even if you already have Home Assistant: This was probably the biggest realization: Home Assistant is a fantastic platform, but it doesn’t cover every niche need. Writing a lightweight companion script gave me a level of transparency and control I didn’t have before.