Large documentation update (#10)

* Docs: overhaul of all the docs.  The main README.md is much leaner and broke off the steps.  Added a FAQ, greatly expanded the Wiring section.

* Fixed FAQ link

* Fixed some relative links

* Fixed another link

* Minor

* Reverted title of the Project - didn't mean to do that.

docs/backup_firmware.md

@@ -0,0 +1,35 @@
+# Back up the original firmware
+
+Prerequisites:
+- You need the [ESPHome CLI](installing_esphome.md) installed.
+- You need the Vue device to be [wired to a USB to TTY adapter](wiring_and_usb.md).
+
+Reminder: triple check your wiring, making sure you are not connecting +5V to anything other than pin 6.  You can now plug it into your computer and the Vue device should power up and start working as normal.
+
+**1. Determine which USB port it is connected to:**
+  - on Windows, it shows up as `COM3`, `COM4`, etc
+  - on Mac and Linux, look in the filesystem: `/dev/ttyUSB0`, `/dev/ttyUSB1`, or `/dev/tty.usbserial-xxxx`.
+
+**2. To backup the firmware:**
+
+Note: the `esptool` command is used here, which comes installed with the ESPHome CLI.
+```
+# esptool --port <usb_port> --chip esp32 -b 115200 read_flash 0x0 0x400000 <filename>
+```
+
+Examples:
+```
+(on Windows)
+# esptool --port COM3 --chip esp32 -b 115200 read_flash 0x0 0x400000 vue_original_firmare.bin
+
+(on a Mac)
+# esptool --port /dev/tty.usbserial-BG01UJXF --chip esp32 -b 115200 read_flash 0x0 0x400000 vue_original_firmare.bin
+```
+
+Reminder: if it doesn't work, swap pins 1 and 2.
+
+**3. If you need to restore:**
+
+```
+# esptool --port <usb_port> --chip esp32 -b 115200 write_flash --flash_freq 80m 0x0 <filename>
+```

docs/basics.md

@@ -0,0 +1,34 @@
+# Basics Explained
+
+The [Emporia Vue Utility Connect](https://shop.emporiaenergy.com/products/utility-connect) is a small device you can buy online.  It plugs into any standard USB A power supply, and Emporia ships the unit with one.  It wirelessly connects to some utility smart meters to read real-time energy usage.
+
+Note that it is only compatible with some utilities, and you will need to pair it with your smart meter.
+
+## With stock firmware
+Once paired and working, it reads your meter every 30 seconds and sends that to Emporia cloud servers via your home wifi and internet.  You can access that information using the Emporia app on a smart phone.  Your smart phone does not connect directly to the Vue device.
+
+![Vue with stock firmware](vue_stock.jpg)
+
+## With this ESPHome firmware
+If you want this data in Home Assistant, this project provides firmware built with ESPHome that you flash to the Vue device.
+- It overwrites the stock firmware.
+- Once flashed and working, the Vue device will push data to Home Assistant.  It will no longer send data to Emporia servers, and the Emporia app will no longer work.  Note that you can back up the stock firmware, and restore it later if you need to.
+- You can customize the firmware.  For example, I have mine reading the meter every 15 seconds.
+
+![Vue with ESPHome firmware](vue_flashed.jpg)
+
+## What is this data?
+
+Since the Emporia Vue Utility Connect reads directly from your utility smart meter, the data you see is the **official measurement** used by your power company.
+- **Real-time power flow** — how many watts your home is currently pulling from or sending to the grid (if you have solar panels).  
+- **Cumulative totals** for both import and export, which makes the data compatible with the **Home Assistant Energy dashboard**.  
+
+## What you get in Home Assistant
+The ESPHome Integration is quite magical.  Once your device is flashed and running, Home Assistant automatically detects the Vue and creates a device record with sensor entities.  These can be used to build automations, dashboards, and long-term storage however you like.
+
+![ESPHome integration](esphome_screenshot.png)
+
+### Energy module
+You can add the cumulative sensors **Watt-hours Consumed** and **Watt-hours Produced** Home Assistant’s [Energy module](https://www.home-assistant.io/home-energy-management/). This provides a polished and feature-rich view of your energy usage and production.
+
+![Energy module](energy_screenshot.png)

docs/faq.md

@@ -0,0 +1,126 @@
+# Frequently Asked Questions
+
+
+### 1. Why is your product so good?
+
+Why do so many FAQ's include this?
+
+---
+
+### 2. Where can I get help?
+
+The [Community thread](https://community.home-assistant.io/t/emporia-vue-utility-connect/).
+
+If you are brave: [read the ESPHome docs](https://esphome.io/components/)
+
+---
+
+### 3. I'm new, I just want something that works.  Why are there multiple YAML files and why are they different?
+
+They’re just starting points with different features enabled or disabled. Pick one, add your Wi-Fi/password, and it should work.
+
+Later, if you want to customize, see the [section on YAML configuration](yaml.md).  
+
+
+---
+
+### 4. I'm getting intermittent drop outs.  What do I check?
+
+- First: **swap the USB power supply**.  We have seen a number of the Emporia supply USB power supplies fail, which started out like this and the device failing.
+- You may need to move the device closer to your power meter.
+- You may need to contact your utililty and ask them to reprovision the device.
+
+---
+
+### 5. What is the API encryption key in the YAML file?
+
+Looks like this in the YAML:
+```
+api:
+  encryption:
+    key: "blah"
+```
+
+- Just a security measure.  It’s a shared secret between your device and Home Assistant.
+- Ensures all ESPHome API traffic (state updates, commands, etc.) is encrypted and authenticated.
+- Without it, someone on your LAN could spoof commands to the device.
+- ESPHome generates one if you don’t set it yourself.
+
+---
+
+### 6. What is the OTA password in the YAML file?
+
+Looks like this in the YAML:
+```
+api:
+  encryption:
+    key: "blah"
+```
+- Just a security measure.  It’s a shared secret between your device and Home Assistant.
+- Used only when flashing new firmware over Wi-Fi (OTA updates).
+- Prevents anyone on your LAN from pushing firmware to the device without authorization.
+- ESPHome generates one the first time if you don’t set it yourself.
+
+---
+
+### 7. Do I need MQTT?
+
+Not at all.  This is in the example YAML files in case you want to access the data via MQTT topics.  By default, the ESPHome firmware discovers and talks via it's native API (not MQTT discovery).
+
+---
+
+### 8. What happens to the Vue device + entity records in HA if I rebuild the firmware?
+
+As long as you build the firmware with:
+- the same esphome.name (in YAML)
+- the same sensor names (in YAML)
+- and running on the same physical device
+
+.. HA will reuse same entities.  You can reflash and switch between the ESPHome CLI and ESPHome Device Builder, and you will not lose sensor entities or data.  See next question for more details.
+
+---
+
+### 9. How do the device and entities work in HA for the Vue?
+
+**The Device ID**
+- comes from the firmware when it is discovered over the API (when you turn it on).
+- It is based on this in the YAML file:  
+  ```
+  esphome:
+    name: vue-utility
+  ```
+- HA makes a “device” record around that, so all the sensor entities are grouped.
+
+**Entity IDs (e.g. sensor.vue_utility_watts)**
+- Generated from each sensor defined in YAML
+- They’re based on the device name + the component name.
+- Example:  
+  ```
+  esphome:
+    name: vue-utility
+
+  sensor:
+    - platform: emporia_vue_utility
+      name: "Watts"
+  ```
+  gives sensor.vue_utility_watts.
+
+**Unique IDs**
+
+Under the hood, every device and entity in HA is allocated a unique ID.  Unique IDs for devices and entities in HA work differently, but for the Vue both are tied to the Vue's hardware MAC address:
+
+**1. Device**
+HA - the unique ID is generated and tied to the MAC address of the device.
+- If you rename the device (e.g. 'vue-utility' to 'uncle_bob'), HA keeps the same underlying device, it will just be renamed.
+- Let's say your Vue breaks and you buy another one, it will come with a different MAC address and a different device will be created in HA for it.
+
+**2. Sensor entities**
+Likewise, HA stores a unique ID for each sensor which is directly based on the MAC address of the device + sensor name.  e.g.
+```
+  {
+    "entity_id": "sensor.vue_utility_watts",
+    "unique_id": "20:60:AB:E5:12:24-sensor-watts"
+  }
+```
+
+---

docs/installing_esphome.md

@@ -0,0 +1,68 @@
+# Installing ESPHome
+
+## Introduction
+
+If this is your time to [ESPHome](https://esphome.io/), the terminology can be confusing. You may hear “flash ESPHome on your device”, but that’s not a totally accurate statement.
+
+ESPHome is a **framework and toolchain** for creating custom firmware. This usually involves:
+- Writing a short YAML configuration file, and optionally providing custom code.
+- Using ESPHome software to:
+  - Compile that configuration (and maybe custom code) into custom firmware.
+  - Flash that firmware onto a device.
+
+So when you hear someone say "a device running ESPHome", what they really mean is "a device running firmware built with ESPHome".
+
+There are multiple ways to do this and much of the documentation out there is difficult for new comers.  This document walks you through:
+- the high level of ESPHome
+- the two of the ways to use ESPHome to build and flash the firmware for this project
+
+
+## Overview
+
+You will interact with two distinct parts of ESPHome:  
+1. **ESPHome** — this is software you will use this to back up the stock firmware, compile this project, and flash the Vue.  There are two main ways to do this, with some caveats.
+2. **ESPHome integration** — you only need to install the [ESPHome Integration](https://www.home-assistant.io/integrations/esphome) in Home Assistant.  It is quite magical.  Once the Vue is flashed and running on Wi-Fi, it will auto-discover it and create the sensor entities automatically.
+
+
+### How ESPHome works
+With ESPHome, you pass it a YAML configuration file and it will:
+1. Fetch dependencies
+2. Generate code
+3. Compile it into firmware
+4. Flash the Vue
+
+![Wiring the Vue](esphome_diagram.png) 
+
+There is an overwhelming amount of documentation on writing a YAML file.  You do not need to start from scratch.  There are [example_yaml files](../example_yaml) in this project to get you started, which you just need to complete (mostly just your wifi credentials).
+
+### The two main ways to run ESPHome
+
+
+#### 1. ESPHome CLI --- [install guide](https://esphome.io/guides/installing_esphome/)
+
+This gives you a CLI (command-line interface) to run ESPHome.  You run this in the terminal of your computer: Windows, Mac or Linux.  It is completely separate from Home Assistant.  It does not run with or on Home Assistant, and does not talk to it in any way.
+ - you simply run: `esphome run <yaml_file>`
+ - runs on your own computer (Python or Docker).
+ - better for developers: quick iteration, local editing of YAML/Python/C++.
+ - flexible: you can use local custom code without publishing to GitHub.
+ - full control over build logs, debugging, and advanced options.
+ - you **can** backup the original firmware with this.
+
+
+#### 2. ESPHome Device Builder --- [install guide](https://esphome.io/guides/getting_started_hassio/#installing-esphome-device-builder)
+
+This is a Home Assistant add-on.  It is basically a UI wrapper for the ESPHome CLI, and runs on the Home Assistant server.  You use the Home Assitant web interface to access it.
+ - you use an in-browser editor to paste YAML code
+ - runs inside Home Assistant (web UI).
+ - better for end users: simpler UI.
+ - initial flashing still happens on your PC.  It works using WebSerial (via a browser).
+ - you **cannot** use this to backup the original firmware.
+
+
+### What is best for you?
+
+- If you are adamant about [backing up the original Vue firmware](backup_firmware.md), minimally install the CLI just to get the backup step done.  You can use either the CLI or the Device Builder to program and flash, and you can switch between them as well.
+
+- If you like working from the command line and want full control, the ESPHome CLI is straightforward: you write YAML by hand, build, and flash. It’s lean, reproducible, and great if you’re already comfortable with terminals.
+
+- The Device Builder adds a friendlier interface.  It’s easier for beginners, integrates neatly with Home Assistant, and makes troubleshooting or rebuilding devices more point-and-click.

docs/pinout.md

@@ -1,4 +1,4 @@
-## Pinouts
+# Pinouts
 
 ![Emporia Vue Utility device](device_800.jpg)
 

docs/protocol-meter-reading.md

@@ -1,4 +1,4 @@
-# Meter Response Payload
+# Meter Reading
 
 There are at least two known response payloads sent by the MGM111 chip. MGM Firmware version 2 sends a 152 byte payload while version 7 sends a 44 byte payload. Check the section below relevant to your version.
 

docs/wiring_and_usb.md

@@ -0,0 +1,108 @@
+# Wiring and USB Adapter
+
+The [ESPHome documentation covers this subject](https://esphome.io/guides/physical_device_connection/), but not this specific device.
+
+### 1. You need a USB to TTL Serial Aapater
+
+You will need a USB to TTL serial adapter.  Almost any adapter will work.  It needs to supply 3.3V signaling, and preferrably also provide a supply of 5V.
+
+I can confirm this one works :- [Amazon: DSD TECH SH-U09C5 USB to TTL serial adapter](https://www.amazon.com/dp/B07WX2DSVB)
+
+
+### 2. Wire up to port "P3"
+
+<img align="right" width="300" src="device_800_labeled.jpg" />
+
+You need to open the cover of the Vue device, and locate the port marked "P3".
+
+The port pins consist of through holes and you will need to wire them up to USB to TTL adapter.  There are many ways to do this. I chose to solder some Cat 5 wires directly to the board, and used a breadboard and Dupont jumpers to wire those to the USB adapter.  See pictures below.
+
+<br>
+<br>
+
+![Wiring the Vue](vue_wiring_1200.jpg)
+
+### 3. Connections
+
+This is the how each pin should be wired to the usb adapter:
+
+| Pin | Description | Connect to USB-TTL |                       |
+| --- | ----------- | -------------------| --------------------- |
+| 1   | IO0         | RTS or DTR **(*)** | 3.3 volts             |
+| 2   | EN          | DTR or RTS **(*)** | 3.3 volts             |
+| 3   | GND         | GND                |
+| 4   | TX          | RX                 | 3.3 volts             |
+| 5   | RX          | TX                 | 3.3 volts             |
+| 6   | +5v         | +5v                | 5v to power the board |
+
+<br>
+NOTES:
+
+- You **must use 3.3V signaling**.
+
+- Pin 6 (the pin just above the text "EmporiaEnergy") is 5 volts, not 3.3v.  It is used to power the device.  **Do not to apply 5V to the wrong pin** or magic smoke may come out.  You can avoid connecting pin 6 by plugging in the device into a regular usb power supply/port to provide power.
+
+- (*) The wiring depends on the USB to TTL serial adapter you are using.
+  - For example, the [DSD TECH SH-U09C5 USB to TTL serial adapter](https://www.amazon.com/dp/B07WX2DSVB), Pin 1 is DTR and Pin 2 is RTS.
+  - Nothing will be damaged if these are swapped.
+  - If one way of wiring does not work, swap pins 1 and 2.
+  - This has tripped quite a few people.  See below for a more complete explanation.
+
+<br>
+
+These are the pins on the [DSD TECH SH-U09C5 USB to TTL serial adapter]((https://www.amazon.com/dp/B07WX2DSVB)):
+
+![DSD TECH SH-U09C5 USB to TTL serial adapter](usb_to_tty_1200.jpg)
+
+
+Here is an example of how to do this:
+
+![Wiring the Vue](vue_breadboard_1200.jpg)
+
+
+<br>
+
+
+## Why are the instructions for Pins 1 and 2 **"to just swap them if they don't work"?**
+
+ESP32 needs two lines to enter the UART bootloader:
+
+- **EN** (reset): pulse low to reset the chip
+- **IO0** (boot select): hold low while EN rises to boot into the loader
+
+Flashing tools (like `esptool`, used by ESPHome) do not have “EN/IO0” pins. They drive **DTR** and **RTS** (modem control signals) in a specific timing pattern that assumes:
+
+- **DTR → EN**
+- **RTS → IO0**
+
+On some USB-serial adapters, those labels or polarities are effectively flipped. That’s why you’ll sometimes need to swap the connections.
+
+
+Many ESP32 dev boards (NodeMCU/WROOM devkits) hide this logic behind transistors/inverters, so you never think about it. **Bare USB-TTL dongles expose raw DTR/RTS**, where names and polarity are not standardized.
+- **Label drift:** Some boards label the header “RTS” where the chip is actually driven like DTR (or vice-versa).
+- **Polarity/inversion:** CH340, CP210x, FTDI, etc. often invert DTR/RTS differently. A vendor may “fix” this on the PCB without changing the silkscreen names.
+- **Hidden glue logic:** Dev boards add transistors that cross-wire/invert the signals so any mapping “just works.” Bare adapters do not.
+- **Manual bootloading masks wiring errors:** If you short IO0 to GND at power-up, flashing will work even if DTR/RTS are wrong—making the bad mapping look “fine.”
+
+---
+
+### Symptoms that your mapping is wrong
+
+- `esptool` can’t sync unless you use `--before usb_reset` or you physically unplug/replug.
+- Flashing completes, but the device won’t reboot cleanly unless you power-cycle.
+- You must keep holding IO0 to GND by hand for it to work.
+
+Those all mean EN/IO0 aren’t being driven with the expected timing and polarity.
+
+---
+
+### “But my adapter has a dedicated RESET pin…”
+
+`esptool` does not drive arbitrary “RESET” header pins. It toggles only DTR/RTS. Unless your adapter internally mirrors DTR to that RESET header, using it won’t help the auto-sequence. Stick to DTR/RTS.
+
+---
+
+### Why `--before usb_reset` sometimes works
+
+`--before usb_reset` avoids DTR/RTS entirely by bouncing the USB device. That can momentarily create the right conditions for the bootloader, but it’s crude and unreliable.  It also explains why some people need to manually unplug/replug after flashing. Fixing EN/IO0 mapping makes this unnecessary.
+

docs/yaml.md

@@ -0,0 +1,130 @@
+# YAML configuration
+
+## Introduction
+
+ESPHome comes with a [large library of components](https://esphome.io/components/) and configuration options, which translates to features and functionality in the firmware.  Some people want some features, others won't.
+
+The example YAMLs should work out of the box, provided you fill in the necessary secrets.  This section is to help get you started on customizing.  You may want to do this to add features, or take some away to make it leaner.
+
+### Examples
+
+- The [Web Server](https://esphome.io/components/web_server/) component adds a browser-accessible web UI with sensors and logs. Neat, but it duplicates functionality already in Home Assistant, and consumes a lot of memory on the device. If you won’t use it, disable it.
+- The [WiFi Signal Strength](https://esphome.io/components/sensor/wifi_signal/) component creates a sensor in HA showing the devices Wi-Fi strength.  You can also see this is the device logs.  Handy for troubleshooting, but once the connection is stable you may not need an extra entity cluttering HA.
+- The [Uptime Sensor](https://esphome.io/components/sensor/uptime/?utm_source=chatgpt.com) reports how long the device has been running since the last reboot. Handy for debugging stability (e.g. “is my device randomly restarting?”). But if you don’t actively monitor it, it’s more HA clutter.
+- [MQTT](https://esphome.io/components/mqtt/) is a powerful but more complex alternative to HA’s native ESPHome API.  It is essential for some setups, but if you don’t already use MQTT, you can remove it.
+
+
+## Vue configuration
+
+### Sensors
+
+This Vue sensor component provides **6** output values and takes **3** configuration options.
+
+A full configuration can look like this:
+
+```yaml
+# Boilerplate UART configuration.
+uart:
+  id: emporia_uart
+  rx_pin: GPIO21
+  tx_pin: GPIO22
+  baud_rate: 115200
+
+# The actual sensor configuration.
+sensor:
+  - platform: emporia_vue_utility
+    uart_id: emporia_uart
+    debug: true
+    update_interval: 15s
+    power:
+      name: '${name} Watts'
+    power_export:
+      name: '${name} Watts Returned'
+    power_import:
+      name: '${name} Watts Consumed'
+    energy:
+      name: '${name} Wh Net'
+    energy_export:
+      name: '${name} Wh Returned'
+    energy_import:
+      name: '${name} Wh Consumed'
+```
+
+You likely do not need all of these fields, and you can remove (or comment out) the sensors you do not need.
+
+
+#### UART
+
+We need to talk to the MGM chip that talks to the utility meter. This chip is connected to the ESP via `GPIO21` and `GPIO22` and the configuration is:
+
+```yaml
+uart:
+  id: emporia_uart
+  rx_pin: GPIO21
+  tx_pin: GPIO22
+  baud_rate: 115200
+```
+
+Then, within the `sensor:` configuration, you can add `uart_id: emporia_uart`. I believe it is optional if there is only one `uart:` defined.
+
+Example with `uart_id` added:
+
+```yaml
+sensor:
+  - platform: emporia_vue_utility
+    uart_id: emporia_uart
+    ...
+```
+
+#### Poll rate
+
+The `update_interval` lets you define the rate at which we ask the utility meter for a new reading. This defaults to `30` seconds, which is also the default in the Emporia stock firmware.
+
+Note that the utility meter has its own update rate, and if you poll more frequently it will just give you the same value as before. Some users report being able to receive new values every 5 seconds while I only see a new value every 15 seconds. You are welcome to trial-and-error and see how frequently you can get updated values and settle on a value that makes sense to you.
+
+Example with `update_interval` added:
+
+```yaml
+sensor:
+  - platform: emporia_vue_utility
+    update_interval: 15s
+    ...
+```
+
+#### Debug logs
+
+Add `debug: true` if you need to see additional details about issues you're experiencing.
+
+When enabled, each reading will cause the log to output something like this:
+
+```
+[00:05:17][D][emporia_vue_utility:360]: Meter Cost Unit: 1000
+[00:05:17][D][emporia_vue_utility:361]: Meter Divisor: 1
+[00:05:17][D][emporia_vue_utility:362]: Meter Energy Import Flags: 00596680
+[00:05:17][D][emporia_vue_utility:363]: Meter Energy Export Flags: 0078f478
+[00:05:17][D][emporia_vue_utility:364]: Meter Power Flags: 00038d2a
+[00:05:17][D][emporia_vue_utility:365]: Meter Import Energy: 5858.944kWh
+[00:05:17][D][emporia_vue_utility:366]: Meter Export Energy: 7926.904kWh
+[00:05:17][D][emporia_vue_utility:367]: Meter Net Energy: -2067.960kWh
+[00:05:17][D][emporia_vue_utility:368]: Meter Power:  909W
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes   0 to   3: 18 91 01 00
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes   4 to   7: 00 00 25 80
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes   8 to  11: 66 59 00 00
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  12 to  15: 00 01 00 00
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  16 to  19: 25 78 f4 78
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  20 to  23: 00 00 00 01
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  24 to  27: 03 00 22 01
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  28 to  31: 00 00 02 03
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  32 to  35: 00 22 e8 03
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  36 to  39: 00 00 04 00
+[00:05:17][D][emporia_vue_utility:377]: Meter Response Bytes  40 to  43: 2a 8d 03 00
+```
+
+Example with `debug` added:
+
+```yaml
+sensor:
+  - platform: emporia_vue_utility
+    debug: true
+    ...
+```