Posted on Leave a comment

App Fairy Under the Hood

This is part 2 of our App Fairy mini-seriespart 1

The below chapters explain how we turned the existing cross-platform GUI-wrapper around the Espressif esptool.py into an app store.

Concept

The whole idea of the App Fairy is to flash IoT applications together with their configurations. Users need to do this over a serial connection from a host computer. Why so? Well, unless an application is connected to the internet it has nowhere to potentially pull any content from. Oh, and yes, without any configuration the device can not connect to the internet; catch 22.

The solution – rather our solution – to this bootstrapping problem is to write the user configuration into a file on the host computer. Subsequently, when the user flashes the application to the device the App Fairy also writes the configuration file to the device. Finally, when the application boots it will read that file and apply its configuration. The question is:

How to transfer a file from host computer to ESP8266/ESP32?

For NodeMCU-based applications there’s e.g. NodeMCU-Tool. Likewise, for MicroPython there’s e.g. MicroPython Tool. However, for ESP-IDF or Arduino-based applications the firmware SDKs don’t really have any built-in support for that. While they do support SPIFFS or LittleFS file systems there is no way we know of to interact with them from the host. In short, the only (non-TCP) solution is to build a SPIFFS image on the host and flash it to a defined memory address.

Building a SPIFFS Image

So, we needed a way to build a SPIFFS image on the host computer. Furthermore, ideally it had to be cross-platform since we wanted to distribute the App Fairy as a standalone self-contained executable. You may have heard of mkspiffs which offers executables for many platforms but it is not as such cross-platform.

This project only really started when we learned about spiffsgen.py. It is part of ESP-IDF and thus even maintained by Espressif – a promising trait. In its most simple form you invoke it like so (more parameters are supported)

python spiffsgen.py <image_size> <base_dir> <output_file>

The base_dir is the directory that contains all the files to make available on the device file system. The image_size is at least as large as the base_dir content and at most what you reserved for SPIFFS when you built the application. For ESP32 the file system size is a parameter of the chosen partition table. For ESP8266 Arduino SPIFFS size is part of the board configuration you select through the menu.

Most importantly, however, is to make sure that spiffsgen.py uses exactly the same dozen or so parameters to build a SPIFFS image as your app does. Consequently this means that for every single app (revision) the SPIFFS parameters may be different. More on that below.

Initially spiffsgen.py only supported building for ESP32. Thankfully the great Ivan Grokhotkov extended spiffsgen.py in GitHub issue #6717 to also support ESP8266 Arduino apps.

Dynamic Parameters & UI

The App Fairy is fully dynamic in terms of the apps it supports. If new apps or updates to existing apps become available they will automatically load the next time you run the App Fairy. Hence, you do not need to update it to use the new apps.

To achieve this we do not just store the apps on our servers but also instructions for the App Fairy how to handle them. So, after the user selects one of the supported device types the App Fairy loads information about available apps for that device. What it receives is something like this:

- name: "Bluetooth Speaker"
  description: "Bluetooth loudspeaker with 8 band spectrum analyzer."
  releases:
    - version: 1.0
      path: "bluetooth-speaker-1.0"

It uses this data to populate the application dropdown widget.
ThingPulse App Fairy app selector

Likewise, something similar happens when the user selects an application (application plus version to be precise). The App Fairy loads configuration and SPIFFS properties from our servers:

spiffs:
  page-size: 256 
  block-size: 4096 
  use-magic-len: true
  aligned-obj-ix-tables: false
  meta-len: 4
  offset: "0x210000"
  size: "0x1F0000"
application:
  - prop: countdownSeconds
    label: Countdown duration in seconds
    type: number
    default: 120
  - prop: ssid
    label: WiFi SSID
    type: text
  - prop: password
    label: WiFi password
    type: password
  - prop: volume
    label: "Audio volume, 0-21"
    type: number
    default: 4

The SPIFFS properties drive the generation of the SPIFFS image. Also, its offset property defines to which memory location the generated image is flashed to upon installation. On the other hand, the application properties are used to render a dynamic GUI dialog for the user to complete.ThingPulse App Fairy configuration dialogNote how we support different property types like text, number, password, and so on.

Packaging Apps

When the user finally hits that “Install” button, the App Fairy uses esptool.py to flash the app binary to address 0x00000 on the device (and the SPIFFS image to its offset location). Those familiar with the ESP8266/ESP32 memory layout may wonder: what is in this app binary to make this work?

For our older ESP8266 apps built with the Arduino IDE the answer is simple. The the ESP8266 platform tools in the Arduino IDE produce a self-contained .bin file that can directly be flashed to address 0x00000. Navigate to the temporary folder denoted in the below screen shot to find a <sketch>.ino.bin file to install.Arduino IDE ESP8266 compile output

These days our preferred IoT tools are VS Code & PlatformIO for ESP32. If you add the verbose flag to the PlatformIO upload target, it will reveal which components it will flash to which memory address: platformio run --target upload -v.PlatformIO ESP32 compile output

We use this information for a shell script to produce a single aggregated binary for address 0x00000. Said script uses the srec_cat command from the SRecord package to place the individual files at their respective address. In addition, it pads/fills the gaps with 0xFF. Excerpt for two pieces: ...-fill 0xff 0x0000 0x8000 "$partitions" -binary -offset 0x8000...

  • Fill 0x0000 to 0x8000 with 0xFF
  • Place the partitions file at 0x8000
  • And so on..

As this script is integrated into the PlatformIO build process via a post-build action we also get an aggregated .bin for the app store with every build. Very convenient!

Leave a Reply

Your email address will not be published. Required fields are marked *