Embed Binary Data on ESP32

There is a reason the title of this post specifically says “Embed Binary Data on ESP32“. This suggests that it is different than embedding binary data on let’s say ESP8266. Yes indeed, that’s part of the story. The other is that to embed binary data on ESP32 you don’t need to jump through hoops anymore like on ESP8266. Instead, you will do what likely feels most natural: store the binary data in a file in the project directory and have the compiler slurp it from there. Easy as pie!

However, let’s rewind first to understand where we are coming from. Skip the first section if you only came for the TL;DR.

All the examples below revolve around a very common need for “binary” data on embedded platforms: SSL certificates to establish encrypted communication channels to (cloud) data platforms. Why the quotes around ‘binary’ you ask…well, isn’t all data ultimately binary in computer science? Anyway, enough of the fluff, let’s move to the stuff.

Warning: if you got into IoT more recently and never worked with ESP8266 or traditional Arduino the first section might feel like a tech-history lesson to you.

This post is part of a series of articles about what you learn if you attend a ThingPulse ESP32 workshop.

Embed Binary Data on ESP8266/Arduino

Embedding in source code

Traditionally you would embed SSL certificates directly into your Arduino/C/C++ source code. To achieve that there are two main routes you can take here but the results aren’t too different from each other.

The first option is to produce a hex dump by running xxd -i -a filename to create “output in C include file style. A complete static array …”. For our SSL certificate example this would look like this:

const char* cert = \
"-----BEGIN CERTIFICATE-----\n" \
"MIIF6TCCA9GgAwIBAgIQBeTcO5Q4qzuFl8umoZhQ4zANBgkqhkiG9w0BAQwFADCB\n" \
"iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n" \
[..]
"0m9jqNf6oDP6N8v3smWe2lBvP+Sn845dWDKXcCMu5/3EFZucJ48y7RetWIExKREa\n" \
"m9T8bJUox04FB6b9HbwZ4ui3uRGKLXASUoWNjDNKD/yZkuBjcNqllEdjB+dYxzFf\n" \
"BT02Vf6Dsuimrdfp5gJ0iHRc2jTbkNJtUQoj1iM=\n" \
"-----END CERTIFICATE-----\n";

That has worked fine for years but has three major drawbacks

you need xxd – or similar – which may be an issue on Windows
it’s an ugly notation that clutters up your source code
plus the data will come to reside in RAM, of which you normally don’t have plenty on microcontrollers

In order to save RAM you will want to see your binary data loaded into PROGMEM (program memory) instead. PROGEM is an Arduino AVR feature “which places the variable in the .irom.text section in flash.” Furthermore, in addition to the RAM savings the following notation is much more convenient as you can skip the hex dump hoop and paste data as-is. Note how the lines are no longer individual string literals joined by \n and \.

You can choose between R"=====()=====" and R"EOF()EOF" to wrap your data.

static const char cert[] PROGMEM = R"=====(
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
[..]
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
)=====";

static const char cert2[] PROGMEM = R"EOF(
 - - -BEGIN CERTIFICATE - - -
[..]
 - - -END CERTIFICATE - - -
)EOF";

However, at this point this is all still part of your source code. Sure, you will likely tuck it away in some .h header file which you then #include but still.

Loading from SPIFFS

To remedy the data-in-source-code issue you could place the certificate on the filesystem (given that your platform has one). For example, on Arduino for ESP8266 you would then read it from SPIFFS like so

File cert = SPIFFS.open("/ca.crt.der", "r");

Side note, on ESP8266 you can also provide an entire certificate store from which BearSSL will pick the correct root cert.

The downside of this approach is the somewhat bumpy build and deployment model. The certificate is no longer part of your app binary or firmware. As a result, you need to ensure yourself – through some other means – that firmware and SPIFFS content are installed on the micro controller as a single consistent package.

ESP32 doesn’t just offer better hardware over ESP8266. Also, thanks to the Espressif ESP-IDF we have a more modern hardware abstraction and SDK to work with. The ESP-IDF programming guide documents a very elegant way to embed binary and text data. You can simply place the file in your project directory structure and register it as an IDF component. Note in the examples below that this works for both binary files with COMPONENT_EMBED_FILES and text files with COMPONENT_EMBED_TXTFILES.

idf_component_register(...
                       EMBED_FILES server_root_cert.der)
idf_component_register(...
                       EMBED_TXTFILES server_root_cert.pem)

The build adds contents to the .rodata section in flash. Remember the PROGMEM section above? In both cases the data will not end up in RAM. In your code the content is available via symbol names as follows:

extern const uint8_t server_root_cert_pem_start[] asm("_binary_server_root_cert_pem_start");
extern const uint8_t server_root_cert_pem_end[] asm("_binary_server_root_cert_pem_end");

The names are generated from the full name of the file, as given in COMPONENT_EMBED_FILES. However. the characters /, ., etc. are replaced with underscores. The _binary prefix in the symbol name is added by “objcopy” and is the same for both text and binary files.

The fact that you have access to both the start and the end of the content allows to easily calculate the number of bytes or Content-Length (hint: HTTP header) of a file.

With PlatformIO

PlatformIO is our preferred development environment for ESP32. Its advantages over Arduino IDE are too numerous to list here. On the other hand, setting everything up has become really simple these days. Head over to platformio.org to learn more.

Due to the abstraction layer PlatformIO offers the ESP-IDF component settings for embedding data can be added directly to platformio.ini like so:

[env:esp-wrover-kit]
platform = espressif32
board = esp-wrover-kit
build_flags = -DCOMPONENT_EMBED_TXTFILES=src/rootCA.pem
              -DCOMPONENT_EMBED_FILES=src/cat.jpg:src/icon.png

Again, this will then give you access to start and end of each file via the extern const uint8_t file_start[] asm() notation shown above. Note how multiple files are separated with a colon :. Also remember that / in the file path (directory/file.bin) will be translated to _. Thus, that start of the certificate file is expressed as

extern const uint8_t cert_start[] asm("_binary_src_rootCA_pem_start");

Summary

We learned how easy it has become to embed binary data on ESP32 thanks to ESP-IDF. Also, thanks to the abstraction PlatformIO offers adding the necessary COMPONENT_EMBED_FILES configuration as a build flag in platformio.ino is a one-liner.

If you like this short introduction into embedding binary data on ESP32 then you might also like the other articles about what you learn if you attend a ThingPulse ESP32 workshop. Subscribe to this site’s feed with your favorite RSS reader or follow us on Twitter to never miss new content.

Embed Binary Data on ESP8266/Arduino

Embedding in source code

Loading from SPIFFS