There is a reason the title of this post specifically says “Embed Binary Data on ESP32“. This suggests that it is different than embedding binary data on let’s say ESP8266. Yes indeed, that’s part of the story. The other is that to embed binary data on ESP32 you don’t need to jump through hoops anymore like on ESP8266. Instead, you will do what likely feels most natural: store the binary data in a file in the project directory and have the compiler slurp it from there. Easy as pie!
However, let’s rewind first to understand where we are coming from. Skip the first section if you only came for the TL;DR.
All the examples below revolve around a very common need for “binary” data on embedded platforms: SSL certificates to establish encrypted communication channels to (cloud) data platforms. Why the quotes around ‘binary’ you ask…well, isn’t all data ultimately binary in computer science? Anyway, enough of the fluff, let’s move to the stuff.
Warning: if you got into IoT more recently and never worked with ESP8266 or traditional Arduino the first section might feel like a tech-history lesson to you.
This post is part of a series of articles about what you learn if you attend a ThingPulse ESP32 workshop.
Embed Binary Data on ESP8266/Arduino
Embedding in source code
Traditionally you would embed SSL certificates directly into your Arduino/C/C++ source code. To achieve that there are two main routes you can take here but the results aren’t too different from each other.
The first option is to produce a hex dump by running
xxd -i -a filename to create “output in C include file style. A complete static array …”. For our SSL certificate example this would look like this:
const char* cert = \ "-----BEGIN CERTIFICATE-----\n" \ "MIIF6TCCA9GgAwIBAgIQBeTcO5Q4qzuFl8umoZhQ4zANBgkqhkiG9w0BAQwFADCB\n" \ "iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl\n" \ [..] "0m9jqNf6oDP6N8v3smWe2lBvP+Sn845dWDKXcCMu5/3EFZucJ48y7RetWIExKREa\n" \ "m9T8bJUox04FB6b9HbwZ4ui3uRGKLXASUoWNjDNKD/yZkuBjcNqllEdjB+dYxzFf\n" \ "BT02Vf6Dsuimrdfp5gJ0iHRc2jTbkNJtUQoj1iM=\n" \ "-----END CERTIFICATE-----\n";
That has worked fine for years but has three major drawbacks
- you need xxd – or similar – which may be an issue on Windows
- it’s an ugly notation that clutters up your source code
- plus the data will come to reside in RAM, of which you normally don’t have plenty on microcontrollers
In order to save RAM you will want to see your binary data loaded into PROGMEM (program memory) instead. PROGEM is an Arduino AVR feature “which places the variable in the .irom.text section in flash.” Furthermore, in addition to the RAM savings the following notation is much more convenient as you can skip the hex dump hoop and paste data as-is. Note how the lines are no longer individual string literals joined by
You can choose between
R"EOF()EOF" to wrap your data.
static const char cert PROGMEM = R"=====( -----BEGIN CERTIFICATE----- MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/ MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT [..] JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ -----END CERTIFICATE----- )====="; static const char cert2 PROGMEM = R"EOF( - - -BEGIN CERTIFICATE - - - [..] - - -END CERTIFICATE - - - )EOF";
However, at this point this is all still part of your source code. Sure, you will likely tuck it away in some
.h header file which you then
#include but still.
Loading from SPIFFS
To remedy the data-in-source-code issue you could place the certificate on the filesystem (given that your platform has one). For example, on Arduino for ESP8266 you would then read it from SPIFFS like so
File cert = SPIFFS.open("/ca.crt.der", "r");
Side note, on ESP8266 you can also provide an entire certificate store from which BearSSL will pick the correct root cert.
The downside of this approach is the somewhat bumpy build and deployment model. The certificate is no longer part of your app binary or firmware. As a result, you need to ensure yourself – through some other means – that firmware and SPIFFS content are installed on the micro controller as a single consistent package.
Embed Binary Data on ESP32
ESP32 doesn’t just offer better hardware over ESP8266. Also, thanks to the Espressif ESP-IDF we have a more modern hardware abstraction and SDK to work with. The ESP-IDF programming guide documents a very elegant way to embed binary and text data. You can simply place the file in your project directory structure and register it as an IDF component. Note in the examples below that this works for both binary files with
COMPONENT_EMBED_FILES and text files with
idf_component_register(... EMBED_FILES server_root_cert.der) idf_component_register(... EMBED_TXTFILES server_root_cert.pem)
The build adds contents to the .rodata section in flash. Remember the PROGMEM section above? In both cases the data will not end up in RAM. In your code the content is available via symbol names as follows:
extern const uint8_t server_root_cert_pem_start asm("_binary_server_root_cert_pem_start"); extern const uint8_t server_root_cert_pem_end asm("_binary_server_root_cert_pem_end");
The names are generated from the full name of the file, as given in
COMPONENT_EMBED_FILES. However. the characters
/, ., etc. are replaced with underscores. The
_binary prefix in the symbol name is added by “objcopy” and is the same for both text and binary files.
The fact that you have access to both the start and the end of the content allows to easily calculate the number of bytes or
Content-Length (hint: HTTP header) of a file.
PlatformIO is our preferred development environment for ESP32. Its advantages over Arduino IDE are too numerous to list here. On the other hand, setting everything up has become really simple these days. Head over to platformio.org to learn more.
Due to the abstraction layer PlatformIO offers the ESP-IDF component settings for embedding data can be added directly to
platformio.ini like so:
[env:esp-wrover-kit] platform = espressif32 board = esp-wrover-kit build_flags = -DCOMPONENT_EMBED_TXTFILES=src/rootCA.pem -DCOMPONENT_EMBED_FILES=src/cat.jpg:src/icon.png
Again, this will then give you access to start and end of each file via the
extern const uint8_t file_start asm() notation shown above. Note how multiple files are separated with a colon
:. Also remember that
/ in the file path (directory/file.bin) will be translated to
_. Thus, that start of the certificate file is expressed as
extern const uint8_t cert_start asm("_binary_src_rootCA_pem_start");
We learned how easy it has become to embed binary data on ESP32 thanks to ESP-IDF. Also, thanks to the abstraction PlatformIO offers adding the necessary
COMPONENT_EMBED_FILES configuration as a build flag in
platformio.ino is a one-liner.
If you like this short introduction into embedding binary data on ESP32 then you might also like the other articles about what you learn if you attend a ThingPulse ESP32 workshop. Subscribe to this site’s feed with your favorite RSS reader or follow us on Twitter to never miss new content.