Microcontroller projects

work in progress

Programmimg ESP's by using the ESPToolbox library

last updated: 2022-07-06

Intro

Why this? There are cool books and tutorials on the net with code snippets to program ESP's. You can find e.g. a really useful guide from Pieter P on github.io.

I work often with both microcontroller from Espressif, the ESP8266 and the ESP32. And I'm a forgetful man, an often search quite a long time to find back pieces of code already written and used in my projects. So here I wrote an Arduino library to hold all those pieces of code and named it ESPToolbox. The code is intended to work on ESP8266 and ESP32.

Boards used

I often use the LOLIN (WEMOS) D1 mini pro. The MHEtLive ESP32MiniKit is small and nearly (RxD and TxD are interchanged!!) pin compatible with the LOLIN board and comes handy if we need a more powerful controller.

flash memory

In the middle of the picture we see the prior version of the LOLIN board named Wemos D1 mini pro.

Other ESP8266 and ESP32 boards should also work with the following software.

Naming conventions

I was often confused and did not know what convention for naming identifiers (names given to a program element) to use in different languages, and so was sometimes not consistent in my own programs.

In last years I often use Python with a quite clear convention (PEP8) and as this convention does not conflict with C++ (Arduino) I will try to stick to that convention beginning today ;). Here my rules:

Logging and debugging

Let's start with the most important chapter :).

Debug with BUILTIN_LED

Most boards have one or more built in LEDs. When no serial interface or UDP connection is available, it's good to use the LED's to pass debug infos. In the library we have little helper methods to use the built-in LED.

For many boards LED_BUILTIN is already defined. If this is not the case or if you want to use another LED use the overloaded method set_led_log(bool flag, byte led_pin) to change the pin number. The LED's on ESP8266 and ESP32 often use negative logic! If the logic on your board is negative use the overloaded method: `set_led_log(bool flag, bool pos_logic).

Here the example code using the library:

    /* esp_debug_with_builtin_LED.ino
       www.weigu.lu
       Led is on after enabling log!  */

    #include <ESPToolbox.h>

    ESPToolbox Tb;                        // create an ESPToolbox object

    /****** SETUP *************************************************************/

    void setup() {
      Tb.set_led_log(true);               // LED logging. (pos logic, LED on)
      //Tb.set_led_log(true,false);       // use neg. logic: 2 par = false)
      //Tb.set_led_log(true,5);           // other pin if LED_BUILTIN not def.
      //Tb.set_led_log(true,5,false);     // alt. pin + neg. logic
      delay(2000);
    }
    /****** LOOP **************************************************************/

    void loop() {
      Tb.blink_led_x_times(3);            // default blink with 100ms (f=5Hz)
      delay(2000);                        // (LED was on and is on after blink)
      Tb.blink_led_x_times(3,500);        // blink with 500ms delay (f=1s)
      delay(2000);
      Tb.led_off();
      delay(2000);
      Tb.led_on();
    }

There exists a bunch of other LED helper functions not connected to logging to work with LED's:

Debug/Log with Serial

Logging per serial monitor helps a lot when programming and debugging. Unfortunately TxD and/or RxD from Serial0 (Serial) are also needed for programming most ESP boards or are needed for communication to a sensor or a device. ESP8266 has fortunately a second serial interface (Pin D4 for WEMOS/LOLIN board, look here). ESP32 has even three serial interfaces. The second and the third serial interface are accessed through Serial1 ans Serial2.

With a serial to USB adapter cable (3.3V!! for ESP boards) and a terminal program like Cutecom we can log over Serial1 or Serial2.

The library contains the following methods for serial logging:

To debug or log messages we use the log() or log_ln()functions.

    /* esp_log_serial.ino
      www.weigu.lu
      Serial1 on Wemos D1 Mini: pin D4 (only Tx)
      Serial1 on MH ET Live ESP32 MiniKit (Tx): SD3  (Pin can be changed!)
      Serial2 on MH ET Live ESP32 MiniKit (Tx): IO17 (Pin can be changed!)  */

    #include <ESPToolbox.h>

    ESPToolbox Tb;                        // create an ESPToolbox object

    /****** SETUP *************************************************************/

    void setup() {
      Tb.set_serial_log(true);            // enable LED  serial logging on Serial
      Tb.set_led_log(true);               // enable LED logging (pos logic)
      // overloaded method to choose Serial1 (1) or Serial2 (2, only ESP32)
      //Tb.set_serial_log(true,1);
    }
    /****** LOOP **************************************************************/

    void loop() {
      Tb.blink_led_x_times(3);            // default blink with 100 ms
      Tb.log("Hel");                      // no new line
      Tb.log_ln("lo");                    // add newline (\n) at the end
      delay(2000);
    }

Debug/Log with UDP

Serial is cool over the Arduino serial monitor. But if this is not possible, it gets cumbersome because additional hardware is needed. The ESP's have WiFi on board and allow us to use UDP to send messages to a computer. If we have a Linux computer (e.g. raspberry pi) the simple netcat program allows us to listen to our log information.

So we will see all the logging information in a terminal window of the Linux computer (e.g. raspberry pi)!

To do so we open a terminal window and type:

    nc -kulw 0 6464

The port (here 6464) we want to use can naturally be changed at will.

The library contains the following methods for UDP logging:

As we use WiFi and we need to use a WiFi method explained in the next chapter.

Config.h

We also have to to pass our user name and password. For UDP we need to have a fixed IP address for our Linux computer and a port. To make these settings more accessible, they are contained in a config file named config.h. This file must be in the same folder as our sketch (.ino file).

When we want to distribute the software, we don't want to share the password in the config file. So here is another way:
We create a new folder called e.g. Secrets in the libraries folder in our Arduino sketchbook folder (the path to the sketchbook folder can be found in the Preferences Window (File Menu in Arduino IDE)). Now we create a new text file called secrets_xxx.h (xxx will be replaced with a meaningful extension) in the Secrets folder. Then we copy the content of the config file config.h to the secrets_xxx.h file.

Last we uncomment the line #define USE_SECRETS in the Sketch. Here an example with the config data in a file named secrets_udp_logging.h.

    /* esp_log_udp.ino
       www.weigu.lu
       for UDP, listen on Linux PC (UDP_LOG_PC_IP) with netcat command:
       nc -kulw 0 6464                                                   */

    /*!!!!!!       Make your changes in config.h (or secrets_xxx.h)      !!!!!!*/

    /*------ Comment or uncomment the following line suiting your needs -------*/
    #define USE_SECRETS

    /****** Arduino libraries needed ******/
    #include "ESPToolbox.h"            // ESP helper lib (more on weigu.lu)
    #ifdef USE_SECRETS
      // The file "secrets_xxx.h" has to be placed in a sketchbook libraries
      // folder. Create a folder named "Secrets" in sketchbook/libraries and copy
      // the config.h file there. Rename it to secrets_xxx.h
      #include <secrets_udp_logging.h> // things you need to change are here or
    #else
      #include "config.h"              // things you need to change are here
    #endif // USE_SECRETS

    /****** WiFi and network settings ******/
    const char *WIFI_SSID = MY_WIFI_SSID;         // in secrets_xxx.h or config.h
    const char *WIFI_PASSWORD = MY_WIFI_PASSWORD; // in secrets_xxx.h or config.h
    IPAddress UDP_LOG_PC_IP(UDP_LOG_PC_IP_BYTES); // in secrets_xxx.h or config.h

    ESPToolbox Tb;                     // Create an ESPToolbox Object

    /****** SETUP *************************************************************/

    void setup() {
      Tb.set_udp_log(true, UDP_LOG_PC_IP, UDP_LOG_PORT);
      Tb.set_led_log(true); // enable LED logging (pos logic)
      Tb.init_wifi_sta(WIFI_SSID, WIFI_PASSWORD);
    }
    /****** LOOP **************************************************************/

    void loop() {
      Tb.log_ln("Hi there :)");
      delay(5000);
      Tb.blink_led_x_times(3);
    }

Here the output of the program:

    Connected to SSID mywifi with IP 192.168.178.144
    Signal strength is -50 dBm
    Hi there :)
    Hi there :)

The other way around if we want to use the file config.h from the .ino folder we comment the line #define USE_SECRETS:

    //#define USE_SECRETS

Here the content of a minimal config.h (or secrets_xxx.h) file:

    /****** WiFi SSID and PASSWORD ******/
    const char *MY_WIFI_SSID = "your_ssid";
    const char *MY_WIFI_PASSWORD = "your_password";

    /****** WiFi and network settings ******/
    // UDP logging settings if enabled in setup(); Port used for UDP logging
    const word UDP_LOG_PORT = 6464;
    // IP address of the computer receiving UDP log messages
    const byte UDP_LOG_PC_IP_BYTES[4] = {192, 168, 178, 100};

WiFi methods and a static IP address

Above we used the simplest WiFi method to connect an ESP to a router (Wifi station mode STA). There exist other overloaded methods to add mDNS, a own hostname or a fixed IP address for our project:

Let's look at an example with fixed IP, mDNS and hostname.

First we need to add things to our config e.g. secrets file (secrets_static_ip.h):

    /****** WiFi SSID and PASSWORD ******/
    const char *MY_WIFI_SSID = "your_ssid";
    const char *MY_WIFI_PASSWORD = "your_password";

    /*+++++++ Things you can change: +++++++*/

    /****** WiFi and network settings ******/
    // UDP logging settings if enabled in setup(); Port used for UDP logging
    const word UDP_LOG_PORT = 6464;
    // IP address of the computer receiving UDP log messages
    const byte UDP_LOG_PC_IP_BYTES[4] = {192, 168, 178, 100};
    // optional (access with UDP_logger.local)
    const char *NET_MDNSNAME = "UDP_logger";
    // optional hostname
    const char *NET_HOSTNAME = "UDP_logger";
    // only if you use a static address (uncomment //#define STATIC in ino file)
    const byte NET_LOCAL_IP_BYTES[4] = {192, 168, 178, 155};
    const byte NET_GATEWAY_BYTES[4] = {192, 168, 178, 1};
    const byte NET_MASK_BYTES[4] = {255,255,255,0};

In our main file a little function init_wifi_sta() was added to get clearer code.

A new line lets you choose if a static IP address is used or not (comment (//#define STATIC) or uncomment the line).

    #define STATIC       // if static IP needed (no DHCP)
    /*
      esp_static_ip.ino
      www.weigu.lu
      for UDP, listen on Linux PC (UDP_LOG_PC_IP) with netcat command:
      nc -kulw 0 6464*/

    /*!!!!!!       Make your changes in config.h (or secrets_xxx.h)      !!!!!!*/

    /*------ Comment or uncomment the following line suiting your needs -------*/
    #define USE_SECRETS  // if we use secrets file instead of config.h
    #define STATIC       // if static IP needed (no DHCP)

    /****** Arduino libraries needed ******/
    #include "ESPToolbox.h"            // ESP helper lib (more on weigu.lu)
    #ifdef USE_SECRETS
      // The file "secrets_xxx.h" has to be placed in a sketchbook libraries
      // folder. Create a folder named "Secrets" in sketchbook/libraries and copy
      // the config.h file there. Rename it to secrets_xxx.h
      #include <secrets_static_ip.h> // things you need to change are here or
    #else
      #include "config.h"              // things you need to change are here
    #endif // USE_SECRETS

    /****** WiFi and network settings ******/
    const char *WIFI_SSID = MY_WIFI_SSID;         // in secrets_xxx.h or config.h
    const char *WIFI_PASSWORD = MY_WIFI_PASSWORD; // in secrets_xxx.h or config.h
    IPAddress UDP_LOG_PC_IP(UDP_LOG_PC_IP_BYTES); // in secrets_xxx.h or config.h
    #ifdef STATIC
      IPAddress NET_LOCAL_IP (NET_LOCAL_IP_BYTES);// 3x optional for static IP
      IPAddress NET_GATEWAY (NET_GATEWAY_BYTES);  // look in config.h
      IPAddress NET_MASK (NET_MASK_BYTES);
    #endif // ifdef STATIC

    ESPToolbox Tb;                                // Create an ESPToolbox Object

    /****** SETUP *************************************************************/

    void setup() {
      Tb.set_udp_log(true, UDP_LOG_PC_IP, UDP_LOG_PORT);
      Tb.set_led_log(true); // enable LED logging (pos logic)
      init_wifi_sta();
    }
    /****** LOOP **************************************************************/

    void loop() {
      Tb.log_ln("Hi there again with static IP :)");
      delay(5000);
      Tb.blink_led_x_times(3);
      if (WiFi.status() != WL_CONNECTED) {   // if WiFi disconnected, reconnect
        init_wifi_sta();
      }
    }

    // init WiFi (overloaded function if STATIC)
    void init_wifi_sta() {
      #ifdef STATIC
        Tb.init_wifi_sta(WIFI_SSID, WIFI_PASSWORD, NET_HOSTNAME, NET_LOCAL_IP,
                        NET_GATEWAY, NET_MASK);
      #else
        Tb.init_wifi_sta(WIFI_SSID, WIFI_PASSWORD, NET_MDNSNAME, NET_HOSTNAME);
      #endif // ifdef STATIC
    }

Using Over The Air programming (OTA)

ESPs have WiFi and so Over The Air programming is very convenient if your project is in a closed housing in the garden or cellar.

As for the other features a define line lets you choose if OTA is used or not (comment (//#define OTA) or uncomment the line).

    #define OTA       // if Over The Air update needed (security risk!)

We only need one method in setup():

Inside the loop we use the method from the OTA lib: ArduinoOTA.handle().

But first we need to add two lines our config e.g. secrets file (secrets_use_ota.h):

    /****** WiFi SSID and PASSWORD ******/
    const char *MY_WIFI_SSID = "your_ssid";
    const char *MY_WIFI_PASSWORD = "your_password";

    /****** WiFi and network settings ******/
    // UDP logging settings if enabled in setup(); Port used for UDP logging
    const word UDP_LOG_PORT = 6464;
    // IP address of the computer receiving UDP log messages
    const byte UDP_LOG_PC_IP_BYTES[4] = {192, 168, 178, 100};
    // optional (access with UDP_logger.local)
    const char *NET_MDNSNAME = "UDP_logger";
    // optional hostname
    const char *NET_HOSTNAME = "UDP_logger";
    // only if you use OTA (uncomment //#define OTA in ino file)
    const char *MY_OTA_NAME = "esp_with_ota"; // optional (access with esp_with_ota.local)
    // Linux Create Hasgh with: echo -n 'P@ssword1' | md5sum
    const char *MY_OTA_PASS_HASH = "myHash";     // Hash for password

Here the basic code:

    /* esp_use_ota.ino
      www.weigu.lu
      for UDP, listen on Linux PC (UDP_LOG_PC_IP) with netcat command:
      nc -kulw 0 6464  */

    /*!!!!!!       Make your changes in config.h (or secrets_xxx.h)      !!!!!!*/

    /*------ Comment or uncomment the following line suiting your needs -------*/
    #define USE_SECRETS  // if we use secrets file instead of config.h
    #define OTA          // if Over The Air update needed (security risk!)

    /****** Arduino libraries needed ******/
    #include "ESPToolbox.h"            // ESP helper lib (more on weigu.lu)
    #ifdef USE_SECRETS
      // The file "secrets_xxx.h" has to be placed in a sketchbook libraries
      // folder. Create a folder named "Secrets" in sketchbook/libraries and copy
      // the config.h file there. Rename it to secrets_xxx.h
      #include <secrets_use_ota.h> // things you need to change are here or
    #else
      #include "config.h"              // things you need to change are here
    #endif // USE_SECRETS

    /****** WiFi and network settings ******/
    const char *WIFI_SSID = MY_WIFI_SSID;         // in secrets_xxx.h or config.h
    const char *WIFI_PASSWORD = MY_WIFI_PASSWORD; // in secrets_xxx.h or config.h
    IPAddress UDP_LOG_PC_IP(UDP_LOG_PC_IP_BYTES); // in secrets_xxx.h or config.h
    #ifdef OTA                                    // optional OTA settings
      const char *OTA_NAME = MY_OTA_NAME;         // look in config.h
      const char *OTA_PASS_HASH = MY_OTA_PASS_HASH;
    #endif // ifdef OTA

    ESPToolbox Tb;                                // Create an ESPToolbox Object

    /****** SETUP *************************************************************/

    void setup() {
      Tb.set_udp_log(true, UDP_LOG_PC_IP, UDP_LOG_PORT);
      Tb.set_led_log(true); // enable LED logging (pos logic)
      Tb.init_wifi_sta(WIFI_SSID, WIFI_PASSWORD);
      #ifdef OTA
        Tb.init_ota(OTA_NAME, OTA_PASS_HASH);
      #endif // ifdef OTA
    }
    /****** LOOP **************************************************************/

    void loop() {
      #ifdef OTA
        ArduinoOTA.handle();
      #endif // ifdef OTA
      Tb.log_ln("Hi there; program me over the air :)");
      delay(5000);
      Tb.blink_led_x_times(3);
      if (WiFi.status() != WL_CONNECTED) {   // if WiFi disconnected, reconnect
        Tb.init_wifi_sta(WIFI_SSID, WIFI_PASSWORD);
      }
    }

Get the time from an Network Time Protocol (NTP) server

Most projects working with sensor data need an exact time. As we are connected to WiFi we can get the time from an NTP server in the Internet.

WE get the following methods:

We have to add the server address and the time zone the config e.g. secrets file (secrets_use_ntp.h):

    /****** WiFi SSID and PASSWORD ******/
    const char *MY_WIFI_SSID = "your_ssid";
    const char *MY_WIFI_PASSWORD = "your_password";

    /****** WiFi and network settings ******/
    // UDP logging settings if enabled in setup(); Port used for UDP logging
    const word UDP_LOG_PORT = 6464;
    // IP address of the computer receiving UDP log messages
    const byte UDP_LOG_PC_IP_BYTES[4] = {192, 168, 178, 100};
    // optional (access with UDP_logger.local)
    const char *NET_MDNSNAME = "UDP_logger";
    // optional hostname
    const char *NET_HOSTNAME = "UDP_logger";
    // if you want to use an NTP server
    const char *NTP_SERVER = "lu.pool.ntp.org"; // NTP settings
    // your time zone (https://remotemonitoringsystems.ca/time-zone-abbreviations.php)
    const char *TZ_INFO    = "CET-1CEST-2,M3.5.0/02:00:00,M10.5.0/03:00:00";

And the program giving us the whole time structure (you can choose what you need :)) and the time.

    /* esp_use_ntp.ino
      www.weigu.lu
      for UDP, listen on Linux PC (UDP_LOG_PC_IP) with netcat command:
      nc -kulw 0 6464 */

    /*!!!!!!       Make your changes in config.h (or secrets_xxx.h)      !!!!!!*/

    /*------ Comment or uncomment the following line suiting your needs -------*/
    #define USE_SECRETS  // if we use secrets file instead of config.h

    /****** Arduino libraries needed ******/
    #include "ESPToolbox.h"            // ESP helper lib (more on weigu.lu)
    #ifdef USE_SECRETS
      // The file "secrets_xxx.h" has to be placed in a sketchbook libraries
      // folder. Create a folder named "Secrets" in sketchbook/libraries and copy
      // the config.h file there. Rename it to secrets_xxx.h
      #include <secrets_use_ntp.h> // things you need to change are here or
    #else
      #include "config.h"              // things you need to change are here
    #endif // USE_SECRETS

    /****** WiFi and network settings ******/
    const char *WIFI_SSID = MY_WIFI_SSID;         // in secrets_xxx.h or config.h
    const char *WIFI_PASSWORD = MY_WIFI_PASSWORD; // in secrets_xxx.h or config.h
    IPAddress UDP_LOG_PC_IP(UDP_LOG_PC_IP_BYTES); // in secrets_xxx.h or config.h

    ESPToolbox Tb;                                // Create an ESPToolbox Object

    /****** SETUP *************************************************************/

    void setup() {
      Tb.set_udp_log(true, UDP_LOG_PC_IP, UDP_LOG_PORT);
      Tb.set_led_log(true); // enable LED logging (pos logic)
      Tb.init_wifi_sta(WIFI_SSID, WIFI_PASSWORD);
      Tb.init_ntp_time();
    }

    /****** LOOP **************************************************************/

    void loop() {
      Tb.get_time();
      Tb.log("\nHere is the time structure: ");
      Tb.log_time_struct();
      Tb.log("\nTo get e.g. the time use Tb.log_ln(Tb.t.time); and you get: ");
      Tb.log_ln(Tb.t.time);
      delay(2000);
      Tb.blink_led_x_times(3);
      if (WiFi.status() != WL_CONNECTED) {   // if WiFi disconnected, reconnect
        Tb.init_wifi_sta(WIFI_SSID, WIFI_PASSWORD);
      }
    }

Method for using millis() instead of delay()

As I often need millis() do do something non-blocking in loop() I wrote three helper methods for this.

They allow to control from 1 to 3 different events, and return true e.g. a number if time is up and the event should be executed.

Here a little example with 3 different events.

/* esp_non_blocking_delay.ino
   www.weigu.lu  */

    #include "ESPToolbox.h"            // ESP helper lib (more on weigu.lu)

    ESPToolbox Tb;                     // Create an ESPToolbox Object

    const byte PIN_LED2 = D1;          // Wemos D1 mini pro
    const byte PIN_LED3 = D2;          // Wemos D1 mini pro

    void setup() {
      Tb.init_led(false);              // Wemos D1 mini pro has neg. logic!
      Tb.init_led(PIN_LED2,true);
      Tb.init_led(PIN_LED3,true);
    }

    void loop() {
      switch (Tb.non_blocking_delay_x3(100, 200, 400)) {
        case 1:
          Tb.led_toggle();
          break;
        case 2:
          Tb.led_toggle(PIN_LED2);
          break;
        case 3:
          Tb.led_toggle(PIN_LED3);
          break;
        case 0:
          break;
      }
      yield();                         // feed the (watch) dog
      // do whatever you want here
    }

Using MQTT

Downloads