Espressif/Arduino Core library upgrade

[proddy]

Following up from #1804, after 3.7.0 is out look at migrating some of the core libraries and frameworks to see if it improves performance, memory consumption and the firmware binary filesize isn't too big.

• https://github.com/pioarduino/platform-espressif32 to use Arduino v3.1/IDF v5.3 (from v2.0.x/IDF v4.4.x)
• Web Server to https://github.com/mathieucarbou/ESPAsyncWebServer and then PsychicHTTP with SSL when it's mature
• MQTT client espmqttclient -> https://github.com/mlesniew/PicoMQTT (no more AsyncTCP)

[MichaelDvP]

The factory partition for 16M is already nearly full. When creating it the idea was to put only a small loader in, not the full emsesp as now. So better use symmetrical partition sizes (3.6.5 have different partitions, so it fits in the braking change of 3.7.0 if we change now). Should also be used in emsesp-flasher.
Suggestion:

# Name,   Type, SubType, Offset,   Size,    Flags
nvs,      data, nvs,     0x9000,   0x005000,
otadata,  data, ota,     ,         0x002000,
boot,     app,  factory, ,         0x480000,
app0,     app,  ota_0,   ,         0x490000,
app1,     app,  ota_1,   ,         0x490000,
nvs1,     data, nvs,     ,         0x040000,
spiffs,   data, spiffs,  ,         0x200000,
coredump, data, coredump,,         0x010000,

[proddy]

Are you suggesting we keep boot smaller and put in https://github.com/emsesp/EMS-ESP-Loader ? I'm fine with that. Now is the time for breaking changes.

It would mean changing a lot though, from platform.io, the bbqkees flash too, the Flash Tool and the Web Installer. And we probably need to put back in the WebUI the option to restart another partition.

PIO always uploads to boot but doesn't adjust the partition info, so sometimes, when you restart after and esptool.py, it will go back to the old firmware version. I think this is a bug in platformio that Jason2866 just fixed in pioarduino.

I've updated the notes I keep on the partitions here. Can you review it please?
https://github.com/emsesp/bbqkees_flash/blob/main/README.md#partition-explanation

(not to others; this is private repo so you may not have access)

[MichaelDvP]

Are you suggesting we keep boot smaller and put in https://github.com/emsesp/EMS-ESP-Loader ?

No, i suggust to use normal emsesp and make boot nearly same size as ota (now boot is 280000, i want to enlarge to 480000).
The loader is only usefull if we ran out of mem in 4M chips.

[proddy]

ok. why not make all app sizes the same size? This change will erase the nvs data (nrg's and bbqkees's codes). BBQKees codes should be ok as this is not yet in production. The energy counters, is this going to hurt people? If so, maybe make export to Json in the WebUI and upload. The settings will also go, but they can be backed up. I do think we need to do this asap in 3.7.0. And I really want 3.7.0 out in the next weeks as I'm going to be busy at work for the next 6 months

[MichaelDvP]

ok. why not make all app sizes the same size?

It's nearly same size (64k diff on 4,7M partition), but i don't want to move nvs1/spiffs and keep sizes/bounderies on round values.

[proddy]

ah I see. Ok, let's do it.

[proddy]

btw I tried the latest pioarduino 3.0.7 and used heap went down from 208->196, max alloc too, and bin size +280KB so I think we'll stick with Arduino 2.x until 3.1 comes out.

[MichaelDvP]

With pioarduino the network scan still is broken: "no networks found"

[proddy]

With pioarduino the network scan still is broken: "no networks found"

should be fixed in the 3.1.0 - https://github.com/espressif/arduino-esp32/issues/ 10281 hopefully

more info at https://discord.com/channels/1263397951829708871/1263397951829708874/1287027557677662261

[proddy]

We can try Arduino Core 3.1 and ESP-IDF 5.3 with

platform = https://github.com/pioarduino/platform-espressif32/releases/download/53.03.10-rc2/platform-espressif32.zip

but will crash without patching NTP and AsyncTCP. See https://github.com/espressif/arduino-esp32/issues/ 10526

[proddy]

I started porting over to Arduino Core 3.1 / IDF 5.3 and Pioarduino. I also used the latest AsyncTCP and AsyncWS libraries, which I know work. One nice thing about these new libraries is that they give us back 6-7KB of available heap.

But I'm getting stack dumps and reboots when started. It's hard to trace where or why it's breaking, and I have a feeling it memory memory-related, but I can't figure out why it used to work. If I take the backtrace and use for example:

addr2line -pfiaC -e .pio/build/s3_16M_P/firmware.elf 0x4037f57b:0x3fcb36e0 0x4037718f:0x3fcb3710 0x403771c9:0x3fcb3730 0x4037f759:0x3fcb3750

it doesn't tell me much. So very hard to debug. I'll keep digging.

@MichaelDvP FYI

It's in my branch at https://github.com/proddy/EMS-ESP32/tree/core_update_2108

[MichaelDvP]

First look: you have reduced the arduino loop stack to 4k (default 8k), this is critical low.
Scheduler stack is a bit low when using http-client, maybe enlarge or disable schedules for testing.
Afaik arduino3 have changed uart handlng, could be a test to enlarge the uart stack size.

[proddy]

Thanks Michael! I removed the -D ARDUINO_LOOP_STACK_SIZE=4096 and it runs again.

[proddy]

now I need to fix the Network scanning and add back the Tabs underline in the WebUI

[proddy]

Got it all working on Arduino 3.1/IDF 5.3 except it still takes up 50K more heap and max alloc, so really need to try and get the Hybrid working to remove the unnecessary components.

[proddy]

I pushed the PR (#2328) that changes the src code locations and uses the remote libraries for ESPAsync and ESPAsyncWebServer. In 3.7.2-dev-8.

Only yesterday I got a reboot of my EMS-ESP so hope it's not related. If it happens again, I'll roll back the changes

[Delta1977]

I'm now for 16h on dev8 without problems. No unexpected reboot after upgrade.