Description
The ESP32s clock is very inaccurate (eg 10 seconds in 40 minutes in one test) due to the default 150 kHz source. There is an option to use an internal 8.5 MHz source which requires recompiling the Arduino/ESP32 core. This is meant to be more accurate so worth a try.
The simplest fix we can put in is to connect to the internet and do a time sync using NTP or the mobile network every hour to keep this drift in check.
More complicated fixes are to either include a 32 kHz crystal on the board which takes 2 ESP32 pins, or to include a separate RTC and crystal which will need to be communicated with via I2C or SPI.