Scheduled jobs get stuck in DST transition window

[Pwuts]

Last Sunday, 4 graph execution jobs that were scheduled on Europe/ time zones landed in the European DST transition window and got stuck because of it. This also blocked the scheduler from responding to RPCs, breaking all our scheduling-related functionality.

This Sunday, USA has its DST transition. We need to make sure this doesn't happen again then or later.

I have traced this to agronholm/apscheduler#1021, so we need to either upgrade to fixed (pre-release) version v4.0.0a6 or contribute a fix for v3.11.0 of apscheduler.

[Pwuts]

I figured out a way to unstuck our scheduler manually in case of emergency:

1. Query all jobs (from the store) where next_run_time is the DST transition timestamp.
2. For each entry, run this logic to get an updated next_run_time and job_state if necessary:

import pickle
from datetime import timedelta

job_state_pickled = "0192837465afbecd..."  # remove the \x prefix if any

job_state = pickle.loads(bytes.fromhex(job_state_pickled))

if job_state["next_run_time"].fold:
    job_state["next_run_time"] += timedelta(hours=1)

    print(f"📋 Updated next_run_time: {int(job_state['next_run_time'].timestamp())}")

    job_state_pickled = pickle.dumps(job_state).hex()
    print(f"📋 Updated job_state: {job_state_pickled}")
else:
    print("✅ Not affected, no update needed")

3. Update these values in the job store

[Pwuts]

Status: hoping for a swift resolution here agronholm/apscheduler#1079