Dates:
Start Time: Monday, June 19, 2023, at 10:31 UTC
End Time: Monday, June 19, 2023, at 12:35 UTC
Duration: 124 minutes
What happened:
Users were being logged out of our WebUI frequently – within 1-2 minutes of logging in. Users could successfully login again, but the new session would also expire quickly.
Why it happened:
The cache of logged in users held in our Redis database was being cleared every 1-2 minutes. This caused all user sessions to expire and new logins to be required. We have yet to ascertain why the cache was being periodically cleared at frequent intervals.
How we fixed it:
We restarted the pods running the Redis database and the cache behavior returned to normal.
What we are doing to prevent it from happening again:
We will investigate further to learn why the Redis cache was being frequently cleared.