Outages are never fun and we've learned a few things with this one. Our last major outage (3 hr+ ingestion delay) was back in June 2017.

Our database (MongoDB) died approximately 5:19am UTC. The primary was stuck in a connection loop (out of threads for new connections). We got the alert, but it seemed to have self-healed thru migrating to its secondary. The primary resynced and everything appeared to be fine. The alerts turned back green. What we didn't realize was that between 5:44am and 8:20am UTC, there was a series of small hiccups (queue buildups, flushes, repeat) that we hadn't noticed. Rollbar complained about Mongo connection errors sporadically.

Over the next several hours, the primary kept failing again and again and we've had to restart the primary after it got stuck in its loop in subsequent failures. We've seen our fair share of Mongo errors and this one didn't seem to match anything we've seen before. During these small periods of outages, authentication would fail for new connections and that prevented ingestion of some log data. In most cases, once reconnected, missed logs during the disconnected phase would flush on reconnect.

As a logging company, naturally we log many things, including slow operations. We've started noticing that one of the queries was taking more than 50000ms to complete and it showed a query that was 900kb long, which seemed odd. It would cause the query queue to build up and our reconnection code to Mongo basically created thousands of connections until the server ran out of threads. We tracked it down to a worker that update host group members and realized it was trying to update more than 125,000 hosts in a single query for a single customer. Since that particular query only ran sporadically, it was a bit harder to track down. We've since put a 1000 host limit on that around 10am UTC and that resolved the issue.

To be honest, MongoDB has been a solid performer for us and we haven't had any major outages. We do use MongoDB everywhere and we'll investigate how to cache more Mongo data if/when it becomes unavailable. Over the next few days, we're going to reevaluate our reconnection mechanism to Mongo and ensure safeties where possible.