We have applied the caching, query, and instance-type changes that were mentioned in the previous status updates and have been seeing positive results. We have not seen further problems from this issue since the changes were applied. We are working on another large improvement that we expect to further boost caching performance. We will continue to keep this on our radar and will monitor it accordingly.
Posted Jan 31, 2020 - 16:38 MST
Update
We are continuing remediation efforts on this issue. We are also closely monitoring affected resources to ensure that the recent changes have been successful thus far. We are seeing improvements, and we will continue to provide updates as they become available.
Posted Jan 26, 2020 - 18:34 MST
Update
This issue is still actively being worked on as a highest-priority issue. The most recent efforts include building additional indexes, information gathering and tuning of slow queries, and upgrading our proxy instances to a memory-optimized instance type that can support more robust caching strategies.
Posted Jan 24, 2020 - 10:20 MST
Update
We are continuing to work on this issue. We have built new indexes on the affected database, tuned slow queries, and added more aggressive caching on high-throughput routes. We are seeing performance improvements which are leading to lower error rates as well.
Posted Jan 23, 2020 - 10:46 MST
Identified
We are currently working on a performance issue with one of our production MongoDB clusters. This issue is causing intermittent latency and error rate spikes. We are implementing caching improvements at the proxy layer and tuning queries at the application layer as part of our effort to remedy this problem.