The API error rates have returned to normal and users are no longer being affected by this issue. New caching improvements are currently in the review stage of our deployment process and we believe that they will be impactful in preventing another issue such as this. This issue has been resolved.
Posted Jan 27, 2020 - 16:35 MST
Update
We are continuing to monitor for any further issues.
Posted Jan 27, 2020 - 16:34 MST
Update
The elevated error rates, customers of the API's experienced, were likely caused by a large influx of cache updates. The servers were not able to keep up with the number of cache write operations which led to outages on highly trafficked routes. The issue is being addressed so that the cache will be more stable in the future. We are continuing to monitor this issue.
Posted Jan 27, 2020 - 15:14 MST
Monitoring
The error rates are continuing to stabilize and consumers of these APIs are experiencing fewer interruptions. We are closely monitoring this issue.
Posted Jan 27, 2020 - 14:21 MST
Identified
We have identified this issue and we are beginning to see error rates decline. We are continuing remediation efforts at this time.