Workout Load Failure Postmortem

Date

2019-10-17

Summary

An issue with a caching server caused some API responses to return error messages instead of cached responses.

Impact

Roughly one third of HTTP requests for certain API routes would have received an error instead of the expected response. This would have affected several distinct services and applications, mostly involving consumer facing equipment.

Root Causes

A hard disk on one of a set of servers in a caching layer in front of our API servers malfunctioned, which caused that server to respond to HTTP requests with errors instead of cached responses.

Resolution

The server with the problematic hard disk was replaced with a server with a properly functioning hard disk.

Action Items

• Update our periodic health check of the caching layer servers to detect hard disk failures.
• Fix some improperly configured monitoring on the number of errors being returned from the caching layer, so that we receive an earlier notification if this type of issue happens again.