High latency
Incident Report for iFit
Postmortem

An ongoing increase in traffic caused an overall increase in latency across much of our infrastructure. Certain high traffic, high through actions, such as recording and querying workout logs caused latency (and ultimately errors) across must of the iFit application.

In response, we increased the total size of the infrastructure to handle the increase in load. We also offloaded certain actions, such as querying activity logs, to separate infrastructure. We also tweaked various caching strategies to improve overall performance.

From here, we will continue to monitor. We do expect traffic to increase further in the near future. We are taking various steps at both the infrastructure and application levels to improve performance and tolerance as traffic increases.

Posted Nov 24, 2020 - 10:54 MST

Resolved
This incident has been resolved.
Posted Nov 24, 2020 - 10:32 MST
Update
Service levels have returned to normal, and have remained within standard operational limits for over 12 hours.
Posted Nov 24, 2020 - 09:48 MST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 23, 2020 - 14:35 MST
Update
We are continuing to work on a fix for this issue.
Posted Nov 23, 2020 - 12:27 MST
Identified
The issue has been identified and a fix is being implemented.
Posted Nov 23, 2020 - 12:25 MST
Update
We are continuing to investigate this issue.
Posted Nov 23, 2020 - 09:29 MST
Update
We are continuing to investigate this issue.
Posted Nov 23, 2020 - 07:38 MST
Investigating
We are investigating elevated latency on our database, API, and website.
Posted Nov 23, 2020 - 06:17 MST