Intermittent Database Performance Issues
Update - We are continuing remediation efforts on this issue. We are also closely monitoring affected resources to ensure that the recent changes have been successful thus far. We are seeing improvements, and we will continue to provide updates as they become available.
Jan 26, 18:34 MST
Update - This issue is still actively being worked on as a highest-priority issue. The most recent efforts include building additional indexes, information gathering and tuning of slow queries, and upgrading our proxy instances to a memory-optimized instance type that can support more robust caching strategies.
Jan 24, 10:20 MST
Update - We are continuing to work on this issue. We have built new indexes on the affected database, tuned slow queries, and added more aggressive caching on high-throughput routes. We are seeing performance improvements which are leading to lower error rates as well.
Jan 23, 10:46 MST
Identified - We are currently working on a performance issue with one of our production MongoDB clusters. This issue is causing intermittent latency and error rate spikes. We are implementing caching improvements at the proxy layer and tuning queries at the application layer as part of our effort to remedy this problem.
Jan 22, 12:06 MST
API Operational
Website Operational
Shop Operational
Blog Operational
Database Degraded Performance
Third Party Operational
PagerDuty Notification Delivery Operational
AWS cloudFront Operational
AWS ec2-us-east-1 Operational
AWS sqs-us-east-1 Operational
Atlassian Bamboo Cloud Operational
Travis CI Linux Builds (container-based) Operational
Shopify Checkout Operational
Shopify Third party services Operational
Shopify API & Mobile Operational
Shopify Support Operational
Stripe API Operational
Stripe Webhooks Operational
Stripe Site Operational
Stripe JS Operational
Slack Operational
Stripe Emails Operational
Stripe Checkout.js Operational
Shopify Storefront Operational
LaunchDarkly API Operational
LaunchDarkly Feature flag CDN Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
Public API Availability
Fetching
API Response Time
Fetching
Website Availability
Fetching
Blog Availability
Fetching
Shop Availability
Fetching
Public API Apdex
Fetching
Past Incidents
Jan 27, 2020
Elevated API Error Rates
Resolved - The API error rates have returned to normal and users are no longer being affected by this issue. New caching improvements are currently in the review stage of our deployment process and we believe that they will be impactful in preventing another issue such as this. This issue has been resolved.
Jan 27, 16:35 MST
Update - We are continuing to monitor for any further issues.
Jan 27, 16:34 MST
Update - The elevated error rates, customers of the API's experienced, were likely caused by a large influx of cache updates. The servers were not able to keep up with the number of cache write operations which led to outages on highly trafficked routes. The issue is being addressed so that the cache will be more stable in the future. We are continuing to monitor this issue.
Jan 27, 15:14 MST
Monitoring - The error rates are continuing to stabilize and consumers of these APIs are experiencing fewer interruptions. We are closely monitoring this issue.
Jan 27, 14:21 MST
Identified - We have identified this issue and we are beginning to see error rates decline. We are continuing remediation efforts at this time.
Jan 27, 13:43 MST
Investigating - We are currently investigating this issue.
Jan 27, 13:23 MST
Jan 25, 2020

No incidents reported.

Jan 21, 2020
Elevated Error Rate on Website
Resolved - The affected resources have continued to report latency and error rate metrics within normal ranges. This incident has been resolved.
Jan 21, 00:30 MST
Monitoring - This issue is now under control and we do not expect users to be affected at this time. We have applied minor configuration changes and are working on a fix for the issue that we believe caused this incident. We will continue monitoring all affected resources.
Jan 20, 23:19 MST
Update - We are continuing to work on this issue. We are seeing intermittent latency spikes on workout APIs. During these spikes, some users may experience latency and intermittent errors. We apologize for the inconvenience and will provide further updates as they become available.
Jan 20, 19:35 MST
Identified - This issue stems from latency on one of our APIs that is leading to an increased number of concurrent executions and ultimately throttling on some lambda functions that consume said API. We are currently working on this issue.
Jan 20, 18:15 MST
Investigating - We are currently investigating an issue that is causing an increased error rate on the website. We are also seeing increased API latency at this time.
Jan 20, 17:44 MST
Jan 20, 2020
Elevated Error Rate on API
Resolved - The website and API error rates have remained at normal levels and we are seeing healthy latency metrics across all affected resources. This incident has been resolved.
Jan 20, 11:02 MST
Monitoring - We have identified this issue and error rates have returned to normal. The website and APIs are back up and reporting healthy metrics. We will continue to monitor this issue.
Jan 20, 09:23 MST
Investigating - We are currently investigating this issue.
Jan 20, 09:12 MST
Jan 19, 2020

No incidents reported.

Jan 18, 2020

No incidents reported.

Jan 17, 2020

No incidents reported.

Jan 16, 2020

No incidents reported.

Jan 15, 2020

No incidents reported.

Jan 14, 2020

No incidents reported.

Jan 13, 2020

No incidents reported.

← Incident History Powered by Statuspage