Resolved -
The latest Production build released this past Wednesday night/Thursday morning included the aforementioned fix that was implemented to help mitigate the original incident that started on February 3rd, as well as the second incident that occurred on February 13th. With that, the fix is now live on all of our web deployments.
We have been monitoring network traffic closely, and the results of our mitigations have been favourable. At this point, we are considering these particular incident(s) resolved. A post-mortem summary will be drafted, however we do not have an ETA on when that will be posted at this time.
Feb 20, 15:06 EST
Monitoring -
A fix has been implemented and is currently in the build process. Our aim is to push it live in the next Production build, tentative Wednesday, February 18th in the late evening (to avoid any potential service disruptions from occurring during the majority of our customers' peak business hours).
At the moment, the underlying cause of the incident appears to have mostly subsided and network performance has returned to a more stable state. We will continue to monitor the situation and update this incident with any new developments.
Feb 14, 14:50 EST
Identified -
We are aware of the situation resulting in many of our customers' sites being slow or entirely unresponsive. A fix has been identified, but will take time to implement. We will update this incident when we have more information to provide.
Feb 13, 18:55 EST
Investigating -
We experienced a massive burst in network activity that has forced us to run an emergency reboot of one of our Relational Databases.
Feb 13, 16:00 EST
Monitoring -
Mitigation attempts appear to be having a positive affect. Network impact has lessened, but not fully recovered. We will continue to implement mitigations and monitor the results.
Feb 13, 15:18 EST
Identified -
The issue has been identified and we are actively attempting to mitigate the impact.
Feb 13, 12:40 EST
Investigating -
We are currently investigating another occurrence of this situation.
Feb 13, 10:50 EST
Monitoring -
Network performance has begun to stabilize. Thank you for your patience, and we apologize for the inconveniences this has caused.
Feb 5, 18:51 EST
Identified -
The issue has been identified and a fix is being implemented.
Feb 5, 11:06 EST
Update -
We are continuing to investigate this issue.
Feb 5, 10:05 EST
Investigating -
We are aware of an ongoing issue that is causing intermittent performance degradation for our customers' websites. Our developers are investigating to identify the root cause, and we are actively monitoring the situation.
Feb 3, 17:27 EST