TEST Status

asdasd

09/03/2014 11:35 PDT - We are seeing issues with our RightNet functionality, which is resulting in longer wait times for actions to complete, as well as different statuses remaining in queued for longer than normal, as well as errors being thrown for inputs or running scripts in general. We are currently investigating and working on correcting the issue.

09/03/2014 1:27pm PDT - RightNet functionality has fully recovered, however any script stuck in a "Preparing" or "Failed" status will need to re-run the script as it will not happen automatically at this point.  We will provide more information regarding this outage shortly.

RCA: At 11:20 AM PDT today, we had released new rightnet changes to production which caused the inability to execute scripts or send other requests. This behavior impacted both shards (us-3 and us-4) of production.  At Approximately 12:20 PM Ops identified the  problem and began fixing all the impacted systems.  At 12:59 PM all systems had been updated, and service was back to normal.  The failure was caused by an error in the code deployed that did not parse input values correctly.  We will be revisiting our no downtime release procedures to prevent issues like this in the future and apologize to those affected.

Another thing to note, is that related to this outage, you may have seen various tasks fail to schedule or execute scripts, particularly in bulk. A fix for this particular piece of the issue was pushed out at approximately 6:50PM PDT.