tag:status.ease.io,2005:/historyEASE Status - Incident History2024-03-28T15:47:34ZEASEtag:status.ease.io,2005:Incident/203049922024-03-22T23:30:56Z2024-03-22T23:30:56ZDatabase Maintenance - March 22<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>23:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>23:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>22:10</var> UTC</small><br><strong>Scheduled</strong> - On March 22, 2024 at 4PM PDT (11PM UTC), Ease Database team will be conducting maintenance. This will last an estimated 30 minutes. We apologize for any inconvenience.</p>tag:status.ease.io,2005:Incident/200424722024-02-25T17:38:44Z2024-02-25T17:38:44ZDatabase Maintenance - Global<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>17:38</var> UTC</small><br><strong>Completed</strong> - Maintenance in completed.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>23:11</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>23:08</var> UTC</small><br><strong>Scheduled</strong> - Please note that on **Sunday, February 25 from 8:00 until 10:00 AM PST,** the EASE servers will be undergoing routine, scheduled maintenance. During this time period, you will be unable to access EASE.</p>tag:status.ease.io,2005:Incident/200424902024-02-25T16:49:09Z2024-02-25T16:49:09ZDatabase Maintenance - app.ease.io<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>16:49</var> UTC</small><br><strong>Completed</strong> - Maintenance is completed. Thank you for your patience.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>23:11</var> UTC</small><br><strong>Scheduled</strong> - Please note that on **Sunday, February 25 from 8:00 until 10:00 AM PST,** the EASE servers will be undergoing routine, scheduled maintenance. During this time period, you will be unable to access EASE.</p>tag:status.ease.io,2005:Incident/199064922024-02-04T17:03:27Z2024-02-04T17:03:27ZDatabase Maintenance<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>17:03</var> UTC</small><br><strong>Completed</strong> - We have completed and verified our database maintenance. Thank you for your patience.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>16:04</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>16:03</var> UTC</small><br><strong>Scheduled</strong> - We are conducting maintenance on our production database this morning from 8-10AM PST.</p>tag:status.ease.io,2005:Incident/195979312024-01-04T19:24:23Z2024-01-04T19:24:23ZConnectivity issues<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>19:24</var> UTC</small><br><strong>Resolved</strong> - We have been monitoring our endpoints for our EU environment and have not experienced any more issues. Our platform provider is investigating from their side.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>04:51</var> UTC</small><br><strong>Monitoring</strong> - The connectivity issues are no longer affecting our public endpoint. We are monitoring and communicating with our platform provider about the cause of the issue. We will update this as we have more information.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>03:04</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing issues with connectivity on our EU environment. We are investigating the issue.</p>tag:status.ease.io,2005:Incident/188467392023-10-19T12:04:12Z2023-10-19T12:04:12ZValeo service interruption due to issue during maintenance<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>12:04</var> UTC</small><br><strong>Resolved</strong> - The environment has returned to normal state and we are continuing to investigate the cause of the outage.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>01:29</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>01:28</var> UTC</small><br><strong>Monitoring</strong> - The cause of the issue was the inclusion of a new monitoring tool. We have removed the new monitoring tool and are investigating why this happened.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>00:07</var> UTC</small><br><strong>Investigating</strong> - During standard maintenance to the Valeo environment, we encountered an issue and are investigating the cause.</p>tag:status.ease.io,2005:Incident/187448192023-10-10T16:59:58Z2023-10-10T16:59:58ZWe are experiencing issues with our production environment<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>16:59</var> UTC</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>16:31</var> UTC</small><br><strong>Monitoring</strong> - We have identified the issue and have applied a patch. We are monitoring the environment.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>16:29</var> UTC</small><br><strong>Identified</strong> - We have identified the issue and are working on the fix. We will report when we have it in place.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>15:54</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.ease.io,2005:Incident/185517742023-10-04T18:43:49Z2023-10-04T18:43:49ZSystem outage<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>18:43</var> UTC</small><br><strong>Resolved</strong> - This incident was resolved. No other issues have been reported.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>03:58</var> UTC</small><br><strong>Monitoring</strong> - We believe that we have resolved the issue. We are monitoring the environment.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>03:41</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an issue with our production environment, app.ease.io. We are investigating the issue and will update as we have more information</p>tag:status.ease.io,2005:Incident/185086502023-09-15T09:14:44Z2023-09-15T09:14:44ZWe are experiencing issues with our production environment<p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>09:14</var> UTC</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>08:52</var> UTC</small><br><strong>Update</strong> - This issue has been resolved.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>08:36</var> UTC</small><br><strong>Monitoring</strong> - We have identified the issue and have applied a patch. We are monitoring the environment.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>08:35</var> UTC</small><br><strong>Identified</strong> - We have identified the issue and are working on the fix. We will report when we have it in place.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>08:08</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.ease.io,2005:Incident/183489112023-09-01T16:35:35Z2023-09-01T16:35:35ZWe are experiencing issues with our production environment<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>16:35</var> UTC</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>16:27</var> UTC</small><br><strong>Monitoring</strong> - We have identified the issue and have applied a patch. We are monitoring the environment.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:27</var> UTC</small><br><strong>Identified</strong> - We have identified the issue and are working on the fix. We will report when we have it in place.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>13:25</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing issues with our production environment. Our DevOps team is investigating and we will update as soon as we have more information.</p>tag:status.ease.io,2005:Incident/182027372023-08-20T16:43:40Z2023-08-20T16:43:40ZDatabase Maintenance<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:43</var> UTC</small><br><strong>Resolved</strong> - Maintenance is completed. Thank you for your patience.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:02</var> UTC</small><br><strong>Monitoring</strong> - We are currently undergoing some planned maintenance on our Global/EU environment. We will be in maintenance mode for 2 hours. This is for Database maintenance and we will let you know when we have completed the maintenance.</p>tag:status.ease.io,2005:Incident/181468972023-08-14T23:26:59Z2023-08-14T23:27:55ZScheduled Database Maintenance.<p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>23:26</var> UTC</small><br><strong>Resolved</strong> - Scheduled Database Maintenance Completed. Thank you for your patience.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>23:01</var> UTC</small><br><strong>Investigating</strong> - We are currently conducting scheduled Database maintenance. This should only take 30 minutes.</p>tag:status.ease.io,2005:Incident/178773092023-07-19T19:20:48Z2023-07-19T19:20:48ZExperiencing connectivity issues<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>19:20</var> UTC</small><br><strong>Resolved</strong> - We have monitored no further issues. Our development team is reviewing the health check and will update it accordingly.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>13:41</var> UTC</small><br><strong>Update</strong> - We have had reports of users able to access the application who were previously unable to connect. We are still investigating the cause of the issue, but it appears that a health check was not reporting the status properly for a web instance. We removed this web instance from usage and the issue resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>13:26</var> UTC</small><br><strong>Monitoring</strong> - We believe that we have found the cause of the issue. Our connection errors have discontinued, and we are monitoring closely.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>12:01</var> UTC</small><br><strong>Investigating</strong> - We are experiencing connection issues between our front end and our database. We are investigating the issue.</p>tag:status.ease.io,2005:Incident/174577282023-06-02T22:55:14Z2023-06-02T22:55:14ZMaintenance on our Global Environment<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>22:55</var> UTC</small><br><strong>Resolved</strong> - Our Database maintenance on our global environment is complete.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>21:57</var> UTC</small><br><strong>Monitoring</strong> - We are currently undergoing maintenance on our Global Environment. This maintenance is going on longer than expected, we have identified the delay and are continuing to move along.</p>tag:status.ease.io,2005:Incident/173073842023-05-20T00:00:16Z2023-05-20T00:00:16ZUndergoing Production Database maintenance<p><small>May <var data-var='date'>20</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>Resolved</strong> - Maintenance is complete.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>23:31</var> UTC</small><br><strong>Monitoring</strong> - We are undergoing production database maintenance. We will be in maintenance mode for 30 minutes.</p>tag:status.ease.io,2005:Incident/173017322023-05-19T15:03:57Z2023-05-19T15:03:57ZWe are experiencing an issue with our hosting platform<p><small>May <var data-var='date'>19</var>, <var data-var='time'>15:03</var> UTC</small><br><strong>Resolved</strong> - We have not had any reports of issues since we made the fix. Please contact support if you need any assistance.<br /><br />The cause of the issue was due to an issue with our Domain registrar. We have resolved the issue with them.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>14:09</var> UTC</small><br><strong>Update</strong> - We are seeing that our endpoints are now accessible in 90% of locales. We are continuing to monitor while the updates to our DNS are being propagated. We have had a few notifications from customers that the site is still not accessible but we have seen that it's coming back quickly in all regions.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>13:09</var> UTC</small><br><strong>Monitoring</strong> - A fix has been made. Our ease.io domains are coming back up. We are monitoring the situation.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>12:52</var> UTC</small><br><strong>Identified</strong> - We have identified the cause of the issue. Please use our easeinc.com endpoints for now instead of ease.io. <br />app.easeinc.com<br />global.easeinc.com<br />govcloud.easeinc.com<br />valeo.easeinc.com</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>12:23</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an issue with their DNS routing. We are working on the issue and will update this as we learn more.</p>tag:status.ease.io,2005:Incident/171642302023-05-06T18:12:35Z2023-05-06T18:12:35ZUndergoing Database Maintenance<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>18:12</var> UTC</small><br><strong>Resolved</strong> - We have completed our database maintenance. Should you experience any issues, please contact our support.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>14:28</var> UTC</small><br><strong>Monitoring</strong> - From 7AM PT to 11AM PT May 6th, we are undergoing database maintenance. Thank you for your patience.</p>tag:status.ease.io,2005:Incident/167512102023-04-04T19:36:27Z2023-04-04T19:36:27ZAll Findings List Issue Update<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>19:36</var> UTC</small><br><strong>Resolved</strong> - All Findings have been recovered. Thank you for your patience. If you are still experiencing any issues, please contact our support.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>17:32</var> UTC</small><br><strong>Update</strong> - We have restored the correct status on All Findings up until 5AM PT this morning. We are still working on restoring the correct status on Findings submitted between 5AM-9AM PT.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Identified</strong> - We are currently investigating a database error affecting the All Findings list view for customers who access EASE via app.ease.io. We are working on a fix for the issue and will provide updates throughout the day until it is resolved. We apologize for any inconvenience this may cause.</p>tag:status.ease.io,2005:Incident/152066142022-12-15T04:22:57Z2022-12-15T04:22:57ZUnscheduled maintenance<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>04:22</var> UTC</small><br><strong>Resolved</strong> - Our maintenance has completed. We are sorry for the inconvenience.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>23:13</var> UTC</small><br><strong>Identified</strong> - We are currently undergoing some unscheduled maintenance. We are working on data optimization and will update this status page as we move further in the process.</p>tag:status.ease.io,2005:Incident/151394022022-12-14T04:33:47Z2022-12-14T04:33:47ZDatabase Maintenance<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>04:33</var> UTC</small><br><strong>Resolved</strong> - The Maintenance is complete.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>03:36</var> UTC</small><br><strong>Identified</strong> - We are conducting some database maintenance.<br />The reason for this was to optimize the database which required that no new data be added during the optimization.<br />Thank you for your patience. Your data integrity is our priority</p>tag:status.ease.io,2005:Incident/125963342022-10-26T17:01:22Z2022-10-26T17:01:22ZWe are experiencing issues with our production environment<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>17:01</var> UTC</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>17:07</var> UTC</small><br><strong>Update</strong> - The minor change is complete and full functionality should be available to users now. We apologize for the inconvenience.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>Update</strong> - We are making a minor change that will cause some performance issues with the reporting module for 5-10 minutes. We apologize for the inconvenience.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>14:42</var> UTC</small><br><strong>Monitoring</strong> - We have identified the issue and have applied a patch. We are monitoring the environment.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>14:34</var> UTC</small><br><strong>Identified</strong> - We have identified the issue and are working on the fix. We will report when we have it in place.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>14:17</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing issues with our production environment. Our DevOps team is investigating and we will update as soon as we have more information.</p>tag:status.ease.io,2005:Incident/125945812022-10-24T13:54:25Z2022-10-24T13:54:25ZExperiencing a minor issue<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>13:54</var> UTC</small><br><strong>Resolved</strong> - We have identified the cause and have remediated it. The application should not experience any more performance issues.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>13:34</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing a minor issue. Our Devops team has made a change and the issue should clear up in a few minutes. We are monitoring the issue and will update when the issue is resolved.</p>tag:status.ease.io,2005:Incident/125067652022-10-22T22:00:26Z2022-10-22T22:00:26ZPlanned Maintenance<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>Resolved</strong> - Our maintenance has completed and the app.ease.io endpoints are active. This concludes our maintenance today. <br />Our changes today included database maintenance and optimization.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>20:43</var> UTC</small><br><strong>Update</strong> - GovCloud maintenance is completed and govcloud endpoints are now avaialble.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>Monitoring</strong> - We are conducting planned maintenance on our app.ease.io and govcloud.ease.io environments for the next two hours.</p>tag:status.ease.io,2005:Incident/113021252022-09-28T21:55:22Z2022-09-28T21:55:22ZAWS Outage<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>21:55</var> UTC</small><br><strong>Resolved</strong> - AWS has resolved the issue. Closing the ticket on our side.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>18:09</var> UTC</small><br><strong>Update</strong> - Update from AWS:<br />10:59 AM PDT We continue to see elevated error rates and latencies for invokes on API Gateway endpoints in the US-WEST-2 Region. While engineers continue to work towards root cause, we have deployed traffic filters from sources with significant increases in traffic prior to the event. As a result of these traffic filters, we are seeing a reduction in error rates and latencies, but continue to work towards full recovery. Although error rates are improving, we do not yet have an ETA for full recovery. The issue is also affecting API requests to some AWS services, including those listed below. Amazon Connect is experiencing increased failures in handling new calls, chats, and tasks as well as issues with user login in the US-WEST-2 Region. We will continue to provide updates as we progress.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>17:37</var> UTC</small><br><strong>Update</strong> - Update from AWS: 10:33 AM PDT [10:33 AM PDT] We are investigating increased error rates for invokes in the US-WEST-2 Region. We do not yet have a root cause, but are investigating multiple potential root causes in parallel. In addition, we are implementing filters on inbound traffic from a set of sources with recent significant traffic shifts, which may help mitigate the impact. We do not yet have a solid ETA, but will continue to provide updates as we progress.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>17:10</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>17:07</var> UTC</small><br><strong>Monitoring</strong> - We are being affected by the current AWS outage. Our users are experiencing intermittent connectivity issues on all environments. <br />We are monitoring the status of the issue and will update the status as we see changes in the AWS outage.</p>tag:status.ease.io,2005:Incident/111102712022-09-12T12:00:00Z2022-09-12T17:27:01ZApp.ease.io/beacon.easeinc.com Outage<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Resolved</strong> - Today from ~5:00AM PDT to 6:15AM PDT, we experienced an outage for our production environment. This was caused by a misconfiguration during yesterday's scheduled maintenance. We have resolved the configuration issue and the incident was confirmed resolved at 6:16AM PDT.</p>