tag:z3mvdbpvy7yh.statuspage.io,2005:/historyKeen Status - Incident History2024-03-28T11:40:35-07:00Keentag:z3mvdbpvy7yh.statuspage.io,2005:Incident/197217702024-01-30T10:21:40-08:002024-01-30T10:21:40-08:00Service migration<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:21</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>09:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>00:43</var> PST</small><br><strong>Scheduled</strong> - We will move our services to a new Kubernetes cluster.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/196518362024-01-08T18:16:36-08:002024-01-08T18:16:39-08:00Writing events might be delayed<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>18:16</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>18:00</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>17:40</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>16:56</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/192090302023-11-24T02:31:18-08:002023-11-24T02:31:19-08:00Writing events might be delayed and some queries might fail<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>02:31</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>02:04</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>01:54</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>01:48</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/183586932023-09-02T03:47:40-07:002023-09-02T03:47:40-07:00Writing events might be delayed and some queries might fail<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>03:47</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>03:01</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>01:48</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>01:27</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>01:06</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>00:50</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/180767812023-08-07T11:06:58-07:002023-08-07T11:06:58-07:00Some events streamed to S3 may be delayed<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:06</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>10:48</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>09:59</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/179066582023-07-20T03:14:43-07:002023-07-20T03:14:43-07:00Cached Datasets updates might be delayed<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>03:14</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>17:20</var> PDT</small><br><strong>Monitoring</strong> - Big backlog of data is being processed and some indexes updates might be delayed.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/173375052023-05-23T01:08:32-07:002023-05-23T01:08:34-07:00Writing events might be delayed<p><small>May <var data-var='date'>23</var>, <var data-var='time'>01:08</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>23:17</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>22:59</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/171952092023-05-09T10:57:59-07:002023-05-09T10:58:00-07:00Some actions may return errors<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>10:57</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>10:52</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>10:44</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>10:13</var> PDT</small><br><strong>Investigating</strong> - Some actions, especially ones that use access keys, may return errors. We're investigating the issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/169859052023-04-27T23:30:05-07:002023-04-27T23:30:07-07:00Query durations are higher than usual and some queries might fail<p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>23:30</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>16:03</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>14:51</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'>27</var>, <var data-var='time'>13:56</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/162202812023-02-27T02:03:51-08:002023-02-27T02:03:51-08:00Database upgrade<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>02:03</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>01:30</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>04:42</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled database maintenance during this time. We’re expecting partial outage of the UI. Create/Delete Access Keys operations may return errors as well. APIs might be slower than usual.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/162509832023-02-23T12:47:23-08:002023-02-23T12:47:23-08:00Kafka Inbound - messages are delayed<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>12:47</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>10:33</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>08:59</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>07:37</var> PST</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/162535482023-02-23T12:35:13-08:002023-02-23T12:35:14-08:00Writing events might be delayed<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>12:35</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>10:36</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>10:14</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/162189152023-02-23T03:11:20-08:002023-02-23T03:11:22-08:00Dashboards database upgrade<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>03:11</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>02:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>02:35</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled database maintenance during this time. We’re expecting partial outage of Dashboards for a short period. Other components won't be affected.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/160797032023-02-09T11:39:51-08:002023-02-09T11:39:52-08:00Events streaming to S3 buckets may be delayed<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:39</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>11:13</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:24</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:20</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/160792782023-02-09T09:45:26-08:002023-02-09T09:45:28-08:00Writing events might be delayed<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>09:45</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>09:40</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>09:30</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/156955852022-12-22T12:22:59-08:002022-12-22T12:23:00-08:00Kafka Inbound - messages are delayed<p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>12:22</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>12:19</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>11:08</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/123069022022-10-22T03:05:29-07:002022-10-22T03:05:30-07:00Potential data inconsistencies<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>03:05</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>09:14</var> PDT</small><br><strong>Update</strong> - Most of the inconsistencies have been fixed. We are still scanning the remaining outliers.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>04:05</var> PDT</small><br><strong>Update</strong> - We are still progressing with the fixing discovered inconsistencies.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>18:53</var> PDT</small><br><strong>Update</strong> - We have identified three Clients affected by the data inconsistencies. The required steps have been taken to address discovered issues.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>08:16</var> PDT</small><br><strong>Identified</strong> - The minor data inconsistencies impact a handful of customers. The inconsistencies are being fixed using data from backups.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>06:13</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/112212682022-09-22T08:02:46-07:002022-09-22T08:02:47-07:00Keen Website and Query API partial outage<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>08:02</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>07:29</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/112178872022-09-22T05:45:45-07:002022-09-22T05:45:46-07:00Writing events might be delayed<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>05:45</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>05:24</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>05:12</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>01:45</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/112195922022-09-22T05:45:35-07:002022-09-22T05:45:36-07:00Events streaming to S3 buckets may be delayed<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>05:45</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>05:24</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>05:13</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>04:51</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/107535202022-08-03T12:28:02-07:002022-08-03T12:28:03-07:00Our monitoring shows that streaming events to Keen is slower than usual. We are currently investigating the issue.<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>12:28</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>11:12</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>10:50</var> PDT</small><br><strong>Identified</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>08:52</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/100184542022-05-23T05:00:33-07:002022-05-23T05:00:34-07:00Kafka Inbound/Outbound upgrade<p><small>May <var data-var='date'>23</var>, <var data-var='time'>05:00</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>02:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>02:10</var> PDT</small><br><strong>Scheduled</strong> - Keen will patch the Kafka brokers in our Inbound/Outbound clusters. We will update Kafka version from 2.6.0 to 2.6.3.<br /><br />Please ensure your clients are configured to use multiple broker connection strings to allow for failover if a specific broker is under maintenance.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/99754252022-05-11T17:30:00-07:002022-05-12T06:43:34-07:00Stream and Query API degraded performance<p><small>May <var data-var='date'>11</var>, <var data-var='time'>17:30</var> PDT</small><br><strong>Resolved</strong> - You may observe longer response time and increased error rates</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/95582532022-03-16T12:56:32-07:002022-03-16T12:56:32-07:00S3 streaming is temporarily disabled<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:56</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:30</var> PDT</small><br><strong>Investigating</strong> - Due to an error in S3 streaming we had to completely disable this feature.</p>tag:z3mvdbpvy7yh.statuspage.io,2005:Incident/92732332022-02-15T03:48:40-08:002022-02-15T03:48:41-08:00Database Maintenance<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>03:48</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>02:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>02:38</var> PST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled database maintenance during this time. We’re expecting partial outage of the UI. Create/Delete Access Keys operations may return errors as well. To be clear, the rest of our API will be functioning properly during the maintenance period.</p>