tag:status.block.io,2005:/historyBlock.io, Inc. Status - Incident History2024-03-28T08:47:34-07:00Block.io, Inc.tag:status.block.io,2005:Incident/150464982022-12-12T23:06:43-08:002022-12-12T23:06:43-08:00Ongoing Connectivity Issues<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>23:06</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>18:24</var> PST</small><br><strong>Identified</strong> - Ongoing connectivity issues at our datacenter are causing an elevated error rate and delayed responses across our services. There is no ETA to a resolution, but you can track it here: https://status.linode.com/incidents/n7yt1l21qtyz.</p>tag:status.block.io,2005:Incident/136472232022-11-23T12:18:15-08:002022-11-23T12:18:15-08:00Intermittent Errors/Connectivity Issues<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>12:18</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>11:01</var> PST</small><br><strong>Identified</strong> - Datacenter connectivity issues will cause intermittent errors/delays in various components of our services. The issue is expected to last through November 24th as maintenance crews isolate and fix the cause.</p>tag:status.block.io,2005:Incident/136189102022-11-22T10:55:34-08:002022-11-22T10:55:34-08:00Elevated Errors<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>10:55</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>01:25</var> PST</small><br><strong>Identified</strong> - Ongoing datacenter connectivity issues are causing an elevated number of errors for API calls, delays in webhook delivery, and errors/delay in various UI functionality.<br /><br />This issue is expected to last a few hours.</p>tag:status.block.io,2005:Incident/110216952022-08-30T18:36:53-07:002022-08-30T18:36:53-07:00Unscheduled Maintenance: Transaction Index<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>18:36</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>17:18</var> PDT</small><br><strong>Identified</strong> - New incoming transactions will get indexed slower until this is resolved.</p>tag:status.block.io,2005:Incident/71933472021-08-26T16:04:25-07:002021-08-26T16:04:25-07:00Ongoing Database Maintenance<p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>16:04</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>15:21</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>22:28</var> PDT</small><br><strong>Update</strong> - Bitcoin address data for non-account addresses will be stale for the next 7-14 days as we re-index the larger Bitcoin blockchain.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>21:07</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>20:40</var> PDT</small><br><strong>Identified</strong> - This maintenance will affect querying data for external Bitcoin addresses and WebHooks.<br /><br />We will provide an ETA for this maintenance once an estimate is available.</p>tag:status.block.io,2005:Incident/75948982021-07-24T21:47:29-07:002021-07-24T21:47:29-07:00Ongoing Maintenance<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>21:47</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>20:50</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>14:42</var> PDT</small><br><strong>Identified</strong> - Some functions of the UI and API will be temporarily unavailable for 2-3 hours as we perform maintenance on our infrastructure.</p>tag:status.block.io,2005:Incident/72025492021-06-08T20:36:58-07:002021-06-08T20:36:58-07:00Indexer Maintenance<p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>20:36</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>20:03</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>17:31</var> PDT</small><br><strong>Identified</strong> - Indexer for Block.io account addresses is undergoing maintenance. ETA for resolution is ~2 hours.</p>tag:status.block.io,2005:Incident/71771162021-06-05T19:18:54-07:002021-06-05T19:18:54-07:00Unscheduled Maintenance<p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>19:18</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>18:53</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>18:38</var> PDT</small><br><strong>Update</strong> - Maintenance is 70% complete.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>18:18</var> PDT</small><br><strong>Update</strong> - ETA is ~45min. Apologies for the inconvenience.</p><p><small>Jun <var data-var='date'> 5</var>, <var data-var='time'>18:07</var> PDT</small><br><strong>Identified</strong> - ETA to resolution: 30 minutes.</p>tag:status.block.io,2005:Incident/65184042021-03-13T00:17:24-08:002021-03-13T00:17:24-08:00WebHooks and WebSockets Delayed<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>00:17</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>16:53</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>15:46</var> PST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>15:40</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.block.io,2005:Incident/63931372021-02-24T19:27:06-08:002021-02-24T19:27:06-08:00Delayed WebHooks/WebSocket Notifications<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>19:27</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>18:44</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.block.io,2005:Incident/63613212021-02-22T23:53:27-08:002021-02-22T23:53:27-08:00Litecoin Blockchain Indexing Outage<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>23:53</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>23:43</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>18:28</var> PST</small><br><strong>Update</strong> - Updated ETA is 8-10 hours from now.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>16:08</var> PST</small><br><strong>Identified</strong> - The Litecoin network's indexing is currently delayed and should return to normal in the next 5-6 hours. During this time users will experience delays in new incoming transactions' appearance via parts of the REST API.</p>tag:status.block.io,2005:Incident/60749672021-01-21T00:55:05-08:002021-01-21T00:55:05-08:00Network Connectivity Issues<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>00:55</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>23:22</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>22:22</var> PST</small><br><strong>Identified</strong> - We've identified the issue (downstream links). We're awaiting a remedy for this from the relevant ISPs.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>21:46</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.block.io,2005:Incident/52924712020-10-10T01:12:13-07:002020-10-10T01:12:13-07:00Blockchain Indexing Partial Outage<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>01:12</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>23:50</var> PDT</small><br><strong>Update</strong> - The infrastructure responsible for blockchain indexing will undergo some maintenance for around 60 minutes in the next 2 hours. Blockchain indexing is operating normally at this time.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>23:18</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 9</var>, <var data-var='time'>23:18</var> PDT</small><br><strong>Investigating</strong> - An issue at our hosting provider has affected blockchain indexing for external-incoming transactions at this time. ETA to resolution: 60 minutes.</p>tag:status.block.io,2005:Incident/48764972020-08-17T16:49:28-07:002020-08-17T16:49:28-07:00Network Connectivity Issues<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>16:49</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>15:12</var> PDT</small><br><strong>Identified</strong> - Existing network connectivity issues are due to CloudFlare's outage; see https://www.cloudflarestatus.com/incidents/hmc1qbk6sb3v.</p>tag:status.block.io,2005:Incident/46444562020-07-17T16:02:33-07:002020-07-17T16:02:33-07:00CloudFlare Outage<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>16:02</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>15:11</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>15:10</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:55</var> PDT</small><br><strong>Update</strong> - https://www.cloudflarestatus.com/</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>14:36</var> PDT</small><br><strong>Identified</strong> - CloudFlare and Google DNS outages are causing network connectivity issues globally, and Block.io is affected as well.</p>tag:status.block.io,2005:Incident/38599022020-04-09T20:45:22-07:002020-04-09T20:45:22-07:00Web Wallet Offline<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>20:45</var> PDT</small><br><strong>Resolved</strong> - Infrastructure instability for Web Wallet caused the issue. This is now resolved.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>14:26</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>14:17</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.block.io,2005:Incident/37010312020-03-17T19:13:13-07:002020-03-17T19:13:13-07:00Dogecoin Transaction Index Delayed<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>19:13</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>17:02</var> PDT</small><br><strong>Identified</strong> - Unscheduled maintenance for Dogecoin transactions indexes. ETA to resolution is 10pm Pacific Time. Bitcoin and Litecoin networks are unaffected.</p>tag:status.block.io,2005:Incident/37003552020-03-17T17:00:40-07:002020-03-17T17:00:40-07:00Partial Network Connectivity Issues<p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>15:28</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>14:10</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>17</var>, <var data-var='time'>14:08</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.block.io,2005:Incident/36766892020-03-12T00:45:04-07:002020-03-12T00:45:04-07:00Unscheduled Maintenance<p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>00:45</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>00:36</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.block.io,2005:Incident/36468662020-03-04T20:21:25-08:002020-03-04T20:21:25-08:00Unscheduled Maintenance<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:21</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>19:40</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.block.io,2005:Incident/32546362019-11-25T09:52:04-08:002019-11-25T09:52:04-08:00Bitcoin Blockchain Indexing Node<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>09:52</var> PST</small><br><strong>Resolved</strong> - On Nov 24 12pm Pacific, our indexing node for the Bitcoin blockchain experienced a hardware failure causing downtime for the Bitcoin blockchain-dependent operations of the API and UI wallet only. This has been fixed at this time, and we are taking appropriate steps to mitigate further occurrences such as this.</p>tag:status.block.io,2005:Incident/32031992019-11-11T17:43:47-08:002019-11-11T17:43:47-08:00Incomplete Indexing for Some Bitcoin Transactions<p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>17:43</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>11</var>, <var data-var='time'>03:36</var> PST</small><br><strong>Identified</strong> - We have identified an issue where some users' blockchain transactions might not be indexed properly on our side. This may cause outgoing transactions to fail with a "missing inputs" error. Only if you see this error are you affected. Normal operations are expected after Monday 11pm Pacific.</p>tag:status.block.io,2005:Incident/26652342019-07-06T15:43:48-07:002019-07-06T15:43:48-07:00Indexing Partially Offline<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>15:43</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>15:13</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>15:12</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Update</strong> - We're awaiting word on a hardware fix that's causing the current degraded performance. ETA is still around 1 hour.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>14:09</var> PDT</small><br><strong>Identified</strong> - Blockchain indexing for new events is partially offline. ETA for resolution is 1 hour.</p>tag:status.block.io,2005:Incident/26197362019-06-27T22:01:49-07:002019-06-27T22:01:49-07:00Infrastructure Maintenance<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>22:01</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>21:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>18:53</var> PDT</small><br><strong>Scheduled</strong> - We will be undergoing maintenance during this time.</p>tag:status.block.io,2005:Incident/24677972019-05-29T21:17:56-07:002019-05-29T21:17:56-07:00Unscheduled Maintenance<p><small>May <var data-var='date'>29</var>, <var data-var='time'>21:17</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>21:17</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>21:10</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>21:10</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>20:52</var> PDT</small><br><strong>Update</strong> - ETA is around 20 minutes.</p><p><small>May <var data-var='date'>29</var>, <var data-var='time'>20:19</var> PDT</small><br><strong>Identified</strong> - Indexing of new incoming transactions is currently offline. ETA to resumption is around 60 minutes.</p>