Update Regarding Ergo SmartPool Outage
Many of our SmartPool users are experiencing an extended outage on Saturday June 18th that began approximately 9am EST and has been intermittent for most of the day.
This post hopefully serves as clarification of what is occurring, and how this impacts our users and miners.
Beginning approximately 9am EST, many users of the GetBlok.io SmartPool were unable to access the User Interface at https://ergo.getblok.io. Upon investigation, our team identified an unprecedented traffic on our UI.
Our framework is based on highly distributed cloud-based architecture composed of various layers to handle traffic, availability, and security.
The public facing layer which is responsible for presenting data to users is API based, and provides statistics to our User Interface framework, such as hashrate, miner statistics, payout information. It also receives requests from users to facilitate settings and subpool selection.
In the past 48 hours, our service has grown 5x from our original hashrate (~150GHs) to current state (1.5 THs), 200 miners to 1000. This has required us to scale our infrastructure even further, globally, which we have done in a timely manner. Additional instances of our stratum, api endpoints, and web services were deployed in each of our points of presence (currently 7 regions).
Unfortunately, these efforts did not suffice at addressing the User Interface issues, as we narrowed down the problem to an unsustainable Distributed Denial Of Service (DDoS) attack at one of our public API Endpoints.
This attack has caused a slow down in all of our processing, including block and payout processing due to the unexpected impact in performance. We continue to work with our cloud provider to mitigate this issue.
Here at GetBlok.io, we always strive to ensure ALL of our services were made public, and this means access to our APIs for data extraction and analysis by our user base. This, after all, is the reason for the Smart Contracts-based mining pool operation.
As of 12:04 am EST 6/19/2022, our team is provisioning new API endpoints and additional measures to handle this issue going forward. Meanwhile, the following is important to understand:
- Your mining shares are still being processed and handled accordingly for payout processing. The shares handling is separated on purpose from API handling to prevent this very exact problem.
- Blocks found are still being processed for payout. Due to the impact to our availability, we are slowing making through all the blocks found to pay our miners. As of this writing, we are 30 blocks behind in processing payouts.
- Every block reward owed will be paid out. This is a matter of time, and unfortunately processing smart-contracts based blocks is not a matter of simply sending payment to miners, as it is locked into smart contracts and has to be processed on chain.
- Our payouts remain to be much faster the standard 720 confirmation timing (24hrs) on the Ergo Blockchain
We continue to answer and support questions in our Telegram and Discord channels. We realize that you have put a trust in us for mining in the ERGO blockchain, and we do not take it lightly. We plan to have a full post-mortem of what occurred, including lessons learned and what is being implemented to prevent it again.
Sincerely,
The team at GetBlok.io
