SQL105 and SQL310 are down
Since (presumably) September 13, the database servers SQL105 and SQL310 are down.
What are the symptoms of this issue?
Because of this outage, you may experience the following issues:
- Websites which use MySQL may show database connection errors, MySQL error 2002 and “No route to host” errors.
- The database counter in the control panel is empty.
- The MySQL Databases and phpMyAdmin sections in the control panel don’t work.
How do I know if I’m affected?
You can check which database hostname your account is on through the control panel. In the sidebar on the right hand side, you’ll see the field “MySQL hostname” with a value which looks like “sqlXXX.epizy.com”. If this value shows “sql105.epizy.com” or “sql310.epizy.com”, your account is affected by this issue.
Will this issue cause any data loss?
As far as we can tell, the hardware failure has not resulted in the data on the database servers being corrupted or lost. However, iFastNet staff was unable to extract the data from the servers before the hardware failed entirely. As soon as the broken parts are replaced, we’ll be able to say for certain what the damage is.
What caused this issue?
The servers of sql105 and sql310 are virtualized servers on the same physical server. This physical server suffered from a hardware failure, which caused both virtual servers to go down. Since this is an older model server, the necessary replacement parts were not in stock, so they had to be ordered first.
When will this issue be resolved?
The replacement parts have been ordered and should arrive soon. After that, the operation can begin to restore the database servers.
It is expected that the server will be restored some time next week, but I don’t have enough information to give a more accurate predication at this time.
UPDATE September 18
Database hosts sql105 and sql310 have been brought back online yesterday afternoon. As far as we know, the databases have been restored without any data loss.
The databases have been moved from the old, original hardware onto temporary hardware to restore the service for the moment. A new server is being prepared to become the final home for the databases. It is expected that the databases will be moved again later this week, which may cause a brief interruption to the databases while the connections are moved to the new hardware.
