Late last week after experiencing heavy user load we decided to upgrade our machine to 2Gigs of memory. Shortly after the upgrade took place the machine began to cause strange problems with rebooting. When we initially contacted our server provider thee informed us there had been some issues with power on our rack (the area where our server is located). These issues were intermittent over a two day period so for two days the true problems went unnoticed. During this period the server was continuously rebooting its self causing our mysql database to lock and corrupt some data which caused most of the errors you saw.
By Thursday it became apparent that something else very, very wrong was going on. Initially we thought it was a ram issue so that was replaced. When the problem persisted a new chassis was put in, the problem still persisted. Finally it was decided the entire server OS and hard drives needed to be wiped clean. For whatever reasons this took nearly two days as more hardware was swapped in and out. When the new OS was finally up and running the reboot problem was still there. After speaking to a higher level tech we were given a completely custom install, all of our hardware was replaced and the server was monitored for several hours. This morning (Sunday) we determined the box had enough uptime to indicate the problem had been resolved so we re-configured the machine and brought back the board data.
At this point we are about 95% certain our problem is gone, but because we dont know what the exact problem was (we expect a hardware software conflict), we cant be entirely certain. To alleviate future problems we have increased our backup frequency so there will always be a good copy of ScubaBoards data near by. Tomorrow we will be working with our server provider to try and understand the problem and the remedy a bit better; I will update you as to what is accomplished.
You have my most sincere apologies for the downtime and delay; hopefully you all had a nice weekend away from the computer and in the water!
Please discuss this issue at http://www.scubaboard.com/showthread.php?p=492314
By Thursday it became apparent that something else very, very wrong was going on. Initially we thought it was a ram issue so that was replaced. When the problem persisted a new chassis was put in, the problem still persisted. Finally it was decided the entire server OS and hard drives needed to be wiped clean. For whatever reasons this took nearly two days as more hardware was swapped in and out. When the new OS was finally up and running the reboot problem was still there. After speaking to a higher level tech we were given a completely custom install, all of our hardware was replaced and the server was monitored for several hours. This morning (Sunday) we determined the box had enough uptime to indicate the problem had been resolved so we re-configured the machine and brought back the board data.
At this point we are about 95% certain our problem is gone, but because we dont know what the exact problem was (we expect a hardware software conflict), we cant be entirely certain. To alleviate future problems we have increased our backup frequency so there will always be a good copy of ScubaBoards data near by. Tomorrow we will be working with our server provider to try and understand the problem and the remedy a bit better; I will update you as to what is accomplished.
You have my most sincere apologies for the downtime and delay; hopefully you all had a nice weekend away from the computer and in the water!
Please discuss this issue at http://www.scubaboard.com/showthread.php?p=492314
