Submitted a ticket and awaiting a reply.

I’ll keep my phone off vibrate tonight so hopefully I’ll know when the problem is fixed. I sincerely apologize for the downtime.

EDIT @ 8:36am: Since the server techs cannot get the vps to boot up they are working on restoring a backup from this morning. There may be some data loss but it will be minimal if anything..

EDIT @ 9:15pm: The server was restored. But it appears that the techs used an older April 7th 2010 backup that was dated as this mornings. They are looking to see if there are other backups. I should know more soon. Sorry for the inconvenience this has caused.

EDIT @ 11:28pm: Server techs want to charge 80 euros an hr to do the restore because it’s “emergency out-of-hours support”, or we can wait until 8:30am. IMO they should waive the fee since they fucked up the restore in the first place. But we are only paying for reboots 24hrs apparently and nothing else..

EDIT @ 12:50am: Datacenter wouldn’t budge on the fee, so the correct restore begins @ 8:30am Dublin, Ireland time. Which is 3:30am -5 EST. So hopefully by the time I wake up tomorrow the accounts should be restored correctly.

———————————————- August 21, 2010 ———————————————-

EDIT @ 9:56am: Restore is complete of the correct date. Techs are going through the server and fixing little bugs here and there. Right now there are a lot of cPanel errors.

EDIT @ 10:12am: Having an issue with all the files being in read-only mode. Expect many more updates as is get them.

EDIT @ 11:52pm: Tech tried rebooting the server, it failed. We are currently waiting in the queue for a tech to take a coser look at whats causing it to not boot. Once it hopefully gets up I am going to scp (secure copy) a lot of files and mysql databases to the server where this blog is hosted.. Then a new vps is going to be setup and backups from this past Wednesday will be restored that cPanel runs locally. So any sql or files updated on thursday are gone. If you need them I suggest you send in a support ticket asap. I’ll be online all day today until this is all fixed.
The problem with the snapshots taken every night for vps’ is that corruption is always a risk in backups like these I am told. Currently looking into other backup options.

EDIT @ 12pm: Could be a couple hours before they get to us, they are going to try and get another tech in. They would have more on staff if this were a week day..

EDIT @ 2:26pm: So after getting the vps finally back up it had reverted back to yesterdays restore. This had the techs at the datacenter “mystified”. It will be at least another 4 -6 hours before the correct backup is re-restored. Then this past Wednesdays backups will be restored to a new vps. Again I apologize for all the down time..

———————————————- August 22, 2010 ———————————————-

EDIT @ 12:41am: It’s almost been 3 days now, I really do hate this. Currently there is no eta on such a complex job of re-restoring the August 19th backup, there are a lot of things that are corrupt.. I do hope to have this all resolved later on today.

EDIT @ 9:26am: Nothing new to report yet the techs are still trying to get the August 19th restore up and running. They hope to have something for done by the afternoon..

EDIT @ 10:24pm: The re-restore has begun, it’s about 100gbs so it is going to take a while to complete. After that the accounts have to be restored on a new vps that is ready to go once the server is. I’ll post more tonight if the re-restore finally completes.

———————————————- August 23, 2010 ———————————————-

EDIT @ 11:20am: Old vps is back up, they are just running tests to ensure its stable. Then the account transfers finally begin!

EDIT @ 1:38pm: vps is back up but please dont use it, the techs are working on getting the backups ready to do the restores. All the files are read-only still..

EDIT @ 2:44pm: Transferring the backups to the new vps slowly. More news as I get it.

EDIT @ 7:26pm: Backup was corrupt, vps couldn’t start without a technican being called in to try and repair the file system. Even after that it still looked fairly corrupt. So they stopped. Now we are waiting on another restore to finish tomorrow.
If you would like email for your clients tonight send me an email or fill out the support form include the domain. I’ll set them up shortly.
Currently setting up a temporary vps.

———————————————- August 24, 2010 ———————————————-

EDIT @ 11:08am: Transfers have finally started. Bad news is they are April’s backups. They are having a lot of problems getting the newer backups online. So any account that’s changed since April, which is most are on a big list to be tarballed and scp’ed to the new server. This of course will be from the latest backup they can restore.
I have updated all my nameservers to the new ips. Even xchost.biz, if your still using those nameservers please change them to ns1 and ns2.xc-host.com.
The new nameserver ips are:
ns1: 217.115.117.46
ns2: 217.115.117.113

EDIT @ 11:52pm: Account restores are going very slowly, the old vps is extremely slow and tends to crash a lot. So if any of your accounts haven’t changed since April 17, 2010 please im me or send me an email via the support page so I can take it off the long list of accounts that have to be restored twice. The restore continues in about 3 or 4 hours.

———————————————- August 25, 2010 ———————————————-

EDIT @ 7pm: Not sure where to start so I’ll begin with the good news. All accounts are restored from the April 17, 2010 backup. Also everything else works on the new vps. Since were back in April if any of your sites don’t work because of suPHP please send in a support ticket and I’ll fix it asap.
The bad news is that it will take a long time to get the newer backups restored, if possible. I was shocked when I read this too. It took our vps crapping out for the techs to realize that their backup system hasn’t been working for quite some time. That is why we’ve had so much trouble.. It takes an hour to get the only august backup that works to load, it works for all of 5 minutes and then it dies. Tomorrow the techs will continue to do their best and restore files. I myself am pushing to see if they can just grab mysql, as that’s what’s needed on most sites. Some people need files which might take a while to get..

———————————————- August 26, 2010 ———————————————-

EDIT @ 5:30pm: When I awoke this morning around 11:30am, I found out that a lot of accounts’ daily backups had been successfully transferred over. I was amazed. If any of your accounts were restored today I’ve most likely already let you know. The rest will be moved over (as long as nothing fails) in the next few days. I don’t currently have an eta on this.
The one downside is that I’m having to pay 50 euros for an hour of the techs time. Which is around $63 USD. So if you would like to help out by donating please send me an email via the support page. Thanks..

———————————————- August 28, 2010 ———————————————-

EDIT @ 6:12pm: Was informed this morning that if I want the transfers to proceed it will cost more than 50 euros an hour because it is now the weekend. I’m not made of money nor did I want to put up any of my own to begin with. I am forced to for any accounts that need data after the April 2010 restore. I’ve been able to get the number of accounts to be transferred over to 74. Which is a lot from 111, but it is still a lot to transfer. Each time the old vps dies, it’s another 50 euros to have the tech at the data center restore it. The restore process takes roughly an hour.
So again every reseller here has accounts being transferred from the old vps. If you could help me offset the costs of this it would be greatly appreciated. Here’s a paypal button to donate:

Transfers will begin on Monday. I will be up early around 6am or so to make sure things run smoothly.

EDIT @ 10:50pm: I’ve spoken a couple of my resellers so far asking which accounts don’t need to be transferred. If I haven’t talked to you I’ll do my best to. Likewise if I haven’t talked to you yet and your reading this please get in touch with me. I would really like to limit the amount of time and money I have to spend on this.

———————————————- August 29, 2010 ———————————————-

EDIT @ 9:38pm: Transfers start tomorrow around 4am -5 EST or 9am GMT. I’ll be up @ 6:30am to restore accounts..

Also our off-site backups have begun, the first transfer took 7 hours and 30 minutes. Backups on the vps will begin again once all the account restores are complete.

———————————————- August 30, 2010 ———————————————-

EDIT @ 3pm: Finished the first wave of accounts this morning. Now I have the rest of the 74. Restoring begins in about 10 minutes. So very happy this is finally over. It only took 10 days..

EDIT @ 5:24pm: There were a couple accounts missed during the latest transfer today, which is more money I have to spend. So if there are any accounts you need restored please check tonight. I cannot afford to keep doing transfers… Tomorrow will be the last date I transfer accounts from the August 2010 backup.

———————————————- August 31, 2010 ———————————————-

EDIT @ 9:52am: As far as I know all accounts that needed to be transferred have. If you have any issues please let me know asap. I’ll be gone for most of the day, but I’ll be checking my email periodically.

– Joe

Still no updates on the server move.. But things are underway..

This new script called status2k was originally introduced to me by mjz. Many, many years ago (pre xchost). I liked it but couldn’t afford the $20 or so it was to purchase it. I knew he had found it on wht (web hosting talk) so I eventually figured out his account name and spent a good 30 minutes going through every topic he ever posted in until I found it. Glad I did too because it’s changed A LOT since I last saw it Best money I’ve spent in a long time!!

For those of you completely lost check it out: http://xc-host.com/ss. It’s now a link in the navigation as well

– Joe

On a side note to Peter: I’ll fix the logout issue soon, since I have some spare time now.. EDIT: It’s tested and fixed!