It is finished.

The Philco Phorum is back online as of 11:09 am EST.

We're good to go and ready to launch!

This has been a very stressful week...entire site offline for six days...and then the week was capped off by my staying up until 3 a.m. this morning trying to get the Phorum back online, then giving up and calling for help from the phpBB support board.

Coming together quicker than anticipated...

The DNS nameservers are already pointing to the new host site!

Philcoradio.com is back on the air.

I am currently in the midst of trying to bring the Phorum back from scratch.

So long, WestHost...

The moving process has begun...

I've signed a two year contract with a new hosting company.

I've made multiple backups of the Philco Phorum, so I should be able to recreate it on the new server.

See you on the Web...really soon...

Yea!!

B A C K U P ! ! !

:-)

Philcoradio.com is back!

I just checked...and Philcoradio.com has returned!

Philcorepairbench.com is back online as well!

Now I'm off to backup the Philco Phorum before it goes down again...

Philco Repair Bench down

Well after being up for about a day and a half, my server developed journal errors and other degradations in the RAID array, according to an overnight status email from WH. I guess the hardware survived and was ok for a while but was not 100% evidently.

So, I'm offlline again until they build a whole new server and restore from backups again. They say 24 hrs., but . . .

PITA

Day 7...and STILL offline

The last time WestHost updated the status of server wsl03020 (which is where philcoradio.com resides) was yesterday at 5:40 pm EST - more then THIRTEEN hours ago!

At that time, the server was still "relinking."

In fact the latest update I see for any of their servers is 8:58 pm MST (10:58 pm EST) last night. No word since then.

Read it for yourself here.

I don't think it takes 24 hours for the relinking to take place - it seems to leave the impression that perhaps they gave up and went home. That's just one man's opinion. Of course I have no idea what is really going on there...but my patience is wearing very thin.

I think the technicians who have been working their you-know-whats off deserve a round of applause. But the customer service people who do not answer emails...management who is constantly trying to put a positive spin on this FIASCO...well, don't get me started.

I am just waiting to see if I can get a more up to date backup of my site's Philco Phorum, and then it's Goodbye Charlie and Have a nice day!

Correction on the Philcoradio.com server

I just learned that my site is on wsl03020, not wsl00012 as I had thought.

The latest news from the WestHost Network Status page is that wsl03020 has been restored, but "re-linking" is currently being run. Hopefully, once this process is complete, my site will be back online at which point I will make a new backup of my phpBB files...

Apparently, Chuck's site is on a different server since it has been up and down throughout the past several days. And the Antique Radio Club of Illinois site, also on WestHost, has been up and down as well, not totally "down" as Philcoradio.com has been.

Another update

Here is the latest update on WH server wsl00012, which I just found:

"* UPDATE * 8:40am 2/25/10 The data recovery experts are still working on recovering the data. Their initial prognosis looks good, but we'll have more information in a few hours. Please note: If they are able to recover the data, we're likely talking another 48 hours before we can restore it to a server. This is a very delicate process since the HDD is essentially 'dead'."

BTW, that's 8:40am Mountain Standard Time.

You can monitor these updates here:

http://netstatus.westhost.com/

No flames

I've had a few inquiries as to the fact my site is down, but they were all nice - no flames.

I posted to ARF pointing to this blog in case anyone was wondering what the hell happened to our sites. Knowing that ARF has a wide readership among antique radio enthusiasts...I felt that would be a good way to "get the word out" about what was going on.

About had it too . . .

Yes, I guess I got too cocky yesterday. :-)

I realize that WH cannot personally email every member about their site status, but everyone there is in Clown Car mode, so I don't expect timely missives.

Good thing The PRB is not my retirement plan. I have not heard anyone flaming me for missing my site since the debacle started last Sat.

The click to tocking.......

Unbelieveable!

Chuck, I had not checked your site since late yesterday afternoon or early yesterday evening.

It's unbelieveable to me that WH would have your site running normally...and then take it back down!

Like I mentioned, my entire site is backed up save for the Phorum. You can bet that I will be making my own regular backups of the Phorum henceforth.

Phorum

That's a whole lot better than from scratch Ron!
Let's see....what was it I said back then...?

Everyone will now have the same "Join Date" once it comes back.


C.

Update: I have an old backup of the Philco Phorum!

Well, folks, I fell a little bit better now, and here's why:

I found that I made a backup of the phpBB files that comprise the Philco Phorum back in June 2009.

This may be all that is left of the Phorum, depending on whether WH still has a backup of my site or not.

So...if I have to restore the Phorum from that backup, I will. I'd rather lose eight months of posts than lose the whole thing.

I'll keep you posted, right here...

PRB back off-line again

Since last night, my site has been pulled off again.
No reason given. I assume it's part of the Server Chess Game WH has going.
I may just bail with you Ron - keep in touch.

I've always kept a full "folder hierarchy image" of my site here at work and at home. Mainly makes fixes/changes easier by updating the backup file(s) and then just ftp-ing the new one over the old on-line. All my pages are static pages so I don't have a message database loss like you Ron. Major PITA for you, I know what you're facing. I ran the (now defunct) AWA message board for a while until it fizzled out on it's own. Just the number of support files and index files and data files was enormous, even for the short time it was active.

I suspect once WH settles and the finger pointing is resolved, we may see WH silently be absorbed into another ISP or just go 'Quietly into the Night."

Good luck to you. Starting fresh is not always a bad thing. Think Phoenix......(not Arizona...)

:-)

Day 6...still offline

Last night, WestHost posted the following news regarding two of their shared hosting servers:

"7:58pm MT 2/24/10 - These shared servers suffered a total loss, including the backup server the backups were located on. A few hours ago, we contracted a third party data restoration service (at a great expense) to attempt to recover the data off the backup drives. At this point, we are waiting to hear back from them to determine chances of full, partial, or no backup scenario. As soon as we get the official word, we will update this status post. The situation with this server is dire, but we are exhausting every avenue possible to retrieve as much data as we possibly can."

Folks, this means that there is a very real possibility that my site's Philco Phorum may be "gone with the wind."

I've made an executive decision regarding the future of my website.

If WestHost does not have it back up by Sunday, February 28, I will immediately contract with another hosting company and get my site back online. If this means I have to start the Philco Phorum over from scratch, so be it. I would rather do that than continue to be offline.

I have learned a valuable lesson from all this: Never, ever, trust any hosting company to back up your site for you! Do it yourself, and do so on a regular basis!

At least I have everything (other than the Philco Phorum) backed up, so I can start fresh if necessary. But rest assured, that the Phorum will be backed up by me on a regular basis henceforth.

More news...

It appears the outage occurred at a site in Utah, and that site is not affiliated with the SoftLayer network.

Interesting reading may be found here:

http://www.webhostingtalk.com/showthread.php?t=928683

http://forums.westhost.com/showthread.php?t=14067

http://www.westhost.com/docs/ConsonusSouthDataCenterIncident-2010-02-23.pdf (NOTE: This link requires Adobe Reader.)

Incidentally...I sent WestHost an email asking why they could not have a backup at another site. No repsonse as of yet.

Multiple datacenters? What a concept!

Chuck brought up a good point.

WestHost's site advertises multiple data centers - in Seattle, Dallas and Washington, D.C. Follow the link for more information; you will see a map showing three data centers.

So why has this incident impacted so many websites?

And why did they not have a plan to simply transfer the impacted sites to another data center?

This morning I received an email from a gentleman who works in the IT department of a major university. More specifically...his job is IT security and disaster recovery.

He told me that this university has a backup site in another state in case of an emergency such as the WestHost fiasco.

I, for one, would love to know why WestHost could not/did not do this.

PRB and PhilcoRadio.com

Just re-checked my site and ARCI - both still among the living.

Really sorry about your server Ron. Murphy hit you a low blow.

I thought I read in one of the WH emails that they have a co-location facitlity?

Best of luck to you but I'm still not trusting my server yet.....

And you're right about WH. There should be a few people that need to fall on their swords.

Chuck

Day 5: Still no idea when my site will return...

The latest word from WestHost, from an email I received late last night:

"Dear WestHost Client,

As you already know from the outage notice we emailed you earlier today, your account is residing on one of the 10 shared servers we are still working to repair. Of those remaining servers, we are very happy to report that the backups of your server were unharmed and are available for a complete restore of your account. We hope this news will clear up any concern you may have had regarding possible data loss.

The data on your server WILL require a full system restore from these backups. We are still working to repair some of these backup servers, as well as the necessary tools to perform this backup. Good progress is being made with these remaining backup servers and tools. We still anticipate that this process could take anywhere from 24 to 72 hours to complete."

In other words, we won't be seeing Philcoradio.com until Friday (2/26) at the earliest, and (maybe) Sunday (2/28) at the latest.

People should be fired because of this incident. WestHost should have a better backup plan as well - a rack of backup servers in a different location, I think, would be a very good solution.

In other news, Chuck's Philco Repair Bench seems to be completely normal once again; the pages and graphics load quickly, as they did before the WestHost disaster.

Philco Repair Bench is up for now . . .

Well, since 7:30PM -ish, my site seems to be stable up to now.
I'll give it another 24 hrs. before I call it "healed."
Antique Radio Club of Illinois site (same server evidently) also
seems to be up and running. Time will tell.

Chuck

PRB website not up

As of 6:40PM CST, it is not available.

It's gonna be ups & downs until the dust settles.....

Chuck

Progress Update - Philcorepairbench.com back up

Looks like Chuck's site (Philcorepairbench.com) is back up as of 5:25 pm EST. It is running "slower than molasses" on my DSL line, however. Perhaps the loading speed will return to normal soon, and maybe his site will stay up this time.

My site (philcoradio.com) is still completely offline.

Stay tuned for further developments...

By the way, I found out the classic 1939 RCA test pattern is also in the public domain, so I have presented it above. It represents the current state of my website - off the air for now...

And for an interesting read about the test pattern above, visit this website.

I'm Here as well

I'm here as well for the "duration."
I have not had any email complaints of my site being down, so far, as I have far less traffic than Ron does due to the Phorum.
I may or may not switch ISPs, depending on what transpires. Up to this time WH has been fine and steady for me.

I guess my mileage HAS varied... I'm waiting it out just like the rest of you.

Chuck

The two Philco sites may not be back until Friday (2/26)!

Here's the latest word from WestHost:

"As a result of the fire suppression system error which was triggered at the Data Center during a routine system check by one of the fire system vendors, critical hardware components were damaged across dozens of our servers. Our first course of action was to replace those damaged components. We were able to repair or replace almost every failed hardware component during the first 36 hours through a combination of our on-hand hardware supplies and an expedited hardware shipment from our primary supplier.

Unfortunately, many of these hardware problems impacted hard drives, causing hard drive failures. When this happens, data loss becomes an immediate focus. Retrieving lost data and restoring from backups can take up to 24 hours per server. This is where we are currently with a majority of our downed servers. Our backup process restores three servers simultaneously. We will get to each server as quickly as we can, but given the number of servers needing to be restored from backups, this places our ETA anywhere from 24 to 96 hours. In other words, it could very well be Friday of this week before all servers and accounts are back on line. If ANYTHING changes with regard to this timing (good or bad) we will immediately inform you. We profusely apologize for the .stops and starts., however, once one issue was resolved another problem surfaced which resulted in adjustments to the published ETA.

In addition to the failed hard drives, we have also found other dedicated and shared servers that are experiencing file system problems. While accounts can generally function on these servers, file system problems can negatively impact the performance of the server. We are aware of these servers and are tackling these issues as they become evident."

So all I can say at this point is, "Please stand by..."

(Public domain test pattern image by Emdub510, from Wikipedia)

Philcoradio.com down; Philcorepairbench.com down; When will we be back???

Hello, friends.

I started this blog to keep you aware of things happening at my website, Philcoradio.com.

As you know, my site has been down since Saturday.

Chuck Schwark's website, Philcorepairbench.com, has been up and down during this period. As I write this, his site is down also.

Both of our sites are hosted by WestHost.

Here's what happened, according to the WestHost people:

"On Saturday, February 20th at approximately 2:20 p.m. MDT, the server your account is on experienced some hardware failure as a result of an annual fire system inspection at our Data Center (DC). An inadvertent release of Inergen was triggered in the DC environment. The release was triggered by an actuator that was not removed as required in the DC's pre-test checklist. As such, the entire DC was impacted at some level, however not all servers were affected. This error was not the result of a mistake by WestHost employees, our hardware or systems. The mistake was triggered by an error from a vendor. With that said, we take responsibility and are doing everything we can to restore all systems completely.

There were multiple servers impacted by this outage; some of the servers have had damaged hardware replaced while other servers need to be restored from backups. A large majority of the servers that were affected have since had all services restored and are back online. We still have a few remaining servers that are experiencing problems. We are doing everything we can to restore those remaining servers as quickly as possible."

While I can sympathize with their plight, it is my opinion that a web site hosting company to not have a better backup plan than this is unacceptable. Believe me, I share your frustration at not being able to access Philcoradio.com or the Philco Phorum which I also operate.

I emailed WestHost asking when my site would be back up. They did not bother to reply.

So, here is what I am going to do about it:

Once WestHost has my site back up so that I may access the phpBB files that make up the Philco Phorum, I am going to switch my site to a different hosting company. This means that the Philco Phorum as well as my website may not be available for several days. But rest assured my site will be back, as soon as possible.

Thank you for your patience and understanding.