Site Shut Down

[Admin]

These easiest way to dump mysql is across a network, once you have your new server up to date run mysql slave on it and let the old server run as master. when you are ready to go live just disconnect the old server.Doing it by files blows big time.

I wouldn't use linux's journaling file system in raid, asking for trouble.

 

My advice: Take tuesday off!

It may well be the "easiest" but still risky and extra costs. (remember I am still unemployed and every cent I can spare goes on this site), the planned method would have seen the site restored within half an hour as the 5th disk is even better with it being in the box and after the server was rebuilt, we would have been back live at around 4pm...I would even have had the time to tell the data centre to do it again with the most recent operating system...why put up with say Windows 95 when you should be using Windows 7...as an example

 

 

[fly_tornado]

Next time you need some short term off server backup let me know.

 

 

[Admin]

Next time you need some short term off server backup let me know.

Thanks FT...nothing against you but security is my biggest concern with where the data is and who has access to it...I won't let it leave my sight...so to speak

 

 

[bexrbetter]

they wiped our bloody backup disk

I can not believe how many times I have heard that over the years.

 

Thanks for your efforts Admin, it's appreciated.

 

 

[MarcK]

Ahhhhh I see your problem here. You should have done the upgrade at night don't you know that all the best it people are nocturnal.

 

 

[slartibartfast]

and new drive types of SSD (Stainless Steel Drives)

er - I know you meant to say Solid State Drives.

 

Not much call for rust-free disks.

 

No moving parts on SSD = no problem with rust.

 

 

[Phil Perry]

Well Ian. . . . .

 

All I can say ( And I realise it isn't much. . .) is thank you for your efforts.

 

Fortunately, I know eff all about the finer workings of computers and especially, specialised software, servers and hosting etc. . . . . but I know a lot of people who are involoved in this stuff, and I think I now know why most of them are bald, some at a really young age. . . . . One of my ex- students works for an international disaster management company in the square mile ( London that is ) and recently had to travel to the . . . . well, some islands out in the woop woop pacific and solve a lot of lost accounts due to a server / hard disk / storage system failure, and had to do it quickly before the mafia and a few other grim billionaires started sending unpleasant and antisocial people with silenced weapons to seek out certain high ranking banking executives . . . .

 

Anyway, I'm glad that you seem to have got the site up and running again even if it IS using windows 3.1. . .( ? ) well, it seems to work OK to me.

 

Thanks again ADMIN. And all who assisted ( if any ) I wonder. . . .does "Being a site moderator" pale into a certain amount of insignificance when compared to what you have just gone through. . . . . . . ?

 

Phil

 

 

[damkia]

So Ian,

 

How is your hired help's treatment going at Happy Valley?

 

( )

 

 

[Admin]

er - I know you meant to say Solid State Drives.Not much call for rust-free disks.

No moving parts on SSD = no problem with rust.

I can't believe I wrote stainless steal...not sure what I was thinking at the time...must be something in the water here at Happy Valley

The stupidity still continues...after a scathing email to the data centre saying how long it has taken me to get the server backup and sites uploaded, checked and working, they reply with "Please let them try and recover the data from the 5th disk for me"...WHAT, I think the horse has bolted and is half way to Timbuktu by now

 

 

[Admin]

An update...

 

We are going to have to go through this all again however it will have a much smaller impact on site down time. We have to go through it again to at least get back to an operating system that is recent however instead of doing it all again on our current server, we will be moving to a much better and faster server instead. Not only is the new server much quicker and more modern, but it will allow us to set the new server up exactly as we want it with the latest software, tested and fully operational before we migrate the sites over to it.

 

Migrating sites from one server to another is much quicker which means less impact on you the user. The only unknown at this stage is when it will get done. I would like to have it done during one night however I need to decide whether to do it myself, which means it can get done during the night, or engage my server admin person once he has set the new server up, to handle the migration as well. This means the migration would need to be done during the day on a Saturday or Sunday.

 

The specs on the new server are still being worked out but will either be:

 

LA based server:

 

Processor: i7 3930 (12 cores)

 

Memory: 16gb RAM

 

Hard Drives: 2x300gb Intel SSD in RAID1

 

Backup Drive: 1x1tb SATA II

 

Bandwidth: 10TB Premium Transfer

 

IP Addresses: /29 (5 useable IPs)

 

Port Speed: 100mbps

 

Las Vegas based server:

 

Processor: Intel Quad Core E3-1230 v2 (8 cores)

 

Memory: 16 DDR3-1333 ECC RAM

 

Hard Drives: 2x256GB Samsung 840 SSD in RAID1

 

Backup Drive: 1x1tb SATA II

 

Bandwidth: 10TB Premium Transfer

 

IP Addresses: /27 (29 useable IPs)

 

Port Speed: 1000mbps (GigE)

 

My concerns in deciding which server at this stage are:

 

LA server processor is desktop grade whereas LV server is server grade BUT LA server processor is faster than LV server

 

LV server ram is server grade compared to LA server

 

LV server port speed is 1gig compared to LA server 100mb

 

LA server in LA is closer to Australia than LV server

 

The LA server is with the data centre that stuffed up last weekend

 

LA server is $35 a month dearer

 

LV server has 10m/s longer ping time

 

I have never heard of the LV server data centre

 

Will keep communicating to you as to as and when things will happen

 

 

[Guest Andys@coffs]

The Samsung 840 SSD's are desktop machine SSD's not server class. Cant tell if the same issue with the intel description.........BTW one (but not the only) of the main reasons SSD's fail is that they run out of write cycles......when you RAID 1 an SSD and that failure occurs then your 2nd drive is within a cycle or 2 of the same thing occurring so the protection people inherently believe in with RAID 1 is not necessarily there when you need it..... Not sure that RAID 1 is necessarily the solution with SSD's. How full will the resultant volume be? the closer you are to full capacity the greater the potential issues....

 

Do you know what your write capacity to the physical HDD's is per month? from that you can work out if the write cycle thing is an issue or not. From memory 840's are the new bleeding edge Triple Level Cell (TLC) able to save 3 bits of info per cell. Previous Multi Level Cell, MLC (which was marketing superlatives gone wrong cause it was actually Double Level Cell, each cell holding 2 bits of info) or Single Level Cell SLC which holds a single bit per cell. As I recall SLC has the most write cycles available and MLC and TLC reducing on that further and further.......Its why most Server class SSD's are SLC today.... and most desktop machines will be TLC cause the $$$ will be reduced. Do you in fact need SSD's at all? is Disk the bottleneck in the server? In corporates SSD's tend to only be used on machines that are usually being used in a DB role that is being thrashed constantly .....I know you have MySQL but is it the bottleneck? Be careful of bleeding edge...they don't use that description for no reason!

 

Also worth pointing out that in physical HDD if you write zero's over a sector then that is what happens.....in an SSD writing zero's over a sector to remove any data that you may be concerned about from a privacy perspective does not result necessarily in a zero being written, rather the cell gets remapped to one that already has a zero in it because such a write is deemed by the SSD controller to be wasteful of limited write cycles..... It is the case that using a diskwipe program on an SSD is a complete waste of time.......To put that into context defence allows tightly controlled bit wiping treatments for traditional mechanical HDD's that contained classified info so it can be declassified, they allow only the complete destruction treatment to be applied to SSD's.....

 

BTW not suggesting that there is a right or wrong choice here just providing info that might or might not be useful if you didn't already know it....

 

Andy

 

 

[River]

An update...We are going to have to go through this all again however it will have a much smaller impact on site down time. We have to go through it again to at least get back to an operating system that is recent however instead of doing it all again on our current server, we will be moving to a much better and faster server instead. Not only is the new server much quicker and more modern, but it will allow us to set the new server up exactly as we want it with the latest software, tested and fully operational before we migrate the sites over to it.

 

Migrating sites from one server to another is much quicker which means less impact on you the user. The only unknown at this stage is when it will get done. I would like to have it done during one night however I need to decide whether to do it myself, which means it can get done during the night, or engage my server admin person once he has set the new server up, to handle the migration as well. This means the migration would need to be done during the day on a Saturday or Sunday.

 

The specs on the new server are still being worked out but will either be:

 

LA based server:

 

Processor: i7 3930 (12 cores)

 

Memory: 16gb RAM

 

Hard Drives: 2x300gb Intel SSD in RAID1

 

Backup Drive: 1x1tb SATA II

 

Bandwidth: 10TB Premium Transfer

 

IP Addresses: /29 (5 useable IPs)

 

Port Speed: 100mbps

 

Operating System: CentOS 6.3 64bit

 

Control Panel: cPanel

 

Las Vegas based server:

 

Processor: Intel Quad Core E3-1230 v2 (8 cores)

 

Memory: 16 DDR3-1333 ECC RAM

 

Hard Drives: 2x256GB Samsung 840 SSD in RAID1

 

Backup Drive: 1x1tb SATA II

 

Bandwidth: 10TB Premium Transfer

 

IP Addresses: /27 (29 useable IPs)

 

Port Speed: 1000mbps (GigE)

 

Operating System: CentOS 6.3 64bit

 

Control Panel: cPanel

 

My concerns in deciding which server at this stage are:

 

LA server processor is desktop grade whereas LV server is server grade BUT LA server processor is faster than LV server

 

LV server ram is server grade compared to LA server

 

LV server port speed is 1gig compared to LA server 100mb

 

LA server in LA is closer to Australia than LV server

 

The LA server is with the data centre that stuffed up last weekend

 

LA server is $35 a month dearer

 

LV server has 10m/s longer ping time

 

I have never heard of the LV server data centre

 

Will keep communicating to you as to as and when things will happen

G'day Ian, does the PM (Private Message) service still available as I'd like to offer some thoughts re your ISP/Server setup.

 

River

 

 

[Admin]

G'day Ian, does the PM (Private Message) service still available as I'd like to offer some thoughts re your ISP/Server setup.River

Yes it does Mate

 

 

[Admin]

Thanks Andy...I have discussed the SSD situation to death on several technology forums over the last few days which is why the SSD's are either the Intel 320's in one option or the Samsung 840's in the other.

 

I will say in terms of bottle neck, BOTH options are an over kill for what we basically need but in having such high spec'd, high end servers, we are afforded some luxury in terms of available resources not only for those random attacks but also room to grow...don't forget the chat room does use extra resources as well compared to a normal hosting environment.

 

The two main considerations are:

 

1. Latency is always a given when having US based servers delivering to a majority AUS user base so not only minimising that as much as possible (Aust based server has much less but higher costs and less appeal to International users), but the faster the data can be processed and come out of the box the better. By having the fastest possible components within the box helps to minimise the perceived delay in site use and what is in the box, compared to geographic latency is one thing that I can control and hence why the spec's are so very important.

 

2. We have been using our current server for some time now and everything has a life expectancy. Yes, I know that if something in the server breaks due to age, the Data Centre will replace it so it is not an issue in terms of cost but it does have an issue in terms of down time plus there is also the gradual degradation of components, however when you consider the requirement of getting data in and out of the box as fast as possible, newer and more advanced components, being newer, will hopefully provide a longer life span from today forward compared to continuing to use what we have been using for some time already. So for this very reason, moving to a newer box with newer components, will perhaps assist in ensuring a greater uptime guarantee than say having a processor or ram failure in maybe 1 month, 6 months or 1 year etc

 

Initial discussions commenced with the data centres providing boxes with only a single SSD. This was dangerous should a failure ever happen so I insisted that quotes be given for dual SSD's in RAID1 to cover an SSD failure. I also required a secondary SATA disk as an in box daily backup disk plus for even greater redundancy, backups are taken and downloaded to me on a regular basis should any catastrophe happen to the whole box

 

There is one main consideration over all that and that is cost...I can't afford to be throwing a lot of money, money that I don't have, for the most idealistic solution that perhaps what the Dept of Defence can afford

 

 

[Admin]

Update

 

The new server has been ordered:

 

LA based server:

 

Processor: i7 3930 (12 cores)

 

Memory: 16gb RAM

 

Hard Drives: 2x300gb Intel SSD in RAID1

 

Backup Drive: 1x1tb SATA II

 

Bandwidth: 10TB Premium Transfer

 

IP Addresses: /29 (5 useable IPs)

 

Port Speed: 100mbps

 

Over our current server we will be going to not only a much faster processor but also an extra 4 processor cores (12 cores compared to 8 now) and much faster hard disks with the use of SSD's and staying in LA so no impact on latency for users.

 

Just have to wait for them to build it so I can get it set up.

 

 

[Guest GraemeM]

UpdateThe new server has been ordered:

LA based server:

 

Processor: i7 3930 (12 cores)

 

Memory: 16gb RAM

 

Hard Drives: 2x300gb Intel SSD in RAID1

 

Backup Drive: 1x1tb SATA II

 

Bandwidth: 10TB Premium Transfer

 

IP Addresses: /29 (5 useable IPs)

 

Port Speed: 100mbps

 

Over our current server we will be going to not only a much faster processor but also an extra 4 processor cores (12 cores compared to 8 now) and much faster hard disks with the use of SSD's and staying in LA so no impact on latency for users.

 

Just have to wait for them to build it so I can get it set up.

If you build it, they will come.

 

Graeme.

 

 

[Admin]

Update...just for the technically minded gurus out there...

 

Traditionally we have used generic mySQL as the database system along with predominately innoDB tables with the odd myISAM table where text searching was necessary. We also use an advanced search system of ElasticSearch tuned to the needs of this site.

 

I have decided to move to Percona mySQL due to its faster performance on innoDB engined databases but this will incur an extra couple of days delay due to its implementation and testing...again this should help speed the site up even faster...all these little incremental enhancements, processor, hard disks, database etc will all provide the best possible performance of the site to you the user.

 

 

[Guest GraemeM]

 

Thanks anyway, it is all very much appreciated.

 

Graeme.

 

 

[Admin]

UPDATE

 

Firstly, thank you for all your comments of appreciation...it is those that makes it all worthwhile doing this for everyone that gets some benefit from what this site/resource provides to everyone.

 

In line of keeping you updated with communication on what is happening with the site...I can advise that the new server has now been built and commissioned ready for set up and the migration of the site over to it.

 

I am currently talking to the server admin guy that will be performing this work, different one from before but this guy runs a data centre filled with sites that use the same software as us. Unfortunately he is in Canada and busy running his own data centre so it will all happen at a drop of a hat the moment he gets the time...sorry that I can't give a day or two warning but I will try and give as much warning as possible.

 

 

[Admin]

I have been advised that the site will close possibly around 1pm Melbourne Australia time so it can be migrated over to the new server. Sorry but I can't be more specific with the time as it depends on the setup of the new server and resource availability. It is expected that once the site closes, the migration from one server to another, with both being in the same data centre, should only take a couple of hours...famous last words

 

 

[Admin]

We are now on the new server and wow, it is fast...hope you can see the difference after browsing the site a bit letting the cache do its bit and once again, my apologies for any inconvenience BUT we should now be set up for a very long time

 

 

[slartibartfast]

It certainly is fast.

 

The next page comes up before the mouse click reaches my ears.

 

Well done on a painless transition.

 

 

[Admin]

Still gota few teething problems which is going to take till Canada wakes up to get fixed plus the DNS propagation to fully happen...they ended up putting a Intel Xeon e5-1650 processor in instead of the i7 so we now also have ECC processing plus 16gig of 1600mhz Registered ECC ram...ECC - Error Correcting Code which is Server Grade

 

 

[Admin]

Seems the DNS has resolved for me now and the site is screaming along...remember if you are still seeing the site on the old server, which is displaying a message that it is closed and a link to here, then please clear your cache in your web browser which will help it to go to www.recreationalflying.com here on the new server

 

 

[damkia]

I am currently talking to the server admin guy that will be performing this work, different one from before but this guy runs a data centre filled with sites that use the same software as us.

So, how is the old guy's Happy Valley stay going?