Site downtime

TerlObar's picture
TerlObar
August 22, 2012 - 10:50am
Just wanted to give everyone a heads up about this.  I would have made it a front page announcement as well but I didn't want to overwrite the announcement about the SFMan and I didn't want to wait until that one expired.

The web site access will be probably be spotty for several hours starting midday on Friday, Aug 24 and may be off for as much as a day.  The hosting provider (me) will be getting a network upgrade during that time.  My ISP (which here is a city utility) is rolling out upgraded networks and I volunteered to be a beta tester.  They contacted me yesterday to set up a service appointment.  They are coming out Friday to do the upgrade (which probably means some work on the hardware on the street and maybe a new router in the house).

For most of you, this will just be a downtime that you may not even notice if you're not constantly on the site like I am.  Beyond that you should not notice much change.  However, after the upgrade you will hopefully notice better download speeds for larger files as my upstream bandwidth will more than triple.  (The page load speeds aren't realy impacted by this as they are limited by the complex queries that go on in the database in the background.  That will only speed up if someone buys me a faster server Wink as the queries max out the CPU).

For those who want the technical details:
I'm moving from the base tier to the middle tier service which means my bandwidth is going from 12Mbps down/1.5 Mbps up to 20Mbps down/5Mbps up.  That was only a $10/month increase in cost.  They have a 50Mbps down/10Mbps up option as well but that would have more than doubled my monthly cost so I decided not to do that one.  (If someone wants to send me $40 a month I'll upgrade to the higher speed Smile).

@w00t:  This will probably mean my IP address will change for the first time in nearly 2 years.  I'll let you know if it does via e-mail.  If it does change, you'll need to update the DNS entry for the site since I don't have access to do so.  You should probably remind yourself how to change it just to be ready to minimize downtime.  I just reviewed my DNS provider's site to remind myself for all my sites.

I'll proabably be doing some physical (cleaning out dust, re-applying thermal paste between the heat sink an CPU, etc) and virtual cleaning and updating (new kernel version) of the server hardware and software at the same time but that should happen while it's off line and shouldn't add any extra down time.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine
Comments:

jedion357's picture
jedion357
August 22, 2012 - 11:27am
cool beans, I'll just draw something or write something and check back latter to submit it to the zine.
I might not be a dralasite, vrusk or yazirian but I do play one in Star Frontiers!

CleanCutRogue's picture
CleanCutRogue
August 22, 2012 - 12:43pm
If the site is a burden on you or your server, we could host it on Larry's servers, or back on my host.

I think that old hacked Drupal 5 foundation is past its time anyway, and I've grown in skill with php, drupal, and mySQL.  I should probably remake the site anew and host it on my host's servers, though that would be one hell of a data port job!  ouch!  If you recall, I worked a bit on a new platform and you helped test some of its features with us.  Maybe it's time to prioritize that project again.  Maybe the porting of data could be helped by the users of the site?

btw - I know you said it jokingly, but I do know and appreciate there is a cost to hosting this site.  I hosted it for several years, then it went over to Larry and then to your server.  It's good that it doesn't cost the community money... but I know you're the one enduring any possible burden.

I've always avoided - despite being offered - accepting donations.  I've avoided this legal gray area carefully because it dances around the limited legal permission I have which specifically precludes profit "even to recuperate operational costs"
3. We wear sungoggles during the day. Not because the sun affects our vision, but when you're cool like us the sun shines all the time.

-top 11 reasons to be a Yazirian, ShadowShack


Anonymous's picture
w00t (not verified)
August 22, 2012 - 1:22pm
Tom, thanks for giving the community a heads up. We'd be lost without our SF fix!  Smile

Just ping me via email or call and I'll update DNS, the sooner you know the better as DNS takes from 1 to 24 hours. Hopefully we can re-focus at the end of this year on sf.us 2.0 which is hosted on the DwD Studios servers. That will take the burden off your 89 Function Point, Level 4, 100kg computer! Foot in mouth

TerlObar's picture
TerlObar
August 22, 2012 - 1:43pm
Actually, it's not a burden, it's just piggybacked on things I already do, which is why I've never really worried about it.  I was just stating a fact.  My server is a quad core machine and the VM that the site runs on only uses one core.  So when it gets busy like trying to load the What's New page it just pegs out the single core for a few seconds.  A faster processer would make it go faster but not really all that much.

If we wanted to try to port the site to a newer version, I'll help but that really is up to you guys.  It's fine as is as far as I'm concerned.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
August 23, 2012 - 8:56pm
This upgrade has been temporarily postponed to a future date.  I had to cancel the appointment as I have to take my wife into the hospital at the same time tomorrow for an ultrasound on her gall bladder.  I don't know when the new upgrade time will be but I'll let people know.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
September 21, 2012 - 2:18pm
The network upgrade actually happened as planned back on the 24th.  I forgot to put up a final post.

However there will be some outages this weekend as well as I just got a new wireless router (well not quite.  It's "out for delivery" according ot UPS so it should show up in the next couple of hours) and will be reworking my home network that the server is connected to.  I'm planning on doing the switch over on Sunday afternoon (Sept 23).  I'm also planning on doing the physcial maintenance to the server then as well so if you cannot connect to the site during that time, be patient, it should be back up shortly.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
December 19, 2012 - 11:58am
As mentioned in the front page announcement, the site will be going on and off a bit for the next couple of days as I do some repairs.  I may need to pull a hard drive.  Hopefully the downtime will be minimal.

So here's what happenend.  It looks like near the first of the month the disk I do the nightly backups of this site (and others) died.  Or at least the file system became corrupted.  In any case it became unmounted and the backups were just being written to the main root drive instead.  That disk filled up on the 10th but I didn't notice it until today (I need to implement some better monitoring Embarassed).  The upshot is that I have backups from the 1st through the 9th (I can't trust the ones from the 10th as I don't know if they are complete) plus my offsite one from 16th.  Luckily this didn't affect the main site in anyway so we didn't lose anything and another backup will run fine tonight.

I don't think there is much on that disk other than backups going back a couple of years, so I'm not too worried about it since I have several other disks I can use for backups as well.  I don't look at its contents very often (i.e. I hadn't looked in months) so I'm not sure if I had anything else stashed there.  I'm going to be going through the exercise of recover to make sure however.  After which I'll try to reformat the drive or possibly just retire it (It's and old disk).

Anyway, I'll post updates here for those that are interested.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
December 20, 2012 - 12:12pm
Everything is back up and running.  I did end up pulling the offending drive so I could run checks and repairs on it in a different computer.  It is still testing now so we'll see what happens.  The good news is that it is still under warranty so if it is totally bad I'll get a new one for the price of postage Smile.

If the drive checks out, there will be another short downtime as I replace it into the computer.  Otherwise it will be a couple of weeks before the new drive is returned and has to go back in.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
March 4, 2013 - 12:28pm
Okay.  I think I've got everything back in place and all the bugs worked out for now.  I'll probably be migraging the website VM to a new technology as the one I was using on the older server isn't supported on the new on and there are better services out there but it probably won't happen for several months at this point as I poke around and try things out.  In the meantime, eveything should be back to normal.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
February 12, 2014 - 3:10pm
A few weeks ago I had a serious system crash that has caused me to rebuild the entire system of the main server that the site runs on.  The site itself actually runs in a virtual machine so when it crashed I was able to bring it right back up on a different system.  I've gotten to main system back up and running to my satisifaction and I'll be switching things back to that main system.  If you try to reach the site and it is down in the next day or so, It's probably because I'm in the process of the switch.  When I do the switchover it should only take about an hour or two (I have to shut down the VMs and copy them - about 40 GB of data - back to the main machine and then bring them back up).
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
February 13, 2014 - 11:05am
The switchback is complete.  Let me know if you notice any problems.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
November 15, 2014 - 1:09pm
For some reason my computer has been randomly locking up and I'm not sure why as I haven't changed anything.  So if you find the site down, that's why.  I'm trying to figure it out.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
December 30, 2014 - 5:18pm
I'm going to be performing a hardware upgrade on the computer that runs this site (and Frontier Explorer, Star Frontiersman, and the Star Frontiers Network sites) sometime over the next few days when I get a chance.  So if you try to visit any of those sites and it's off-line, that's why.

For those curious, I'm installing a new power supply and a discrete video card
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine

TerlObar's picture
TerlObar
December 31, 2014 - 9:03am
Upgrade complete.
Ad Astra Per Ardua!
My blog - Expanding Frontier
Webmaster - The Star Frontiers Network & this site
Founding Editor - The Frontier Explorer Magazine
Managing Editor - The Star Frontiersman Magazine