Summer is Here

Summer is most definitely here, and on the correct weekend no less!  As I type this, it is 81 degrees outside with 55% humidity.  Welcome to the midwest.

To kick off summer right, Erin and I went for a bike ride.  It wasn’t that hard to convince her to go, especially since she just got her new bike.  We started out in Lemont at Waterfall Glen, and then headed down to Lockport and did the I & M Canal trail between Lockport and Joliet.  Grand total, probably ~20 miles.  Not bad, especially since we put in ~20 miles earlier in the week.

So, I had never taken the I & M Canal trail that far South out of Lockport, which means I had never seen the old steel mill ruins (steel mill closed in the 1930’s) that are down near Joliet either.  Talk about awesome.  I definitely need to get my camera down there.  Certain parts looked like a bombed out town, and the old locks that are still in the old part of the canal are awesome too.  It’s interesting to see what it used to look like through various photographs, because it sure doesn’t look that way now.

Oh, and for all you Prison Break fans (Aaron), the trail goes right by the old Joliet state prison, which is the one in the show.  Definitely need to get pictures of that too.

Links 5/26

Sorry that some of these are kinda old, but I’ve been busy the last week, and haven’t been able to keep up with news.  Granted, that’s probably not a bad thing, eh?

Published
Categorized as links, web

Crappy Day

I really hope that yesterday isn’t as awful as yesterday.  It wasn’t even like I was that busy, it’s just the things that I was tasked with took about 5 hours too long for various reasons (piss-poor documentation, machine slowness from misc crap installed [production mind you], etc)

I was getting so angry that I actually had to go out for a walk to blow off steam before 2 late afternoon meetings.

Published
Categorized as life, work

A Sad Day

Today a coworker and I made the trek over to City Food as we have previously done about once a week for the past few months.  We were in search of the elusive Roast Chicken Club (RCC), which is, by far, one of the most delectable sandwiches ever to be found.  It started out as an ordinary visit…

But what was this?  The regular cashier, the jovial lady who made every visit enjoyable, was no where to be found.  Instead, there was a new, fairly unhappy and very confused new lady.  And the interior.  Where was the TV, the couch, the atmosphere?  It had all been replaced by cold, dark, two person tables.  And the soda machine wasn’t even functioning (unknown reason).

Thankfully the infamous RCC remained on the menu, and so we were steadfast in our quest.  Once arriving at checkout we ventured to ask, “Has this place changed management?”  “Indeed it has, but nothing has changed.”  We both stared at him blankly, since it was quite obvious things had changed.  Then we questioned, “But will there still be free Big Ass Fridays?”  We were, of course, inquiring about the free large drink Fridays.  The cashier and the man behind the counter looked at each other puzzled, and then sheepishly responded, “Of course.”  I wasn’t sure if they were offended by the language, or that we even bothered to ask.

The doubts surrounding the fate of City Food and the RCC raced through my mind.  Thankfully, for the time being, the sandwich had remained the same, and we were still only charged the $0.50 for our water (Wednesday specials). 

Hopefully on future weekly visits I find the same great sandwich at the same price, otherwise City Food will be dead to me.  Dead.

Special Olympics

Erin asked me about two weeks ago if I wanted to volunteer for the Special Olympics.  Now, honestly, I didn’t really want to, but I said yes anyways.  Two days ago was the day, and even before hand, I really wasn’t all the excited. 

So we get up and head over to Thornwood High School at 9 AM on Saturday.  And then we got there, and it was complete Chaos (yes, with a capital “C”).  We didn’t know where we were going, or what we were doing or anything.  In fact, Erin’s parents who had signed up were initially rejected when they came to the sign-in booth because they didn’t have their IDs.  Both Erin and I thought this was quite strange, seeing as how we weren’t asked for our IDs.

However, everything was worked out by one of the ladies that Erin has been working with at her clinicals for the past 10 weeks.  Apparently she’s been doing this for quite a few years and everyone knows who she is.  We then watch the opening ceremonies and then head over to the softball throw, which we were helping with.  There are basically 4 areas:

  1. Check-in—Calls out the heat numbers and people in the heats
  2. Escorts #1—Take the athletes from the Check-in area to queueing seats for the heats and put them in the correct order for their heat
  3. Escorts #2—Escorts the athletes from the seats to the actual throwing area and then from there to the awards area
  4. Recorder—Records the distances for each thrower, and then figures out the place standings for the heat
  5. Grunts—The people chasing after the thrown balls

Erin was a Recorder, because she’s responsible and stuff (and the person handing out jobs was one of her coworkers) and I was an Escort #1 with Erin’s Dad.

We all had a blast.  Even with the crappy weather, it was crazy fun.  We were all running all over the place and the athletes were great.  I would definitely recommend volunteering at an event like this to anyone. 

Published
Categorized as life

What I Learned

This is essentially part two of the post mortem on the server failure.  The first post was basically just me outlining exactly what happened, while this post will be about what I’ve learned.

1.  System State backups are not the greatest thing in the world.  In fact, they are pretty much useless except for a few key situations.  Basically, in all of the Microsoft Press Books for the MSCE tests (and well, just about any other study material), system state backups are thought of as Gods gift to backups.  In reality this is hardly the case.  In fact, after doing system state backups on all of my servers, the only one that actually worked after a restore is the domain controller.  Granted, this was because there was nothing else on the machine.

All the other machines had software installed when the backups were taken, and now after restoring the system state, the machines are in a weird state where they have all the registry entries for software that isn’t physically on the machine (registry gets restored).  Now, this would be great if I had backed up the whole machine, but I didn’t.  Oh, and don’t even get me started with a system state restore and IIS.  Put simply, your metabase that is restored from the system state, won’t function on your new server, because your machine crypto key is different.

2.  The physical network at the apartment is a mess, and it definitely limits our ability to do a lot of things.  It seems to be further limiting my ability to create a perimeter and internal network with ISA.  For an unknown (as of yet) reason, anything not connected to the bridge/switch that my ESX box is connected to, can not ping the 192.168.2.0/24 network which resides as a virtual switch on the ESX box, even with the static routes set.  What’s really making this aggravating is that if I initiate a ping from the 192.168.2.0/24 network to a specific machine in the 192.168.1.0/24 network, then everything works fine until that tunnel through ISA is closed.  However, once that tunnel is closed, nothing even hits the ISAs external interface, so it’s not really a tunnel through ISA, but a mapped route that’s appearing and disappearing.  Annoying to say the least.  If you feel like you want to help, or see a better explanation, feel free to check out my thread over at isaserver.org.

3.  WinSCP.  I can’t believe I haven’t been using this app with ESX before.  Setting up FTP can be a pain, and is a security hole, so being able to easily upload ISOs or whatever to the ESX box has been unbelievably helpful.

4.  Linux.  It’s amazing how much easier it is to learn things when you actually have a reason to, like when it’s broken.  Unfortunately, with a lot of the original problems I had I wasn’t able to reference them on google.  However, after thinking about it for a bit, and using basic troubleshooting skills, I’ve been able to solve all the linux problems.  Thankfully.

5.  The new Perc controller rocks.  The site is noticeably more performant, and it doesn’t take forever to initialize an array.  It’s amazing what a generation later and 112 MB of cache can do for you.

Post Mortem

Anyways, it’s alive.  It may have taken a little longer than expected, but it’s back.  Hopefully. 

I’ve rebuilt all the virtual machines, mostly from backups, so I didn’t actually lose anything, but I’ve also changed a lot of the layout behind the scenes.  This, along with ordering new parts, and the rest of life, has kept the site off longer than I would’ve liked, but so is life without enterprise level machines and support.

So now it’s time for a post mortem on all this fun stuff.

The week of April 17th is when this story will start.  Basically, the website kept going down and the server hosting it became unresponsive to everything but ping.  I couldn’t SSH into the box or actually log in AT the box or anything.  So, I’d simply restart it.  After this happened a few times, I started scouring the logs to see what exactly was going on.  Basically, I couldn’t find anything.  As you can remember from a previous post, I thought that I had the problem licked.  However, I had never actually seen an error message or anything telling me exactly what was going on.  I was just going on gut instinct. 

So after figuring I fixed the problem, I went on with life, and it did work for quite a few days.  And then it started happening again.  So I decided to reinstall ESX thinking it may be a problem with that.  It still hung a few times, and since I couldn’t actually log in at the box, I decided to log in as soon as I rebooted the server and just leave it logged in.  Maybe something was being written to the display before it hung.  Well, the server worked for awhile, and then sometime on Sunday the 28th it went down again.

At the time I wasn’t at home, and had to wait until I got home, which was around 10 PM.  I go to the machine, and sure enough, I have the first actual error I’ve seen.

SCSI Host 0 Reset (PID 0)
Time Out Again—-

So, looking at the error, I thought that it may be the hard drive on SCSI ID 0.  Looking back, this was the first sign as to what was actually wrong.  I then replaced the hard drive and boot it back up.  The machine doesn’t go anywhere.  No ESX, no nothing other than trying to boot from the NIC.  Definitely not a good sign.  This was a RAID 5 setup, it should’ve recreated the array and everything should’ve been fine after I replaced the hard drive.  Well, apparently it didn’t want to do that, but it was too late now.  This was sign number two as to what the true problem was.  It was now 2 AM on Sunday, with work the following day, so I turned everything off and gave up for the night.

The following day I attempt to fix it again.  Since I still wasn’t sure what was going on, and I wanted to rule out the RAM, I ran MemTest86+ on the machine for a few hours.  No problems found.  I tried to do an upgrade with ESX, but ESX told me it couldn’t find any of the old partitions or installs.  Great.  Well maybe it’s just the partition table that’s gone, and not all the data.  I found this great utility CD called the ultimate boot CD.  On it there’s a program called TestDisk, which can salvage Linux partition tables.  After having to mess with the boot CD awhile to get the MegaRaid SCSI drivers installed on it, I was off and running.  Needless to say, that didn’t work, no partitions found.

Well, that means all the data’s essentially gone, since I was definitely not going to pay someone to get it back.  Thankfully I had started doing backups not more than 2 weeks prior to all this happening!

The rebuild of all the virtual machines then commenced.  However, with the server hanging it took quite awhile in order to get everything back up and running.  What made it even more interesting was the myriad of errors that each hang would create.  Honestly, I don’t think I saw the same error more than twice the whole time it was down.

During this time I also redid the setup to put all my machines in an Internal network behind an ISA server.  Right now there’s the external network (the internet), a perimeter network (my workstation, Binford’s workstation, and some misc machines that don’t need security), and then the internal network (all my enterprise level machines).  There is still one huge problem with this, but I’m still working on it, and it’s not a big deal.  Basically, from my workstation and Binford’s workstation you can’t ping the internal network unless the machine you’re trying to ping, pings out first.  It’s something to do with our messed up physical infrastructure, but hopefully I can fix it.

Basically, this whole time was to try and get the site and back-end up to where it was prior to the problems, and also fix what was wrong.  The more and more it happened, the more and more I thought it was the SCSI card.  So I changed the channel that all the drives were on, and rebuilt the array.  Needless to say that didn’t help much, and so this past Sunday I bought a new Dell Perc 3/DC card on ebay for $61 shipped.  Yesterday it came in, and last night I migrated all the virtual machines off, installed the new card, rebuilt the array, reinstalled ESX, migrated the virtual machines back on, and then brought the machines back on.

Right now we’re flying on the new Perc Card that has 112 MB more cache, and the ability to initialize an array in under 5 seconds as opposed to 100 minutes.  Hopefully we don’t see a hang.  Let’s all hold our breath, mkay?

Links 5/11

It’s slowly coming back, but you’ll need to bear with me.  I’m still waiting on some hardware, which should come either today or tomorrow, but I’ve gotten the site back to where it was.  However, it may go down at any time, and the other page may come back up.  Hopefully it’ll stay up until I the hardware arrives and I take it down for the last time, and there will be a post mortem, it’s actually already started.  But without further ado, here are the links that I thought were good enough to keep even with the downtime.

 

Published
Categorized as links, web

Links 4/28

  • The Big Clip—Could be fun to have a few of those laying around.
  • Buy Electricity When Cheap, Use When Expensive

    The device, called GridPoint Protect, is the size of a small file cabinet and connects to the circuitbreaker panel. (The company also offers a lower-capacity version designed for homes, which costs $10,000.) A built-in computer powered by a Pentium chip will make intelligent purchase decisions, buying when prices are low, then storing the electricity for later use. That will make it possible to run your company during the workday with cheaper electricity that you purchased at 3 A.M.

  • Steve Jobs Endorses New Product Line—This is only news worth because of one quote from Steve: “Everyone wants a MacBook Pro because they are so bitchin’.”  With a base price for the 17” MacBook Pro at $2800, they are bitchin’ Steve, bitchin’ expensive.
  • Make Your Own Wine At Home
  • More Safe, but More Fatalities
  • Lots of Power in a Solstice—This is for the parents, who are purchasing a Sky.  I don’t understand the whole “drifting” thing, but all I can see is 1400 hp coming out of the Solstice’s engine.  Yes please!
  • Downside of Certs—They serve a purpose to get your foot in the door for interviews, same as a degree.  It’ll be nice when everyone understands this.
  • Stop the Madness!—Wow, that’s just scary.  Hello 80’s!
  • Personal Drug Use Legal In MX—Wow, just wow.

    The bill legalizes possession of 25 milligrams of heroin, 5 grams of marijuana (about one-fifth of an ounce, or about five joints), or 0.5 grams of cocaine — about half the standard sale quantity, though half-size packages are becoming more common. It also lays out the legal limits for possession of a wide array of other drugs.

 

Published
Categorized as links, web