rebelpeon.com

Blog

  • Post Mortem

    Anyways, it’s alive.  It may have taken a little longer than expected, but it’s back.  Hopefully. 

    I’ve rebuilt all the virtual machines, mostly from backups, so I didn’t actually lose anything, but I’ve also changed a lot of the layout behind the scenes.  This, along with ordering new parts, and the rest of life, has kept the site off longer than I would’ve liked, but so is life without enterprise level machines and support.

    So now it’s time for a post mortem on all this fun stuff.

    The week of April 17th is when this story will start.  Basically, the website kept going down and the server hosting it became unresponsive to everything but ping.  I couldn’t SSH into the box or actually log in AT the box or anything.  So, I’d simply restart it.  After this happened a few times, I started scouring the logs to see what exactly was going on.  Basically, I couldn’t find anything.  As you can remember from a previous post, I thought that I had the problem licked.  However, I had never actually seen an error message or anything telling me exactly what was going on.  I was just going on gut instinct. 

    So after figuring I fixed the problem, I went on with life, and it did work for quite a few days.  And then it started happening again.  So I decided to reinstall ESX thinking it may be a problem with that.  It still hung a few times, and since I couldn’t actually log in at the box, I decided to log in as soon as I rebooted the server and just leave it logged in.  Maybe something was being written to the display before it hung.  Well, the server worked for awhile, and then sometime on Sunday the 28th it went down again.

    At the time I wasn’t at home, and had to wait until I got home, which was around 10 PM.  I go to the machine, and sure enough, I have the first actual error I’ve seen.

    SCSI Host 0 Reset (PID 0)
    Time Out Again—-

    So, looking at the error, I thought that it may be the hard drive on SCSI ID 0.  Looking back, this was the first sign as to what was actually wrong.  I then replaced the hard drive and boot it back up.  The machine doesn’t go anywhere.  No ESX, no nothing other than trying to boot from the NIC.  Definitely not a good sign.  This was a RAID 5 setup, it should’ve recreated the array and everything should’ve been fine after I replaced the hard drive.  Well, apparently it didn’t want to do that, but it was too late now.  This was sign number two as to what the true problem was.  It was now 2 AM on Sunday, with work the following day, so I turned everything off and gave up for the night.

    The following day I attempt to fix it again.  Since I still wasn’t sure what was going on, and I wanted to rule out the RAM, I ran MemTest86+ on the machine for a few hours.  No problems found.  I tried to do an upgrade with ESX, but ESX told me it couldn’t find any of the old partitions or installs.  Great.  Well maybe it’s just the partition table that’s gone, and not all the data.  I found this great utility CD called the ultimate boot CD.  On it there’s a program called TestDisk, which can salvage Linux partition tables.  After having to mess with the boot CD awhile to get the MegaRaid SCSI drivers installed on it, I was off and running.  Needless to say, that didn’t work, no partitions found.

    Well, that means all the data’s essentially gone, since I was definitely not going to pay someone to get it back.  Thankfully I had started doing backups not more than 2 weeks prior to all this happening!

    The rebuild of all the virtual machines then commenced.  However, with the server hanging it took quite awhile in order to get everything back up and running.  What made it even more interesting was the myriad of errors that each hang would create.  Honestly, I don’t think I saw the same error more than twice the whole time it was down.

    During this time I also redid the setup to put all my machines in an Internal network behind an ISA server.  Right now there’s the external network (the internet), a perimeter network (my workstation, Binford’s workstation, and some misc machines that don’t need security), and then the internal network (all my enterprise level machines).  There is still one huge problem with this, but I’m still working on it, and it’s not a big deal.  Basically, from my workstation and Binford’s workstation you can’t ping the internal network unless the machine you’re trying to ping, pings out first.  It’s something to do with our messed up physical infrastructure, but hopefully I can fix it.

    Basically, this whole time was to try and get the site and back-end up to where it was prior to the problems, and also fix what was wrong.  The more and more it happened, the more and more I thought it was the SCSI card.  So I changed the channel that all the drives were on, and rebuilt the array.  Needless to say that didn’t help much, and so this past Sunday I bought a new Dell Perc 3/DC card on ebay for $61 shipped.  Yesterday it came in, and last night I migrated all the virtual machines off, installed the new card, rebuilt the array, reinstalled ESX, migrated the virtual machines back on, and then brought the machines back on.

    Right now we’re flying on the new Perc Card that has 112 MB more cache, and the ability to initialize an array in under 5 seconds as opposed to 100 minutes.  Hopefully we don’t see a hang.  Let’s all hold our breath, mkay?

  • Links 5/11

    It’s slowly coming back, but you’ll need to bear with me.  I’m still waiting on some hardware, which should come either today or tomorrow, but I’ve gotten the site back to where it was.  However, it may go down at any time, and the other page may come back up.  Hopefully it’ll stay up until I the hardware arrives and I take it down for the last time, and there will be a post mortem, it’s actually already started.  But without further ado, here are the links that I thought were good enough to keep even with the downtime.

     

  • Links 4/28

    • The Big Clip—Could be fun to have a few of those laying around.
    • Buy Electricity When Cheap, Use When Expensive

      The device, called GridPoint Protect, is the size of a small file cabinet and connects to the circuitbreaker panel. (The company also offers a lower-capacity version designed for homes, which costs $10,000.) A built-in computer powered by a Pentium chip will make intelligent purchase decisions, buying when prices are low, then storing the electricity for later use. That will make it possible to run your company during the workday with cheaper electricity that you purchased at 3 A.M.

    • Steve Jobs Endorses New Product Line—This is only news worth because of one quote from Steve: “Everyone wants a MacBook Pro because they are so bitchin’.”  With a base price for the 17” MacBook Pro at $2800, they are bitchin’ Steve, bitchin’ expensive.
    • Make Your Own Wine At Home
    • More Safe, but More Fatalities
    • Lots of Power in a Solstice—This is for the parents, who are purchasing a Sky.  I don’t understand the whole “drifting” thing, but all I can see is 1400 hp coming out of the Solstice’s engine.  Yes please!
    • Downside of Certs—They serve a purpose to get your foot in the door for interviews, same as a degree.  It’ll be nice when everyone understands this.
    • Stop the Madness!—Wow, that’s just scary.  Hello 80’s!
    • Personal Drug Use Legal In MX—Wow, just wow.

      The bill legalizes possession of 25 milligrams of heroin, 5 grams of marijuana (about one-fifth of an ounce, or about five joints), or 0.5 grams of cocaine — about half the standard sale quantity, though half-size packages are becoming more common. It also lays out the legal limits for possession of a wide array of other drugs.

     

  • Photography Shops

    The first day of the photography class the instructor mentioned that the best places to get your camera and supplies (if you already didn’t have one) was at B&H Photo or Adorama, both of which are internet shops located in New York.  At the time, I didn’t much think about it because I had used the same logic when I purchased my camera: cheapest reputible dealer (watch out for NY camera shops, you have to be very careful).

    However, the second week, one of the students in the film class mentioned that he had a hard time finding the tryex (know idea on the spelling) film.  Our teacher then stated how there used to be so many Mom & Pop photography stores, but that they’d all disappeared. 

    Huh, I wonder why?  Maybe it’s because people are told to not use them by their photography teachers.  I mean, come on, you can’t have your cake and eat it too.  You either support the local shops or you don’t and they go out of business.  Is basic economics really that complicated?

    Oh, by-the-way, the Central Camera store on Wabash in Chicago is only bested in chaos by Portillo’s during lunch or dinner rush.

  • Links 4/26

    • Original XBox Harder to Find
    • 1 TV Hour = 1 Bag of Chips

      The study followed 550 children aged 11 to 13 over a period of 20 months. For each hour they spent watching television, their food intake was found at the end of the period to have increased by 167 calories a day. (A packet of crisps contains around 180 calories, while a can of Coke has 140).

    • Monad is now the Windows PowerShell
    • Color Wheel Version 2.0.1
    • LightSnake—Images can be found on their site also. 
    • 16GB Credit Card—I have one of PQI’s really small USB flash drives.  It works great and is incredibly tiny.  I can’t wait for this to hit.

     

  • Under Construction Page

    Well, if you attempted to visit my page yesterday (probably even some into today), you were hit with an Under Construction page.  Basically, I did this so that I could put something up that explains what’s going on, and that it was that the server died, again.

    At least I know it’s definitely not the logs filling it up.  However, I still have no clue what’s going on.  In trying to troubleshoot, I found that the CDROM drive on the server was bad though.  Thankfully I have another, which was pretty much identical.

    I’ve also got a functioning monitor and keyboard hooked up to the machine right now, so hopefully I’ll be able to log onto it and see what’s going on.

    Here’s hoping.

    Oh yeah, and those with @rebelpeon.com email addresses, I had setup forwarding to another of your accounts, so you should’ve all gotten email during the downtime.  I’ve since switched it back to be delivered to rebelpeon.com correctly.  If it happens again, I’ll just keep switching them around so that there is no email downtime (or at the least very small amounts).

    Update 4/25 10:33 PM
    More downtime again.  I’ve finally reinstalled ESX to see what happens, but I still have no idea what’s going on.  I ran some memory tests, and that doesn’t seem to be the culprit either.  If the reinstall doesn’t fix it, the next thing I’m thinking is that it could be power related, so a new surge protector may be in order.

    For those using email, it’s still forwarding to another address.  I’ve just setup the website so I can easily tell if it goes down again or not.

  • Gallery RSS

    For those of you that never actually visit the site, and use some sort of syndication, there’s a new RSS feed on the site for my gallery.  The only downside to it is that I’ll probably be adding pictures in large clumps, so the 10 images it gets will be woefully inadequate. 

  • Links 4/21

    There needs to be a stupid tax edition.

    • GPS vs Your Own Eyes

      Despite warning signs on both sides of the road, and nothing but water straight ahead, local villagers have found themselves pulling an average of two cars a day out of the river for the past week. “When you ask what happened, they say, ‘My sat-nav told me it was this way,’” one resident told The Times.

    • Door-to-Door Free Breast Exam—Seriously, who would think this is real?!

  • Downtime(s)

    I’m sure some of you have noticed the downtime I’ve been having here.  I’m not sure what exactly is going on yet.  Basically my ESX host becomes unresponsive at around 1:30 AM and kills all the virtual machines with it.  I’m planning on playing with it some tonight, so there will probably be more downtime to come.

    Thanks for your patience while I deal with this matter.

    Update 4/21/2006 6:11 PM
    Ugh, more downtime as it happened again, but the good news is that I think I may have found the problem.  I’m not 100% positive, but I’m pretty sure.  I feel pretty stupid about this one too.  Basically, some of the log files had grown a wee bit large, and it looks like I was running out of disk space.  I guess I should setup a cron job to take care of that, eh?

  • Pharmaceuticals

    And my Dad thought that I wasn’t living in the real world because I was in IT; welcome to the world of pharmaceuticals. 

    Last night, Erin and I went to a dinner about Spiriva.  This was both of our first adventures into this sort of setting. 

    Erin made it downtown a little bit early for the drinks & appetizers part of the evening (6:30 PM), so we started out at Petterino’s.  I love that place.  Ed is by far the best bartender, ever, but I could be biased, because he knows who I am.  Anyways, after a Goose Island there, we headed over to Nick’s Fish Market.  Yup, that is where the Spiriva Dinner was hosted.

    Anyways, we walk in to Nick’s and they have the whole main dining room downstairs booked for the drug dinner.  They have us sign in and specify if we’re physicians or just in the healthcare profession.  I was tempted to check physician, but then they’d probably try to hunt me down to push their drug on me later.  The representative at the table made some small talk with us and then we headed into the dining room.  The main dining room encompased around 12 tables with 14-16 people per table.  We sat down and immediately a waiter was over asking if we wanted wine or anything else to drink.  We both had a glass of wine and waited for Erin’s friend from PT school. 

    Appetizers were served, the assorted platter, three to a table.  However, there were only 5 people at our table, so we gourged on the yummy bites.  Overall, the tables were about 75% full.  Not a bad turn out, but quite a few short of what the apparently expected.  We were asked what we would like for our main course at this time, Chilean Sea Bass or Filet Mignon.

    By this time, I think I was on my second glass of wine, and one of the representatives grabbed the mic and introduced the presenter Dr. Nicholas J Gross from Loyola.  He started into his speach and informed us that he was being paid by the pharmaceutical companies behind Spiriva to give the talk.  During this time the second course was served: Manhattan Clam Chowder. 

    Dr. Gross (man, that’s an awesome name) gave his speech about COPD and how, basically, Spiriva is a miracle drug.  All in all, his presentation lasted about 15-20 minutes.  Needless to say, I didn’t know what he was talking about a fair amount of the time, but I had Erin there to translate for me.  Not like it mattered though, it was purely marketing propaganda.  Nowhere were we told what the drug physically does in the body, or how it increases the FEV1 of patients, just that it does.  I was tempted to ask questions along those lines, but the food was too good.  Anyways, nearing the end of his presentation course three was delivered: Ceasar Salad.

    I think I was on my third or fourth glass of wine by now.  It’s hard to tell because the waiters didn’t actually let you finish a glass before refilling.  Talk about service.  Our fourth course came out, and I had picked the Chilean Sea Bass, while Erin had the Filet Mignon.  Talk about delicious. 

    Things started to wind down and Erin’s friend left, but we stuck it out for course #5: banana chocolate cheescake and coffee.  By this time we were both more than stuffed, but it was so good we couldn’t say no. 

    It was around 9:00 PM so we headed home after we had both finished our entire piece of cheesecake.  We stopped back by my office to grab my bag and then proceeded to walk home to try and burn off some of the food.  Man was it good though.

    Afterwards I told Erin, that whenever she gets an invite to one of these dinners again, we definitely need to go.  Granted, I was just along for the food, but she actually wanted to go for the information.  However, she was a little disappointed with the information in the end.  It was quite funny to see the different types of people there though.  The people that they were actually pitching the drug to were easy to spot.  They were the nicely dressed doctors.  Then there was everyone else, which mainly consisted of nurses, some of which were still wearing their scrubs.

    It was definitely a learning experience, and you figure that if only one doctor prescribes this drug to his patients, the whole evening is probably paid for.  I’m just hoping there’s some sort of mailing list that I can get on for other upcoming events.