Category: work

SQL Server Issues

Last week I was beating my head against the table, because a VM I had quickly created wasn’t allowing SQL to install.  I kept receiving the following error in the detailed SQL error log:

Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.
External component has thrown an exception.
The configuration failure category of current exception is ConfigurationFailure
Configuration action failed for feature SQL_Engine_Core_Inst during timing ConfigRC and scenario ConfigRC.
System.Runtime.InteropServices.SEHException: External component has thrown an exception.
at Microsoft.Win32.SafeNativeMethods.CloseHandle(IntPtr handle)
at System.Runtime.InteropServices.SafeHandle.InternalDispose()
at System.Runtime.InteropServices.SafeHandle.Dispose(Boolean disposing)
at System.Diagnostics.Process.Close()
at System.Diagnostics.Process.Dispose(Boolean disposing)
at System.ComponentModel.Component.Dispose()
at Microsoft.SqlServer.Configuration.SqlEngine.SqlServerServiceBase.WaitSqlServerStart(Process processSql)
at Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineDBStartConfig.ConfigSQLServerSystemDatabases(EffectiveProperties properties, Boolean isConfiguringTemplateDBs, Boolean useInstallInputs)
at Microsoft.SqlServer.Configuration.SqlEngine.SqlEngineDBStartConfig.DoCommonDBStartConfig(ConfigActionTiming timing)
at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.ExecuteAction(String actionId)
at Microsoft.SqlServer.Configuration.SqlConfigBase.SlpConfigAction.Execute(String actionId, TextWriter errorStream)
Exception: System.Runtime.InteropServices.SEHException.
Source: System.
Message: External component has thrown an exception.

It turns out that I accidentally downloaded the debug check build version of Windows 2008 R2 SP1, and well, you can’t install SQL with that version.  Needless to say, the error message makes this completely obvious.  Found the hint to look at the ISO I was using on MSDN social.

LAFHA Math

Just throwing this down as finding this math or an explanation of it has eluded us until we actually could do it based on paystubs.  LAFHA is an in/out transaction (deduction and then addition).

  • Monthly Taxable Income =  Monthly Gross – LAFHA
  • Figure Taxes based on Monthly Taxable Income
  • Monthly Net Pay = Monthly Taxable Income – Taxes + LAFHA
Published
Categorized as work

Net.TCP, IIS7, and Classic AppPools

With the application that I’m helping “make fast” one of the optimizations identified by an architect was to use net.pipe or net.tcp on the application tier.  This was because the services on that tier call into each other, and waste a lot of time doing all the encapsulation that comes with wsHttpBinding.

We had tried to first use net.pipe because it is incredibly fast and it is all local.  However, because of how they do security here, it didn’t work.  Next up, net.tcp.

Overall, it wasn’t that difficult to setup the services with dual bindings (wsHttpBinding for calls from other servers and net.tcp for calls originating from the same server).  Granted, there were a lot of changes, and since all the configs are manually done here, was very error-prone.  I will be glad not to do any detailed configurations for a while.

Anyways, we did our testing of the build that incorporated it through 3 different environments, and over the weekend it went into Production.  Of course, that is when the issues started.

Bright and early Monday morning, users were presented with a nice 500 error after they logged into the application.  On the App tier servers we were getting the following error in the Application event log with every request to the services:

Log Name:      Application
Source:        ASP.NET 4.0.30319.0
Event ID:      1088
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Description:
Failed to execute request because the App-Domain could not be created. Error: 0x8000ffff Catastrophic failure

Well, since it mentioned the App Domain, we ran an IISReset and all was well.  I didn’t think too much into it at the time and only did some cursory searching as it was the first time we saw it.  However, today it happened again.

Our app is consistently used between 7AM and about 7PM, but during the night it isn’t used at all.  This is when the appPool is scheduled to be recycled (3AM).  It had appeared as if the recycle was what was killing us, as only the App Domain is recycled and not the complete worker process.

Immediately we had the guys here remove the nightly recycle and change it to a full IISReset.  At least that way we had a workaround until we could determine the actual root cause, come up with a fix, and test said fix.  However, it didn’t take long to determine the actual root cause…

One of the interesting things about this issue, was after it started happening in Production, a few of the other environments started exhibiting similar symptoms: Prod-Like and a smaller, sandbox environment.  Mind you, neither of these environments had these issues during testing.

So, I took some time to dive in and actually figure out the problem.  At first I thought it was because there were some metabase misconfigurations in these environments.  I wouldn’t say that all of these environments were pristine, nor consistent between each other.  I found a few things, but nothing really stood out…until…

While I was doing diffs against the various applicationHost.config files, NotePad++ told me it had been edited and needed to be refreshed (them removing the appPool recycles).  However, as soon as this happened the 500 errors started.  It didn’t help they did both machines at the same time which took down the whole application, but that’s another story.

This led me to believe that it wasn’t something within the configuration, but it also showed me how to reproduce it, at least part way.  The part that I was missing was that I had to first hit the website to invoke the services and then change the configs causing an app domain recycle.

Then I attempted to connect to the worker process with windbg to see what it was doing.  However, that was a complete failure as nothing actually happened to the process.  No exceptions being thrown, no threads stuck, etc.  It appeared to just sit there.

A bit of searching later led me to an article that had the exact same issues we were having, and that changing from a Classic to Integrated appPool fixed it.  However, it didn’t mention why.  Of course I tried it and it worked.  To appease the customer’s inquiries I knew I needed to find out why though.

I still don’t have a great solution, but apparently net.tcp and WAS activation has to be done in Integrated mode.  If it isn’t, you get the 500 error.  But ours works fine until the app domain is recycled.  Well, according to SantoshOnline, “if you are using netTcpBinding along with wsHttpBinding on IIS7 with application pool running in Classic Mode, you may notice the ‘Server Application Unavailable’ errors. This happens if the first request to the application pool is served for a request coming over netTcpBinding. However if the first request for the application pool comes for an http resource served by .net framework, you will not notice this issue.”

That would’ve been nice to know from Microsoft’s article on it, or at least a few more details.  I remember reading an article about the differences between Integrated and Classic, but I sure don’t remember anything specific to this.

Anyways, hope this helps someone who runs into the same issue…

Windows 2008 Performance Alerts

This may seem silly to some of you, but I am still getting used to Windows 2008.  Sadly, I don’t spend as much time actually administering servers as I used to (silly management), so it usually takes me a bit longer to make my way around 2008 than 2003.  I like to think they made everything more complex, but for some reason I’m sure I’ll get booed about that.

Anyways, this morning I was attempting to setup some performance alerts on some servers we’re having issues with.  Basically I wanted to have it email us when it reached a certain threshold.  No big deal, thinking I had this, I created the email app, created a performance counter, and then manually added it in.

Needless to say that didn’t work.  It took me awhile to figure out why too as my little email utility worked fine.  So I began a new search in order to find out how stupid I was being.

Turns out, quite a lot of stupid.  Instead of using the utility, you can now use scheduled task items…which includes an email action!  I basically used the instructions over at Sam Martin’s blog, which, I may add, he posted about in April of this year.  I’m not the only n00b.  Plus, who doesn’t have an enterprise system that deals with this sort of stuff already (at least at the types of clients I work with)?

Perfmon

  1. Open up perfmon
  2. Create a new User Defined Data Collector Set
  3. Choose to create manually after naming
  4. Select Performance Counter Alert
  5. Add in the performance counter you care about (mine was requests executing in asp.net apps 4.0)
  6. Choose the user to run it as
  7. Edit the Data Collector in the Data collector set
  8. Change the sample interval to whatever works for you (I set mine to 60s so we can be on top of issues prior to the users)
  9. Under Alert task, give it a name (e.g. EmailAlert) and give it a task argument (you can combine them to form a sentence like “the value at {date} was {value}”
  10. Start the Data Collector Set
Schedule Tasks
  1. Open up scheduled tasks
  2. Create a task, not a basic task
  3. Name it the exact same name you did in step 9 above (i.e. EmailAlert)
  4. Check “user is logged in or not” so that it runs all the time
  5. Create a new email action under the Action tab
  6. Enter all the info for from, to, subject, etc.  To send to multiple people, comma separate the addresses.
  7. For the body, type whatever you want, and then $(Arg0) will pass the task argument you made in step 9 above.
  8. Enter the SMTP server.

Done!

Since the performance counter was set to an application pool, whenever that pool disappears (IISReset, idle timeout, etc.) the counter stops.

Currently Reading (could take awhile): [amazon_image id=”B000QCS8TW” link=”true” target=”_blank” size=”medium” ]A Game of Thrones: A Song of Ice and Fire: Book One[/amazon_image]

Dummy Files

We are doing some document uploading to SharePoint, and needed some test files of various sizes.  If you have Visual Studio installed, you have the tools required to make these files.  Just make sure you run as administrator, and use the following command.

FSUTIL FILE CREATENEW 100MBTest.mdb 104857600

Usage: FSUTIL FILE CREATENEW [Filename] [Size in bytes]

Upgrade K2 Workflow Instances to a Specific Workflow Version

We were having a specific issue in our Dev and QA environments where K2 was consuming over 16GB of disk space, and subsequently causing our server to run out of disk space.  We had an interim workaround of restarting the K2 service, but within a day of testing, it was possible that K2 would eat it all up again.

This was happening because workflow instances (cases in our example) are tied to specific versions of the workflow.  Similar to .NET websites, in order for you to use a specific version it has to do a lot of pre-compiling.  Now, I’m not sure why it was using so much disk space per version, but that is essentially what was causing all our issues.

There are a few things that could’ve made this better:

  1. Testers and Developers not using old cases which are tied to older versions
  2. Building our K2 workflows in Release instead of Debug

Turns out option #2 reduces the space an individual version uses by orders of magnitude.  Sadly, there is no way to retrofit the processes that are actually already in K2.

The actual solution is to use some of the new APIs, specifically the Live Instance Management APIs (oh and that took awhile to find via searching).  The downside is that these APIs were added in 4.5, so anyone on a version prior to that is screwed.  Thankfully we were on 4.5.1!

Anyways, if you’re lazy, there is actually an already created utility on K2 Underground, and you can find out some additional info about it too.

Just be prepared for it not to work all that great.  We received a ton of Null Reference errors while running the utility against our large database.  It seemed to work fine in our POC, but not against the real thing.  Some cases were changed, but not all, and we still had the same issue.

In the end, we had to manually go and delete the old cases in K2, which is definitely not supported.  However, our app handles it gracefully, so it wasn’t a huge deal.

Clean-Up Winxs Folder

The folder gets large, but that is because of all the patches that are applied to your installation.  Do not simply delete the files, because it could cause all sorts of havoc.  The only way to actually reduce the size somewhat, is to have Windows remove any previous updates after you install a service pack.  You can do this with the following command.  I was able to clean up 4GB using the command.  Nothing major, but amost 25% improvement.

dism /online /cleanup-image /spsuperseded

Run as administrator.

 

Self Signed SSL Certs

I’ve always hated creating self-signed SSL certs.  It never seemed like there was a good and easy way to accomplish this.  Yes, you could download the II6 Resource Kit, but that’s just one more thing I don’t need on my machine.

Well, in IIS7, there is actually an option to automatically create one.  Technet has a good walkthrough of it.  However, there are some limitations:

  • The common name is always the machine name of your IIS server.
  • The certificate is only valid for one week.
  • The certificate is not added to the “Trusted Root Certificate Authorities” of any browser.

Thankfully a remake of SelfSSL was created for IIS7 and it is more powerful, and easier to use.   Take a look at Thomas Deml’s post on it to see the syntax and download the program.  I’ve also uploaded it here, just in case anything happens to that site.

SelfSSL7

Published
Categorized as work

More Visual Studio 2010 Performance Testing "Fun"

This is a continuation of my previous post on the half-baked core features of load testing in Visual Studio 2010.  We had been progressing fairly well, but with some of the new fixes that have gone into the application, we have reached new issues.

I would like to preface this with our application is by no means great.  In fact, it is pretty janky and does a lot of incredibly stupid things.  Having a 1.5MB viewstate (or larger) is an issue, and I get that.  However, the way that VS handles it is plain unacceptable.

With that said, I’m sure you can imagine where this is going.  When running an individual webtest each request cannot be larger than 1.5MB.  This took a bit of time to figure out as many of our tests were simply failing.  The best part of this is that we have a VIEWSTATE extract parameter (see #1 on the previous post), and the error we always get is that the VIEWSTATE cannot be extracted.  Strange, I see it in the response window when I run the webtest.  Oh, wait, does that say Response (Truncated)?  Oh right, because my response is over 1.5MB.

Oh, and that’s not just truncated for viewing, that’s truncated in memory.  Needless to say this has caused a large amount of issues for us.  Thankfully, VS 2010 is nice and you can create plugins to get around this (see below).  The downside is that VS has obviously not been built to run webtests with our complexity, and definitely not bypassing the 1.5MB sized response.

public override void PreWebTest(object sender, PreWebTestEventArgs e)
        {
            e.WebTest.ResponseBodyCaptureLimit = 15000000;
            e.WebTest.StopOnError = true;
            base.PreWebTest(sender, e);
        }

If you use this plugin, be prepared for a lot of painful hours in VS.  I am running this on a laptop with 4GB of RAM, and prior to the webtest running devenv.exe is using ~300MB of RAM.  However, during the test, that balloons to 2.5GB and pegs one of my cores at 100% utilization as it attempts to parse all the data.     Fun!

The max amount of data we could have in the test context is 30MB.  Granted, as mentioned earlier, this is a lot of text.  However, I fail to see how it accounts for almost 100x that amount in RAM.

Thankfully, in a load test scenario all that data isn’t parsed out to be viewed and you don’t have any of these issues.  You just need to create perfect scripts that you don’t ever need to update.  Good luck with that!

Oh and as an update, for #3 in my previous post, I created a bug for it, but haven’t heard anything back.  Needless to say, we are still having the issue.

And I realize that we’ve had a lot of issues with VS 2010, and I get angry about it.  However, I want to re-iterate that no testing platforms are good.

 

A Better Backup Plan

For the longest time, I’ve been using Carbonite as my backup provider.  I can’t say that I was ever really unhappy with it, but I was hoping to find something better as my yearly subscription was expiring.  Some of the issues I was trying to move away from:

  • As was previously mentioned, I have a WHS machine and getting Carbonite to work correctly with it wasn’t as easy as I had hoped.  Since Carbonite wouldn’t follow the tombstone files, I was forced to have my system as a single drive configuration.  Not all bad, and since it was virtualized anyways, it didn’t matter.
  • The UI is very slow.
  • While you can backup an “unlimited” amount of data, somewhere around 50-100GB they start to throttle you significantly.  When I recently added our wedding photos (11GB), it was going to take over 4 days to just add the incremental amount.
  • Very vanilla without many options.

I hadn’t really been shopping around, but when Mozy announced their pricing changes, it sort of peaked my interest to look around again.  I had heard about the Mozy price changes over at TechCrunch, and was reading in their comments about where the various people were flocking to now.  That’s where I found out about CrashPlan, or at least I thought I knew what it was all about.

Well, my Carbonite plan has about 2 months left on it now, and since I have a fair amount of data to backup (just over 200GB), I figured now was the time to make the move.  I mean, since I had based my upload figures on Carbonite’s speeds, that should just about cover the amount of time.

And man, am I glad I moved!  There are so many awesome features in CrashPlan that not only am I going to be using it, I’m going get others in the family to use it too.

First, just like Carbonite, I can pay to have my data up in the cloud.  There are a lot of similar items to Carbonite, but there are some nice advanced options:

  • Easily select which folders you want to upload – Same for both
  • Runs as a service, so you don’t have to be logged in – Same for both
  • Personal Encryption Key – Same for both
  • Follows junctions and tombstone files – Only CrashPlan
  • Can rent a 1TB drive to seed the initial upload (didn’t use, but nice option) – Only CrashPlan
  • Backup sets to have different backup intervals – Only CrashPlan
  • Backs up all file types (unless excluded through filter) – Only CrashPlan
  • No throttling, but can specify client side throttling based on multiple factors – Only CrashPlan

Mind you, those are just for the basic items that Carbonite offers (did I mention CrashPlan is cheaper too?).  However, CrashPlan also has a ton of other features in case you don’t want to upload to their cloud.  The best part?  If you don’t use the cloud services, you don’t have to pay for it.

This is a great feature for those that have a lot of storage in an always-on system and want to make a private cloud solution for family members.  It is actually something I’ve been trying to find so that my parents have a trusted cloud-based backup solution on my hardware.

The even better part?  Based on my testing with my work laptop, it just works!  I have a fairly complex networking structure at home, and while in the office, my laptop was able to connect and start backing up with no issues.  The only difference between what I did, and what my parents will need to do is create an account and “link” it to mine with a backup code that is unique to me.  From there, it starts to sync and they are off to the races.  I can specify a quota for them server side too, so it doesn’t go crazy.

Overall, I wish I had migrated earlier.  I definitely don’t feel bad in moving away from Carbonite, now that I’ve actually played with the software.  It solves all my initial issues, plus solves an ongoing problem I’ve been trying to fix.  Definitely a huge plus!  In fact, based on my experience, I would definitely consider their business service for an initial startup.  Just sayin’.