This is a continuation of my previous post on the half-baked core features of load testing in Visual Studio 2010.  We had been progressing fairly well, but with some of the new fixes that have gone into the application, we have reached new issues.

I would like to preface this with our application is by no means great.  In fact, it is pretty janky and does a lot of incredibly stupid things.  Having a 1.5MB viewstate (or larger) is an issue, and I get that.  However, the way that VS handles it is plain unacceptable.

With that said, I’m sure you can imagine where this is going.  When running an individual webtest each request cannot be larger than 1.5MB.  This took a bit of time to figure out as many of our tests were simply failing.  The best part of this is that we have a VIEWSTATE extract parameter (see #1 on the previous post), and the error we always get is that the VIEWSTATE cannot be extracted.  Strange, I see it in the response window when I run the webtest.  Oh, wait, does that say Response (Truncated)?  Oh right, because my response is over 1.5MB.

Oh, and that’s not just truncated for viewing, that’s truncated in memory.  Needless to say this has caused a large amount of issues for us.  Thankfully, VS 2010 is nice and you can create plugins to get around this (see below).  The downside is that VS has obviously not been built to run webtests with our complexity, and definitely not bypassing the 1.5MB sized response.

public override void PreWebTest(object sender, PreWebTestEventArgs e)
        {
            e.WebTest.ResponseBodyCaptureLimit = 15000000;
            e.WebTest.StopOnError = true;
            base.PreWebTest(sender, e);
        }

If you use this plugin, be prepared for a lot of painful hours in VS.  I am running this on a laptop with 4GB of RAM, and prior to the webtest running devenv.exe is using ~300MB of RAM.  However, during the test, that balloons to 2.5GB and pegs one of my cores at 100% utilization as it attempts to parse all the data.     Fun!

The max amount of data we could have in the test context is 30MB.  Granted, as mentioned earlier, this is a lot of text.  However, I fail to see how it accounts for almost 100x that amount in RAM.

Thankfully, in a load test scenario all that data isn’t parsed out to be viewed and you don’t have any of these issues.  You just need to create perfect scripts that you don’t ever need to update.  Good luck with that!

Oh and as an update, for #3 in my previous post, I created a bug for it, but haven’t heard anything back.  Needless to say, we are still having the issue.

And I realize that we’ve had a lot of issues with VS 2010, and I get angry about it.  However, I want to re-iterate that no testing platforms are good.

 

As I am sure you can tell from the title of this post, we have been having nothing but issues using Visual Studio 2010 on the current solution we are performance testing.  While this is going to be a bitch session, with possible solutions and workarounds we have found, these issues are in no way limited to only Visual Studio.

As one of my coworkers said, “I’m learning that every load testing solution is shit.”  Sadly, the more you work with them, and the more complex the solution, the quicker you come to this realization.  This becomes even more clear as we are expected to bounce around between different performance testing solutions, being pseudo-masters on a smattering of them.  While it is expensive, outsourcing to someone like Keynote may possibly be the best answer (I have used and work with these guys before, and they are great).

Without further ado, let me breakdown the issues we’ve been having so far.  I really, really, really hope that this list doesn’t continue to grow as we are already running out of time.

1. Scripting.  We had a bear of a time with scripting our specific website.  No matter what we did, there was no way to get it to actually work by scripting it with Visual Studio.  The website in general is fairly basic, but is loaded up with a ton of Telerik controls per page (don’t even get me started!).  Each stage of our workflow has a bunch of these controls and then a final submit button that moves it into the next stage.  Scripting with VS always failed on the subsequent AJAX postbacks because it was not correctly parameterizing the values (wasn’t extracting some).  However, this only happened when we scripted all the way to through the final submit.  If we scripted, but did not include the final submit button, the script worked correctly and did not have any of the errors.  Since the Telerik controls have so many forms in the post fixing the parameterization issue by hand would’ve taken hours per page (each workflow has 7 pages and there are 17 workflows).  And we couldn’t figure out how to simply wire up the final submit to the working rest of the script.  No matter what we did we’d always get errors on the final submit POST.  The solution?  Use Fiddler and save the sessions as a webtest.  This has a lot of downsides, such as no parameterization at all and the scripts break pretty easily once any code changes.  Fun.
2. Load Testing Workflows with NTLM Authentication. The next issue we ran into was with wiring up the individual workflow pieces into one large workflow.  The breakdown was that each part of the workflow needed to be handled by a different user, and the users would log in via NTLM.  The most obvious way to do this was to have a webtest call another webtest.  However, we weren’t able to get that to work.  The next way was to use an ordered test, but that didn’t give us reports into individual page loads.  The final way was to create a load scenario that runs scripts in a specific order, but that would require a lot of controllers among other things.  In our desperation, I even created an MSDN question.  The solution?  We created a plugin that cleared the user’s cookies (even though it should’ve been running as a unique user 100% of the time), and also accessed a server redirect page to force the authentication request.  Thankfully we didn’t go down the road of rolling our own queuing system as that would’ve been painful.
3. Lack of Test Logs after a Run. Now that we were actually able to run tests, we were having all sorts of issues with results.  Sadly, we weren’t able to actually view the results because VS wasn’t saving them.  Again, in desperation I created another MSDN question.  With VS2010, you are supposed to be able to capture all the results of failed tests, and the select “Test Log” to see what the results are.  Unfortunately, when we run tests it sometimes shows up, and sometimes not.  However, for anything longer than a 15 minute run, ours were 60 mins, we never received any results.  We also get links to the “Test Log”, but they don’t do anything when you click.  The solution?  Yeah, as of now we don’t have one other than running two controllers: one running the full load and another one running individual tests to hopefully see a similar error message.

I can only hope that there are no other issues.  Hope, hope, hope!