ISA Site-to-Site IPSec VPN

I wasn’t necessarily going to post this, but since ISA seems to be the most linked to thing on this site because of only 2 articles, I figure it can’t hurt to talk about it. Especially since it was a very strange problem I had with it and I’m sure I won’t be the only one with it.

Anyways, at work I am utilizing ISA 2006 Std edition in a front and back wall scenario. Site-to-Site VPN terminate on the external firewall, and all of our local VLANs (55 of them) are routed off of the internal firewall. So far, nothing that complex. It’s just a simple DMZ between the external and internal network setup.

Anyways, I had a site-to-site VPN (IP pre-shared key) between a customer and us. Basically, we just need to hit a single machine, so the remote network contained two IP addresses, one for the client’s gateway (this is added by default in ISA 2006, DO NOT delete it, also be sure that the remote site has added your gateway in as a remote network too!) and another for the machine we needed to hit on their local network. Anyways, it was working fine. Well, actually, nobody was using it quite yet, but testing had been completed, and I was able to access everything that the developers would need. Anyways, the customer decides that they need to add another IP address that we’ll need to access. Again, no big deal. I’ll just add the IP to the network list for this client. Just to make sure everything’s working, I test it. Nothing works to the new IP. However, the old IP still works fine. What the hell?!

For those of you unfamiliar with ISA, it’s not like I created a new VPN for this new IP addition, or anything like that. I simply added the new IP to the existing network. All the routing and firewall rules remained the same. Adding the new IP to the list of remote networks should have allowed it to work.

Working with the IT person at the customer, I learn that when I try to hit the new IP address, the Quick Mode authentication was failing because the ISA server was sending the wrong local network that the request was coming from. The local network that was defined in the rule (by putting a subnet destination in the network rule) was 10.254.95.192/27. However, on the client’s side, he was seeing the request coming from 10.254.64.0/19. In order to create the IPsec tunnel, both the local and remote networks on each end of the tunnel must be identical, but switched (i.e. my local is his remote, and his local is my remote). Needless to say, this 10.254.64.0/27 was screwing everything up. However, when I connected to the original IP that worked, it was sending the correct network of 10.254.95.192/27.

Of course, no where in ISA 2006’s logging can you see it making the IKE requests. All I could see is that requests were being routed correctly from the internal ISA to the external ISA, and then from the external out to the correct network for the customer VPN. In essence, traffic was going in to a black hole. I could also see that the VPN connection (Main Mode) was up and running. I was completely reliant on the customer to let me know what was coming down the pipe to him. That right there is not really something I’m comfortable with, but he seemed to be OK with it. I’m sure it’s because he knew it wasn’t on his end, but on mine.

After deleting the VPN multiple times and recreating it to no avail, restarting the machine, etc, I knew that I would have to get some help from someplace else. Thankfully we have an awesome community of people at work that I could bounce ideas off of. Unfortunately, I never received a response. Also, ISAServer.org is a great place to get information. They have forums there that people keep an eye on. Unfortunately, ISA 2006 is still quite new and not as many people deal with it. I also did not receive a response from there. Needless to say, I was on my own for this one. Not a place I really wanted to be, since I thought I was at the end of my ability.

Actually, the IP isn’t really done in ISA server at all. Much like everything else that ISA server does, it’s just an application that sits on top of the OS and utilizes things that are already built into the OS (in my case Windows 2003 R2). This means that all IP policies, rules, etc are done by Windows and this can be monitored using the IP Security Monitor MMC Snap-in.

Since the VPN tunnel was being created successfully, I knew that Main Mode IKE Policies were correct, it was the Quick Mode policies that were causing me grief. Since we have multiple VPN connections terminating on this firewall, there are a lot of Quick Mode IP policies in place. Especially since all of them use pre-shared keys, which require that two IP policies are created, one for inbound and outbound (otherwise you can have one policy that does both inbound and outbound).

Scanning through the policies I was able to find the inbound and outbound policies for the original customer IP address to the 10.254.95.192/27 network, but I wasn’t able to find it for the new customer IP address. Alas, the problem! The next best policy for the new IP address was for the 10.254.64.0/19, since this policy encompasses the 10.254.95.192/27 subnet. Finally, I felt like I was making progress. Unfortunately, ISA should have been creating these policies when I edit the customer VPN networks. Actually, I still have no idea why ISA isn’t creating these policies. This is why I think there’s a bug which I’m going to submit to Microsoft (via this post actually).

Now that I knew the source of the problem, I had to fix it. Some days diagnosing the problems take longer than fixing them, and some days it’s the other way around. Since it had already taken me about a day to find the problem, I hoped that it wouldn’t take that long to actually fix it.

Needless to say, you can’t add IP policies from the IP Security Monitor MMC Snap-in, because, well, it’s a Monitor not an editor. The IP Policy Manager MMC Snap-in was no use either, as it defines computer level policies. Doh. Well, I can finally say that one of my certifications actually came in handy. That “+ Security” portion of my MCSE gave me the knowledge that there is a way to edit IP policies from the command line. Going on this, a quick Google search gave me exactly what I was searching for. Now which command to actually use?

At first I tried to just create a filter. However, I didn’t know of any filterlist, and none of the current filters were a member of a filterlist. Thankfully you can just make up a name and it creates on. Unfortunately this didn’t solve anything. Nothing showed up in the Quick Mode filters. Lets try again, yeah?

Turns out it’s not a static setting, but a dynamic setting, which makes more sense. Anyways, you can add Quick Mode rules pretty much the same. In that I mean, the command is just as long and gross. Just be aware, that since I wanted to add a Quick Mode rule and not a Main Mode rule, I had to put in the Quick Mode Policy variable.

Another thing that made this so confusing was that in IP Monitor, they are called Quick Mode Filters and at the command line they’re called Rules. Ugh. At least it’s taken care of. And now I think I know more than I ever wanted to about ISA and IP.