Setup Mac OS X VPN Server for Mac & XP Clients

Setup Mac OS X VPN Server for Mac & XP Clients from Technology PostsDecember 29, 2005
Mac OS X Server has included VPN support for some time. And, in true Apple fashion, it brings simplicity to a very complicated and technical server function. Virtual Private Networks, or VPNs, are used to securely connect two networks over the internet. This is done by creating an encrypted tunnel between the two networks. The tunnel wraps around all data that is passed in either direction. This keeps the information safe from prying eyes as it crosses the insecure internet. The tunnel endpoints take care of all the encryption and decryption so that, once the tunnel is established, the network communication is seamless to users.
In many cases, VPNs connect two routers and effectively bridges two networks. In the case of a telecommuter, the home router might establish a tunnel with a corporate router in order to allow the home user access to services on the company network. In this scenario, the two routers are the endpoints for the VPN. Router to router based VPNs are often very difficult to configure, especially when one of the endpoints is a high powered enterprise class device like those provided by companies like Cisco. Router to router VPNs are often hardware based because the routers on either end have hardware built into them that is dedicated to processing VPN traffic.
Mac OS X Server has the ability to create software based VPN tunnels. Combine that with the VPN client software built into the client version of Mac OS X and you have a very powerful and easy to configure VPN solution.
Consider this scenario. A corporation runs Mac OS X server on their network. A number of mobile users need to connect to the corporate network in order to access internal systems. Once the Mac server is properly configured, the remote users can establish a secure VPN tunnel between their desktop machine and the corporate network using nothing more that software already built into their operating system. And, once the VPN tunnel is established, all of the information exchanged between the remote user and the office network is fully encrypted and secure.
In this article, we will set up Mac X 10.4 Server to function as a VPN server. We will also look at the client configurations needed to connect to that server from Mac OS X 10.4 client (the non-server version of the OS) as well as from Windows XP.
Server:The server must be running Mac OS X Server (10.4.3 as of this writing). The VPN server capabilities are not built-in to the client version of the OS.
Open Server Admin, located here on your drive: /Applications/Server. Connect to the server using its IP address and the proper username and password. Once you do, you will see a list of services available on that machine. Click on VPN and the VPN settings will appear on the right.
Select the L2TP tab and use the image above as an example. Note that the IP addresses used in the image are for example only.
When a remote user connects to the internet, they receive an IP address from their service provider. When the VPN tunnel is negotiated with the VPN server, the server assigns the client an IP address from the corporate network. When the client accepts that address as part of the VPN negotiation, it adds it to the network interface in addition to the IP address from the internet service provider (ISP). This means that the VPN client actually has 2 addresses bound to it. One from the ISP, and one from the corporate network.
The VPN server needs to dynamically assign clients IP addresses from a pool of possible addresses. That is what we are specifying in this screen. You must specify both the starting and ending addresses of the IP pool that the VPN server is allowed to hand out to connect clients. Note that when a client disconnects from the VPN, his IP address is freed up and put back in the pool to be used by future clients. It is also essential to be sure that the addresses that are used in this pool are not used by any other computers on the corporate network. If they are, conflicts will occur and neither user will be able to access the network.
Set PPP Authentication to MS-CHAPv2 and specify a Shared Secret. This Shared Secret should be the strongest possible password you can come up with. Make sure it is not a dictionary word. And, the more digits in the Shared Secret, the better. The 3 weakest parts of the VPN are the username and password the user uses to connect, and the Shared Secret. If you use weak passwords or secrets, a tunnel could be established by anyone who might be able to guess that information.
Next, select the PPTP tab. Just as before, you must specify a pool of addresses that can be used by VPN users who connect using PPTP.
Under Mac OS X Server, Mac clients generally connect to the VPN server using L2TP. Windows XP users connect using PPTP. L2TP is considered more cryptographically sound, but since Microsoft did not conform to IPSec based standards when they wrote XP’s VPN client, Windows users are forced to use PPTP.
Finally, select the Client Information tab.
Here we specify the DNS servers the client should use once they have connected to the VPN. Since many corporations use internal DNS servers, the servers specified here will be used on any traffic that is traveling through the VPN.
Under Network Routing Definition we set the rules for the VPN routing. In my example, the corporate network is a Class C or addresses ranging from – In this example, the Network Address is entered as, but it might more appropriately be entered as since the Network Mask of details the assignment of the entire Class C. The final key value here is the Network Type. It is set to Private. This means that any traffic to or from the client that is destined for the 66.62.25.x network is considered internal and should remain on the secure VPN. Any addresses not listed as private here are not secure and the VPN client will route that traffic over the normal internet connection rather than sending it down the VPN tunnel to the corporate network. This is why the VPN client maintains a connection to the ISP assigned IP address in addition to the address that is assigned to it by the VPN server.
Lastly, a user account must be created on the server. This is done through the Workgroup Manager, and application located in the same directory as the Server Admin. When you create the account, be sure to set a strong password for the account. The username and password created here will be the credentials that the remote user will use when they log into the VPN.

Mac OS X VPN Client Configuration:The Mac VPN client is much easier to configure than the Window XP based equivalent.
Select New VPN Connection from the file menu, then choose L2TP over IPSec and continue.
A new profile will open. Don’t fill in the information in this screen. If you do, you will miss one vital piece of information. There is no place to specify the Shared Secret for the connection. Without it, the tunnel will never establish. Select Edit Configurations from the Configuration menu.
Fill in the fields with the appropriate information. The description can be anything you want it to be. The Server Address is the IP address of the Mac VPN server. The Account Name and Password is the login that you created for the user in the Workgroup Manager. Be sure to enter the same Shared Secret that you used when setup L2TP on the VPN server.
VPN On Demand is a new feature in 10.4. When you enable this feature, you are required to list domains that will trigger activation of the VPN tunnel when you try to access them.
When you click OK, your client is all set.
It is worth looking at some of the advanced options available under the Connect menu and then Options. There is an option to send all traffic over the VPN. This can be a powerful option. Normally you would not want to do this as it will increase traffic on the corporate end of the network. But, if you are a user on the road and using a hotspot or public wireless network, it might be a good idea to enable this option. In doing that, all of the traffic becomes protected from other users who might be sniffing traffic on the wireless network.

Windows XP VPN Client Configuration:Windows XP also has a built-in VPN client, but it has some disadvantages. First and foremost, it does not fully comply with standards based VPN servers. Once again, Microsoft has decided that it knows better and went in it’s own direction. On the upside, if you enabled PPTP on your Mac VPN server, XP users can still access the network.
First of all, right click on My Network Places a choose Properties. You will see a list of your network adapters. Click Create a New Connection on the left.
Select Connect to the Network At My Workplace. Its an odd name for it, but this allows you to create a VPN.
Select Virtual Private Network Connection and click Next.
Give your VPN connection a logical name. Anything that works for you is fine here.
Here you specify the IP address of the Mac VPN server.
Click finish here. You’re not really done yet. We need to make some changes to the VPN adapters configuration before you can connect to the Mac server.
Now go back to the Network Connections window. A new adapter should have been added to the screen. It will have the name that you gave the VPN connection when you ran the wizard.
Right click on the VPN adapter and select Properties.
Under the General tab, you should see the IP address of the Mac VPN server.
Under Security, select Advanced and then click Settings.
Select the Allow These Protocols radio button and then uncheck all of the boxes except for Microsoft CHAP Version 2.
Now select the Networking tab and set the Type of VPN menu to PPTP VPN. Click OK and you are done configuring the client. In order to connect the VPN, double click on the VPN adapter in My Network Places. You will be prompted for your login information. Once you click connect, your computer should negotiate the connection with the Mac sever.
Firewalling:Most corporate VPN servers are behind a firewall. In order for people outside of the firewall to gain access to the VPN server, certain Access Controls need to be added to the firewall. In my example, the Mac VPN server is behind a Cisco 2600 series router with its firewall enabled. This ACL shows the ports that were opened to allow both L2TP and PPTP access to the Mac server:
remark SOFTWARE VPN ACCESS RULES: permit udp any eq isakmp permit udp any eq non500-isakmp permit esp any permit gre any host permit tcp any host eq 1723
Update: 6/5/06 3:20pmA couple of people have asked for a more user friendly version of the above ACL (Access Control List). The example is directly from a Cisco router. Here's a more conventional explanation of the firewall rules:
permit udp any eq isakmp
allows UDP traffic from anywhere to any address on the subnet if the UDP port is isakmp (port 500)
permit udp any eq non500-isakmp
allows UDP traffic from anywhere to any address on the subnet if the UDP port is non500-isakmp (i'm not sure what port number this would be)
permit esp any
allows traffic from anywhere to any address on the subnet if the protocol is ESP (protocol #50)
permit gre any host
allows traffic from anywhere to the specific address of if the protocol is GRE (protocol #47)
permit tcp any host eq 1723
allows traffic from anywhere to the specific address of if the port is 1723 (PPTP)
In some cases, you may need to also enable 1701 for both TCP and UDP. Some users have reported their configurations would not work until these rules were added.
The specifics of these rules are beyond the scope of this article.
In my testing of the Mac VPN server, I had some other issues. I attempted to set up the VPN server on my home network so that I could access my files from remote locations. Given the limitations of my Linksys broadband router, I was unable to make the configuration work. I could not establish any rules on the Linksys to allow GRE or ESP traffic as consumer based routers only allow port mapping of TCP and UDP (layer 4 protocols).
Additionally, there may be an issue setting up a Mac VPN server on a corporate network if the address on the VPN server is a NAT’d virtual address. I was able to get a Mac remote client connected through the NAT some time ago, but never had luck connecting with a PC. If you setup a server in this configuration, I am interested in hearing about your experiences. Please leave your comments in the field below.
Closing:Apple has really made VPN easy with the latest release of its OS’s. With only a little knowledge of the subject, it is easy to get a remote secure connection up and running. In addition to the VPN service, Mac OS X Server includes a powerful Apache based web server, a DHCP server, Mail server, DNS server, Jabber chat server, Print server, and fileserver support for both Mac and Windows clients. All in all, a wide range of services and support for a single operating system. I strongly suggest trying it out for yourself!
Read more about Mac OS X Server.
Update: 12/30/05 10:30amHere's an Apple Tech Note that lists 'TCP and UDP Ports Used By Apple Software Products.' The list includes UDP port 1701 (L2TP) and UDP port 4500 (IKE NAT Traversal). I did not need those in my firewall rules, but several readers have emailed explaining that their VPNs work until they activate 10.4's firewall service on the server. Allowing these ports may resolve the issue. Please post your feedback below.
Update: 1/3/06 7:32amVPN Servers, and DNS:When setting up the VPN Server, you have the ability to specify the DNS servers that clients will use when they connect to the VPN. There is one important thing to keep in mind when you specify these addresses. Many DNS servers do not allow recursion. This means that they only allow lookups to be made by clients in select subnets. For example, Comcast DNS servers do not allow DNS lookups to be made by people connecting via AOL.
This is done for a number of reasons, but mainly for security. But it is important to consider this when you specify the DNS servers in the VPN settings. If your VPN server is sitting on your corporate network, be sure to specify the DNS servers that other clients on the corporate network would use. Similarly, if your VPN server is on your home network, specify the DNS servers you would use when you are surfing the web at home (Example: Comcast DNS servers if Comcast provides your internet connection at home). Remember that when clients connect to the VPN server, they receive an IP address from the pool of addresses you specified when you set up the server. Effectively, this makes a VPN user a client of that remote network, and their DNS requests will be made accordingly.If you connect to the VPN server but find that you cannot connect to any other services once you are there, you can easily determine the problem. If you are entering the name of the remote service but cannot connect, open up the Terminal and try to ping that address via its name. Also try to ping the address via its IP. If you can ping it via the IP and not via the name, odds are the DNS servers you specified are your problem.When you do this, you should note the setting of the "Send All Traffic Over VPN Connection" checkbox located in the Internet Connect application under the Connect menu, then Options. If the box is not checked, the pings you send must be located on the VPN servers network. If the box is checked, you should be able to ping any address that would normally be ping able.Network to Network VPN Connections:This should not be a factor when making a client to server VPN connection as we do with the Mac's VPN server, but this point is worth making. Should you work with router to router, or network to network VPN connections in the future, be mindful of the virtual IP addresses distributed on either side of the VPN. By default, most routers use 192.168.1.x as the internal addresses. In router to router VPN connections, it is essential to have unique subnets if virtual addresses are used on both sides of the VPN tunnel. For example, if your home router connects to your corporate router and your business uses an internal NAT subnet of 192.168.1.x, your home network must use a different set of internal NAT address. Try something like 192.168.2.x for your home network.The point should not apply to client to server VPN connections, but I have heard of some users trying to connect to OS X's VPN Server from routers rather than clients. I am not sure how well that works, but this rule will be something to keep in mind.
Update: 1/3/06 1:50pmOne of the cool new features in 10.4's VPN client is the ability to send all traffic over the VPN. As Joe noted in the comments below, this is great for people using public access, like a wireless network at the upcoming MacWorld show. This comment was right on the money, and I thought the idea warranted a little further detail.Setting the VPN client to send all traffic over the VPN has several advantages, and two possible disadvantages. First, the down side.Disadvantages:Consider the bandwidth available to your VPN server. If you are on a corporate network, odds are you have a synchronous internet connection, meaning that the internet connections upstream bandwidth is equal to its down stream. This is the case with the T1 at my office. If your VPN server is using a consumer level broadband provider, odds are your connection is asynchronous. This is often the case with DSL or cable modem connections. The downstream might be a high as 8Mb, while the upstream is limited to 384Kb. That is the case with my cable modem at home.The problem occurs when you route all of your traffic through an asynchronous connection. If the downstream is 8MB and the upstream is 384Kb and I am running my VPN Server from that network, the fastest my VPN client will be able to either send or receive data will be at 384Kb. This is because all traffic is essentially being funneled through the asynchronous network connection before it arrives at the VPN client. Even if your clients access point might offer higher speed access, this performance bottleneck will keep you from surfing at the speeds you might expect. Also, keep in mind that several VPN users in this situation can use up the available bandwidth much quicker than you expect.Advantages:As for the advantages, there are many worth considering. For example, say you are accessing a wireless hotspot from the MacWorld show floor. If you understand how wireless networks function, you realize that everyone on that same wireless node has the ability to sniff your data, unless it's encrypted. That means that your mail servers POP3 login information is sent in the clear for anyone to literally grab out of thin air. So are the contents of your email messages for that matter. The same goes for your FTP login, or any telnet access.When you route all traffic through the VPN tunnel, you effectively protect all of that data. Since the data is passing through the tunnel (both incoming and outgoing), it is unreadable to anyone between you and your VPN server. Once the traffic reaches the VPN Server, it is no longer encrypted and it flows out onto the internet as needed to reach its intended destination. By then, your data is clear of the danger zone. The VPN connection makes you data as safe as it would be if you were sitting right beside the VPN server.This concept is important to consider when you realize that once someone has access to your email login, they have full control over your email. And if you plan on blogging from the show floor, this may be the only way to stay truly secure. If consider any of your internet based traffic confidential, this really is the best way to go.
Update: 2/17/06 11:32amSeveral people have reported that once they login to the VPN, they can contact the VPN server but none of the other clients on the LAN. According to their feedback, enabling the NAT service with IP Forwarding resolved the issue. Apparently this not needed in all situations, but it does correct this issue.I did need the NAT service when I first used VPN on 10.3.x, but I thought the need has been eliminated in 10.4 Apparently the need has only been eliminated in some situations.Thanks to everyone for their comments and feedback!
Update: 3/7/06 11:15amI added a note to the above firewall rule set. Some users could not access their VPN's until they opened up TCP & UDP port 1701. This might be necessary in some configurations.
Update: 6/28/06 7:45am10.4's VPN service is one of the easiest VPN systems to get running. That being said, it can still be a very painful experience. But another software alternative has been released that could solve many users issues. It won't fit the bill for everyone, but please checkout the recent post I did about Hamachi and HamachiX. Hamachi is a powerful VPN alternative and it is very easy to configure and operate.Hamachi is not a replacement for the VPN services that OS X offers. The Mac OS's VPN capabilities are still a personal favorite. That being said, I also think it's important to look at alternatives as all VPN solutions are not created equal. Hamachi is simply a great example of a powerful alternate solution!
Post a Comment

You might also like :

Related Posts with Thumbnails