Internet Protocol security (IPSec)

Internet Protocol security (IPSec) is a framework of open standards for ensuring private, secure communications over Internet Protocol (IP) networks, through the use of cryptographic security services.

  • Before you begin using IPSec with this computer, see Checklist: Defining IPSec policies.
  • For tips about using IPSec, see Best practices.
  • For help with specific tasks, see How to.
  • For general background information, see Concepts.
  • For problem-solving instructions, see Troubleshooting.

Note

  • Portions of IPSec-related services were jointly developed by Microsoft and Cisco Systems, Inc.


Checklist: Defining IPSec policies

Step | Reference
Establish a security plan. | Establishing an IPSec security plan
Review Internet Protocol security (IPSec) policy concepts. | Description of an IPSec policy
Start the IP Security Policies snap-in. | To start the IP Security Policies snap-in
Define an IPSec policy for each scenario in your enterprise security plan. | Define IPSec policies
Assign IPSec policies. | Assign IPSec policy


Best practices

The IP Security Policies snap-in greatly simplifies IPSec deployment. To take advantage of this and to avoid problematic implementations, you can:

  • Evaluate the type of information being sent over your network. Is it sensitive financial data, proprietary information, or electronic mail? Some departments might require a higher level of security than the majority of an enterprise because of their specific function. Consider the following:
    • What should you secure?

      • Should you secure traffic between some computers or all computers, or for only specific protocols or ports?

    • How should you secure it?

      • Should you secure the traffic with integrity only or confidentiality also, and at what strength?

    • Where should you secure it?

      • Should you secure it only over remote access connections, or also through the local area network?

    • Who will manage policy?

      • Should this be domain administrators, server administrators, or local computer administrators?

    • Will encryption settings work for all pertinent computers?

      • Will data be accessed by both computers using strong cryptography (3DES encryption) and computers using standard cryptography (DES)?

  • Determine where your information is stored, how it routes through the network, and from what computers it can be accessed. This provides information about the speed, capacity, and utilization of the network prior to IPSec implementation, which is helpful for performance optimization.
  • Evaluate your vulnerability to various types of network attacks. For more information, see Security issues with IP.
  • Design and document an enterprise-wide network security plan. Take into account the following:
    • The general security framework of Windows 2000, including the use of Active Directory and how security is applied to Group Policy objects. For more information, see Active Directory overview.
    • The communication scenarios that you require, including intranet, remote access, extranets for business partners, and communication between sites (router-to-router).
    • The level of security that is necessary for each scenario. For example, you might decide that only Internet communications require confidentiality.
  • Design, create, and test the IPSec policies for each scenario. This allows you to clarify and refine which policies and policy structures are necessary. During the testing of your deployment scenarios, run standard workloads on applications to gain realistic feedback. During initial tests, you can view packet contents with Microsoft Network Monitor or a network sniffer by using the Integrity only security method level or a custom security method set to AH, since using Encryption and Integrity or ESP with encryption will prevent packet viewing.

For more information about IPSec security planning, see Establishing an IPSec security plan. For information about special requirements for deploying IPSec, see Special IPSec considerations.


How to...

  • Start the IP Security Policies snap-in
  • Define IPSec policies
  • Assign IPSec policy
  • Manage IPSec policy
  • Monitor IPSec activity


Start the IP Security Policies snap-in

Microsoft Windows graphic

To start the IP Security Policies snap-in

  • To start the IP Security Policies snap-in from the Microsoft Management Console:
    1. Click Start, click Run, type MMC, and then click OK.
    2. In MMC, click File, click Add/Remove Snap-in, and then click Add.
    3. Click IP Security Policy Management, and then click Add.
    4. Select the computer for which you want to manage IPSec policies:
      5. To | Do this
Manage only the computer on which this console is running | Click This computer.
Manage IPSec policies for any domain members | Click The Active Directory Domain of which this computer is a member.
Manage IPSec policies for a domain of which the computer that is running this console is not a member | Click Another Active Directory Domain.
Manage a remote computer | Click Another computer.
    5. Click Finish, click Close, and then click OK.
  • To access the IP Security Policies snap-in from Group Policy (Active Directory):
    1. Open Active Directory Users and Computers.
    2. In the console tree, right-click the domain or organizational unit for which you want to set Group Policy.

      Where?

      • Active Directory Users and Computers [DomainControllerName.DomainName]
      • Domain
      • OrganizationalUnit
      • ChildOrganizationalUnit...
    3. Click Properties, and then click the Group Policy tab.
    4. Click Edit to open the Group Policy object that you want to edit. Or, click New to create a new Group Policy object, and then click Edit.
    5. In the Group Policy console tree, click IP Security Policies on Active Directory.

      Where?

      • PolicyName [ComputerName] Policy
      • Computer Configuration
      • Windows Settings
      • Security Settings
      • IP Security Policies on Active Directory
  • To access the IP Security Policies snap-in from Local Computer Policy:
    1. Click Start, click Run, type MMC, and then click OK.
    2. In MMC, click File, click Add/Remove Snap-in, and then click Add.
    3. Click Group Policy Object Editor, and then click Add.
    4. Click Finish, click Close, and then click OK.
    5. In the Group Policy console tree, click IP Security Policies on Local Machine.

      Where?

      • Local Computer Policy
      • Computer Configuration
      • Windows Settings
      • Security Settings
      • IP Security Policies on Local Machine

Notes

  • To start Active Directory Users and Computers, open a Remote Desktop Connection to either a Windows 2000 domain controller or a member server that has Windows 2000 Administration Tools installed. You must log on to the server as a domain administrator in order to complete this procedure.
  • To save console settings, on the File menu, click Save.
  • You cannot administer Active Directory-based IPSec policy from a computer running Windows XP Home Edition.
  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To manage policies in a remote domain, you must be using a computer that is a member of a domain that is trusted by the remote domain. You cannot configure policies in a remote domain from a computer that is a member of a workgroup (also known as a stand-alone computer).
  • To view the saved console, see Related Topics.
  • To revise your initial choice, you can start the Microsoft Management Console, add this snap-in again, and save the console again. If you require multiple configurations, you can save the console with another name.

Related Topics


Define IPSec policies

  • Add or edit IPSec policies
  • Define rules
  • Define advanced IPSec settings


Add or edit IPSec policies

Microsoft Windows graphic

To add or edit IPSec policies

  1. In IP Security Policies, choose whether to define a new policy or edit a current one:
    2. To | Do this
Create a new policy | In the console tree, click IP Security Policies on Name. Next, on the Action menu, click Create IP Security Policy. Complete the instructions in the IP Security Policy Wizard until the Properties dialog box for your new policy appears.
Edit an existing policy | Double-click the policy.
  2. Click the General tab, and then in Name, type a unique name.
  3. In Description, type a description of the security policy, such as which groups or domains it affects.
  4. For domain-based policy, to specify how often the IPSec Policy Agent checks Active Directory for updates, type a value in Check for policy changes every number minute(s).
  5. If you have special requirements for the security of the key exchange, click Advanced.
  6. Click the Rules tab and create or modify any necessary rules for the policy.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • To configure advanced settings, see Related Topics.
  • To configure Rules, see Related Topics.

Related Topics


Define rules

  • Add or edit rules
  • Activate rules
  • Define IPSec authentication methods
  • Specify IPSec connection types
  • Specify an IPSec tunnel
  • Define IP filter lists
  • Define filter actions


Add or edit rules

Microsoft Windows graphic

To add or edit rules

  1. In IP Security Policies, double-click the policy you want to modify.
  2. Decide whether or not you want to use the Security Rule Wizard:
    • To use the Security Rule Wizard to add a rule, confirm that the Use Add Wizard check box is selected, click Add, and then follow the instructions.
    • To add a rule manually, confirm that the Use Add Wizard check box is cleared, and then click Add.
  3. To edit a rule, double-click the rule.
  4. Define properties on the IP Filter List, Filter Action, Connection Type, Authentication Methods, and Tunnel Setting tabs.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • New rules are automatically applied to the policy being edited or created.
  • The rules for a policy are displayed in IP Security Policies in reverse alphabetical order, based on the name of the filter list selected for each rule. There is no method to specify an order in which to apply the rules in a policy. The IPSec driver automatically orders the rules based on the most specific filter list to the least specific filter list. For example, the IPSec driver places a rule containing a filter list that specifies individual IP addresses and TCP ports above a rule containing a filter list that specifies all addresses on a subnet.
  • The default response rule is automatically added (and activated if you select this) in every new IPSec policy. If you do not want this rule to be part of your policy, you can deactivate it by clearing the check box next to <Dynamic>. The default response rule cannot be removed.

Related Topics


Activate rules

Microsoft Windows graphic

To activate rules

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Select the check box next to any rules that you want to activate, or clear the check box to deactivate a rule.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Define IPSec authentication methods

Microsoft Windows graphic

To define IPSec authentication methods

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that you want to modify.
  3. On the Authentication Methods tab, click Add. Or, if you are reconfiguring an existing method, click the authentication method, and then click Edit.
  4. Authentication methods:
    • Click Active Directory default (Kerberos V5 protocol) to use the Kerberos V5 security protocol for authentication services, if this rule applies to computers that are validated by a Windows 2000 Active Directory domain or a trusted Active Directory domain.
    • To use a public key certificate for authentication services, click Use a certificate from this certification authority (CA), and then click Browse to select a root CA.
    • To specify your own key that will be used for authentication, click Use this string to protect the key exchange (preshared key).

Important

  • The preshared key value is stored in a readable format within the IPSec policy. By default, IPSec policy that is stored in Active Directory can be read by any authenticated user, but changed only by directory administrators. Because of this, specifying an authentication key value in directory-based policy is not a recommended security practice. Use authentication keys only when IPSec policy is stored and managed locally on the computer to which it applies. The IP Security Policies snap-in must be run on a domain controller to securely manage IPSec policy that is stored in Active Directory.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • The Active Directory default (Kerberos V5 protocol) authentication method is not supported on computers running Windows XP Home Edition.
  • If you choose to use a certificate for authentication, you must select a CA (most commonly the root CA for your installed computer certificate). You cannot leave this field blank.
  • For preshared key authentication, each IPSec peer must use the same preshared key or authentication will fail.
  • To delete the selected method, click Remove.
  • To move the selected method up one level, click Move up. Repeat until the method is at the required preference level.
  • To move the selected method down one level, click Move down. Repeat until the method is at the required preference level.
  • To open IP Security Policies, see Related Topics.
  • If filters in more than one rule apply to traffic between the same two IP addresses, then the authentication methods list should be identical in both rules. Otherwise, security requests that are initiated by one rule might match the filter of the other rule, but have a different authentication method. In this case, the negotiation will fail. After a rule has initiated a successful main mode security association between two IP addresses, other rules that are filtering different traffic will not renegotiate the authentication and master key (the main mode security association). Instead, these other rules will use the same main mode security association to negotiate security (algorithms and session keys) for traffic that matches their particular filter.

Related Topics


Specify IPSec connection types

Microsoft Windows graphic

To specify IPSec connection types

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Click the rule you want to modify, and then click Edit.
  3. On the Connection Type tab, select the type of network connections to which this rule will apply:
    • To apply this rule to all network connections you have created on the computer, click All network connections.
    • To apply this rule to all LAN connections you have created on the computer, click Local area network (LAN).
    • To apply this rule to any remote or dial-up connections you have created on the computer, click Remote access.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Specify an IPSec tunnel

Microsoft Windows graphic

To specify an IPSec tunnel

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that you want to modify.
  3. On the Tunnel Setting tab, specify the computer that will be the tunnel endpoint:
    4. To | Do this
Disable tunneling for this rule | Click This rule does not specify an IPSec tunnel.
Use tunneled communications to a specific tunnel endpoint | Click The tunnel endpoint is specified by this IP address and type the IP address of the tunnel endpoint.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • Because you cannot mirror filters for tunneled traffic, you must configure two rules. One rule is used for the outbound traffic and the other is used for inbound traffic. For the outbound traffic rule, the tunnel endpoint is the IP address of the IPSec peer on the other end of the tunnel. For the inbound traffic rule, the tunnel endpoint is an IP address configured on the local computer.
  • IPSec tunnels are used primarily for interoperability with other routers, gateways, or end-systems that do not support L2TP/IPSec or PPTP connections. IPSec tunnel mode is supported as an advanced feature, used only in gateway-to-gateway (also known as router-to-router) tunneling scenarios and for server-to-server or server-to-gateway configurations. IPSec tunnels are not supported for remote access VPN scenarios. L2TP/IPSec or PPTP should be used for remote access VPN connections.

Related Topics


Define IP filter lists

  • Add or edit IP filter lists
  • Add or edit IPSec filters
  • Configure advanced IPSec filter settings
  • Select an IP filter list for a rule
  • Manage multiple IP filter lists


Add or edit IP filter lists

Microsoft Windows graphic

To add or edit IP filter lists

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. In the IP Security Policy Properties dialog box, click the rule to which you want to add the new IP filter list, and then click Edit.
  3. If you are adding a new IP filter list, on the IP Filter List tab, click Add. Or, if you are reconfiguring an existing filter list, click the filter list, and then click Edit.
  4. Type a unique name for the list.
  5. Type a description, for example, of which traffic types this list represents.
  6. Create the filters.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • For procedures on creating filters and advanced filter settings, see Related Topics.

Related Topics


Add or edit IPSec filters

Microsoft Windows graphic

To add or edit IPSec filters

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that contains the IP filter list you want to modify.
  3. Do one of the following:
    • If you are adding an IPSec filter list, on the IP Filter List tab, click Add.
    • If you are reconfiguring an existing IP filter list, double-click the IP filter list.
  4. In IP Filter List, do one of the following:
    To | Do this
    Use the IP Filter Wizard to create a filter | Confirm that the Use Add Wizard check box is selected, and then click Add.
    Create a filter manually | Clear the Use Add Wizard check box, and then click Add.
    Reconfigure an existing filter | Double-click the filter.
  5. On the Addressing tab, select the Source Address:
    Select | To secure packets from
    My IP Address | All IP addresses on the computer for which you are configuring this filter.
    Any IP Address | Any computer.
    A specific DNS Name | The Domain Name System (DNS) name that you specify in Host name. The DNS name is resolved to its IP addresses, and then filters are automatically created for the resolved IP addresses. This option is only available when creating new filters.
    A Specific IP Address | The IP address that you specify in IP Address.
    A Specific IP Subnet | The IP address that you specify in IP Address and the subnet mask that you specify in Subnet Mask.
  6. Click Destination Address and repeat step 5 for the destination address.
  7. Under Mirrored, select the appropriate setting:
    To | Do this
    Automatically create two filters based on the filter settings, one for traffic to the destination and one for traffic from the destination | Select the Mirrored check box.
    Create a single filter based on the filter settings | Clear the Mirrored check box.
    Create a filter for an IPSec tunnel | Clear the Mirrored check box. For IPSec tunnels, you must create two filter lists: one list describes the traffic to be sent through the tunnel (outbound traffic) and another describes the traffic to be received through the tunnel (inbound). Then, create two rules that use the inbound and outbound filter lists in your policy.
  8. On the Description tab, in Description, type a description for this filter (for example, specify to what computers and traffic types it applies).
  9. If you require additional IP filtering by a specific protocol or port number, on the Protocol tab, configure advanced filter settings.

Caution

  • Filters are the most important part of IPSec policy for a computer which is protected by IPSec. If you do not specify the proper filters in either client or server policies, or if the IP addresses change before the policy's filters are updated, security might not be provided.
    • Never use DHCP-assigned IP addresses in an IPSec filter. Use My IP Address as a source or destination in the policy that is used by a computer that uses a DHCP address. Changes in IP addresses will then automatically update a My IP Address filter.
    • In the policy of a computer that will be requesting IPSec protection for its traffic, make sure the filter destination address is a static IP address for the destination computer. All IP addresses for the destination must be included in the filter list.
    • When adding a new static IP address to a protected computer:
      • Modify the IPSec policy filters on all clients and servers that make security requests to the protected computer. Ensure that those clients have updated their policy before adding the new address.
      • Inspect the policy being used on the protected computer. If the filters specify static IP addresses for local connections, after adding the new IP address to the interface, edit and save the new filter list to include the new static IP address. My IP Address filters will be automatically updated when the new static IP address is added.
    • If the protected computer is a Web server and your clients use a proxy server, make sure that communications over all network paths are secured by IPSec:
      • Between the Web server and all clients that directly connect
      • Between the Web server and the proxy server
      • Between the proxy server and all clients of the proxy server
    • The following filters are internally defined to permit (not secure) this traffic:
      • Internet Key Exchange (IKE): Source Address = Any, Destination Address = Any, Protocol = UDP, Source port = 500, Destination port = 500
      • IP Multicast traffic
      • IP Broadcast traffic
      • Kerberos V5: Source Address = Any, Destination Address = Any, Protocol = UDP or TCP, Source port = 88, Destination port=88
      • Resource Reservation Protocol (RSVP): Source Address = Any, Destination Address = Any, Protocol = 46

      By default, all Kerberos and Resource Reservation Protocol (RSVP) traffic is permitted. However, you can secure Kerberos and RSVP traffic by editing the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPSEC. Add a new value named NoDefaultExempt and assign to it a value of 1. For more information about adding values to registry keys, see To add a value.

      Caution

      • Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To configure IP protocol, TCP port, or UDP port settings for a filter, see Related Topics.
  • To open IP Security Policies, see Related Topics.
  • For IPSec tunnel rules, only address-based filters are supported. Protocol-specific and port-specific filters are not supported. Tunnel filters should not be mirrored.
  • Filters are applied in the order of most-specific filters first. Filters are not applied in the order in which they appear in the list.
  • If an outbound packet does not match any filter, it is sent unsecured.
  • If an inbound packet does not match any filter, it is permitted.
  • If an IKE security request is received, the source IP address of the request is used to find a matching filter. The security action and tunnel setting that is associated with that filter determines the IKE response.
  • The A specific DNS Name option is used when creating filters to create IP address-based filters by resolving a DNS name to its IP addresses. The use of a computer name is for the one-time resolution of DNS names to IP addresses when creating the filter. The computer name is not used after the filters are created.
  • All filters used in tunnel rules are matched first before end-to-end transport filters are matched.

Related Topics


Configure advanced IPSec filter settings

Microsoft Windows graphic

To configure advanced IPSec filter settings

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that contains the IP filter list that you want to modify.
  3. On the IP Filter List tab, double-click the filter list that you want to modify.
  4. In IP Filter List, double-click the filter that you want to modify.
  5. In Filter Properties, on the Protocol tab, click a setting for Select a protocol type:
    6. To filter | Do this
Any packets sent on any protocol in the TCP/IP protocol suite | Click Any.
Traffic sent on a specific protocol | Click the protocol.
Packets sent on a custom protocol | Click Other, and then type the protocol number.
  6. If you chose TCP or UDP, you can also filter packets by the source port.
    • To filter packets sent on any port used by the protocol type that you selected, click From any port.
    • To filter packets sent on a specific port used by the protocol type that you selected, click From this port, and then type a port number.

    7. If you chose TCP or UDP, you can also filter packets by the destination port.

    • To filter packets received on any port used by the protocol type that you selected, click To any port.
    • To only filter packets received on the port number that you specify, click To this port.

Important

  • If you are unsure of these values, contact your system or network administrator to obtain this information or see the Windows Resource Kits in Related Topics. An erroneous value might cause communication to be blocked or unsecured.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • For IPSec tunneling, only address-based filters are supported. Protocol-specific and port-specific filters are not supported.
  • To open IP Security Policies, see Related Topics.

Related Topics


Select an IP filter list for a rule

Microsoft Windows graphic

To select an IP filter list for a rule

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that you want to modify.
  3. On the IP Filter List tab, select the IP filter list that you want to designate as the active list for this rule.

    4. The IP filter list you selected is now the active filter list for the rule.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Manage multiple IP filter lists

Microsoft Windows graphic

To manage multiple IP filter lists

  1. In IP Security Policies, in the console tree, click IP Security Policies on Name.
  2. On the Action menu, click Manage IP filter lists and filter actions.
  3. On the Manage IP Filters Lists tab, click Add. Or, if you are modifying an existing list, double-click that list.
  4. Define the filters for the filter list.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Define filter actions

  • Add or edit filter actions
  • Add or edit IPSec security methods
  • Configure advanced IPSec security methods
  • Select a filter action for a rule
  • Manage multiple filter actions


Add or edit filter actions

Microsoft Windows graphic

To add or edit filter actions

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that you want to modify, and then click the Filter Action tab.
  3. Choose whether or not to use the Filter Action Wizard:
    • To use the Filter Action Wizard to guide you through the process, confirm that the Use Add Wizard check box is selected, and then follow the instructions that are given.
    • To add a filter action manually, confirm that the Use Add Wizard check box is cleared, and then click Add.
  4. To edit a filter action, double-click the filter action.
  5. Choose a filter action type:
    • Click Permit to allow receiving or sending of packets in plaintext. Security will not be requested for these packets.
    • Click Block to discard packets. Security will not be requested for these packets.
    • Click Negotiate security to use the list of security methods in Security method preference order to provide security for packets. Security requests will be accepted for these packets.
  6. If you chose Negotiate security, add new security methods or edit the existing ones for the filter action.
  7. If you do not want to block incoming, unsecured communications, but you want to ensure that all outgoing communications and subsequent two-way communications are secured, select the Accept unsecured communication, but always respond using IPSec check box.
  8. If you want to enable communication with other computers that do not support IPSec and ensure that communication continues if there is no response to a request for IPSec negotiation, select the Allow unsecured communication with non-IPSec-aware computer check box. After the initial IPSec negotiation has failed, IPSec negotiation will be retried at five minute intervals.
  9. If you want to guarantee that no master keys or master keying material will be reused to generate the session key, select the Session key perfect forward secrecy (PFS) check box.
  10. On the General tab, in Name, type a unique name.
  11. In Description, type a description. For example, you might type what security levels this filter action represents.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • The filter action types Permit and Block and the options to Allow unsecured communication with non-IPSec-aware computer and Accept unsecured communication, but always respond using IPSec are not available in the default response rule.
  • Session key perfect forward secrecy (PFS) renegotiates new master key keying material during every session rekey operation. This is the most secure setting. However, it adds overhead and increases the time required to complete a rekey operation. Both initiator and responder must be configured with the same setting for successful negotiation.

Related Topics


Add or edit IPSec security methods

Microsoft Windows graphic

To add or edit IPSec security methods

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Click the rule you want to modify, and then click Edit.
  3. On the Filter Action tab, double-click the filter action that you want to modify.
  4. On the Security Methods tab, click Negotiate security (optional).
  5. Choose whether to add a security method or edit an existing one:
    • To add a new security method, click Add.
    • If you are reconfiguring an existing method, double-click the security method.
  6. Select the security level:
    • Encryption and Integrity

      • Use the ESP protocol to provide data confidentiality (encryption) with the triple Data Encryption Standard (3DES) algorithm, data integrity and authentication with the Secure Hash Algorithm 1 (SHA1) integrity algorithm, and default key lifetimes (100MB, 1 hour).

    • Integrity only

      • Use the ESP protocol to provide data integrity and authentication with the SHA1 integrity algorithm and default key lifetimes (100MB, 1 hour). ESP is not configured to provide data confidentiality (encryption).

    • Custom

      • Click Settings to configure a custom security method or key lifetimes.

  7. Repeat this procedure to configure additional security methods for the filter action.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • Click Remove to delete the selected security method.
  • Click Move up to move the selected security method up one level. Repeat until the security method is at the required preference level.
  • Click Move down to move the selected security method down one level. Repeat until the security method is at the required preference level.
  • For information about how key lifetimes affect security, see Related Topics.
  • When negotiating security, the initiator proposes this list of security methods to the responder. The responder agrees to the first method (from the top down) that is found anywhere in its own security methods list. If none of the proposed security methods match an entry in the responder's list, then the responder fails the negotiation.
  • Computers running Windows 2000 must have the High Encryption Pack or Service Pack 2 (or later) installed in order to use the 3DES algorithm. If a computer running Windows 2000 receives a 3DES setting, but does not have the High Encryption Pack or Service Pack 2 (or later) installed, the 3DES setting in the security method is set to the weaker DES. Computers running Windows XP support 3DES and do not require installation of the High Encryption Pack.

Related Topics


Configure advanced IPSec security methods

Microsoft Windows graphic

To configure advanced IPSec security methods

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that you want to modify.
  3. On the Filter Action tab, double-click the filter action that you want to modify.
  4. Choose whether to add a new security method or edit an existing one.
  5. To add a new security method, on the Security Methods tab, click Add.
  6. To reconfigure an existing method, double-click the security method.
  7. Click Custom, and then click Settings.
  8. To provide integrity for both the packet's addressing information (IP header) and the data, click Data and address integrity without encryption (AH).
  9. If you chose AH, click the algorithm you want to use for data integrity. You can click one of the following:
    • MD5 to use the Message Digest 5 (MD5) integrity algorithm, which uses a 128-bit key.
    • SHA1 to use the Secure Hash Algorithm 1 integrity algorithm, which uses a 160-bit key.
  10. To provide both integrity and encryption (confidentiality) for the data, click Data integrity and encryption (ESP).
  11. If you chose ESP, click the algorithm you want to use for data integrity. You can click one of the following:
    • <None> to use no data integrity.

      • If you have enabled AH, you can click None for the ESP integrity algorithm to improve performance.

    • MD5 to use the MD5 integrity algorithm, which uses a 128-bit key.
    • SHA1 to use the SHA1 integrity algorithm, which uses a 160-bit key.
  12. If you chose ESP, click the algorithm you want to use for data confidentiality. You can click one of the following:
    • <None> to use no encryption.
    • DES to use Data Encryption Standard (DES) with a 56-bit key.
    • 3DES to use the triple Data Encryption Standard with three 56-bit keys.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • You cannot choose <None> for both the ESP integrity and encryption algorithms.
  • For more information about cryptographic settings see Related Topics.
  • Use caution when specifying lifetimes. The default internal lifetimes should work in most environments and be interoperable with other products. If your security requirements demand shorter lifetime values, you can explicitly set appropriate lifetime values in each security method. In general, the lowest values that are explicitly set on initiator and responder will be used.
  • Using shorter lifetime values does not increase the security with which data is protected. Short key lifetimes will only decrease the amount of data that is revealed if an attacker discovers one encryption key. Instead of adjusting the lifetime values lower, consider using a stronger encryption algorithm to protect data, such as 3DES.
  • Before changing the default lifetimes, you should understand how lifetimes and rekeying work in detail. For more information, see Related Topics.

Related Topics


Select a filter action for a rule

Microsoft Windows graphic

To select a filter action for a rule

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Double-click the rule that you want to modify.
  3. On the Filter Action tab, select the filter action that you want to activate for this rule.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • If you have not removed them, you can activate one of the three predefined filter actions: Permit, Request Security (Optional), or Require Security.

Related Topics


Manage multiple filter actions

Microsoft Windows graphic

To manage multiple filter actions

  1. In IP Security Policies, in the console tree, click IP Security Policies on Name.
  2. Click Action, and then click Manage IP filter lists and filter actions.
  3. On the Manage Filter Actions tab, click Add. Or, if you are reconfiguring an existing filter action, in Filter Actions, click the action, and then click Edit.
  4. Configure the filter action.
  5. Define security methods for the filter action, if appropriate.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Define advanced IPSec settings

  • Configure key exchange
  • Create key exchange security methods


Configure key exchange

Microsoft Windows graphic

To configure key exchange

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Click the General tab, and then click Advanced.
  3. To force reauthentication and the negotiation of new master key keying material each time a new session key is required, click Master key perfect forward secrecy (PFS).
  4. If you require a different setting, type a value in Authenticate and generate a new key after every (minutes), which will cause a reauthentication and new master key generation at that interval.
  5. If you require a different setting, type a value in Authenticate and generate a new key after every (sessions) to set a maximum limit on the number of times a master key or its base keying material can be reused to generate the session key. This limit, when reached, will force a reauthentication and new master key generation. If you have enabled Master key perfect forward secrecy (PFS), the number of sessions is set to 1 and is not configurable.
  6. If you have special requirements for security methods on the master key exchange, click Methods.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • Portions of IPSec-related services were jointly developed by Microsoft and Cisco Systems, Inc.
  • Use Master key perfect forward secrecy (PFS) where it is required for interoperability. The default setting is disabled, which should work in most environments and be interoperable with most other products.
  • A session limit of zero (0) will cause rekeys to be determined only by the time setting.
  • Enabling master key perfect forward secrecy (PFS) might impact performance because each quick mode requires a new main mode negotiation.

Related Topics


Create key exchange security methods

Microsoft Windows graphic

To create key exchange security methods

  1. In IP Security Policies, double-click the policy that you want to modify.
  2. Click the General tab, click Advanced, and then click Methods.
  3. Click Add. Or, if you are reconfiguring an existing method, double-click the security method.
  4. Select an Integrity algorithm:
    • Click MD5 to use a 128-bit key (faster).
    • Click SHA1 to use a 160-bit key (stronger and Federal Information Processing Standard [FIPS]-compliant).
  5. Select an Encryption algorithm:
    • Click 3DES to use the triple Data Encryption Standard algorithm and three unique 56-bit keys.
    • Click DES to use the DES algorithm and a single 56-bit key. Use this option if you are required to connect to computers that do not have 3DES or if you do not need the higher security and overhead of 3DES.
  6. Select a Diffie-Hellman group to set the length of base keying material used to generate the actual keys:
    • Click Low (1) to generate 768 bits of master key keying material.
    • Click Medium (2) to generate 1024 bits of master key keying material (stronger).

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • Portions of IPSec-related services were jointly developed by Microsoft and Cisco Systems, Inc.
  • A key exchange security method is a combination of three settings (integrity algorithm, encryption algorithm, and Diffie-Hellman group). The initiator and the responder must have a method in common (one that uses the same settings) for negotiations to succeed.
  • Click Remove to delete a selected security method.
  • Click Move up to move a selected security method up one level. Repeat until the security method is at the required preference level.
  • Click Move down to move a selected security method down one level. Repeat until the security method is at the required preference level.
  • For more information about cryptographic settings, see Related Topics.

Related Topics


Assign IPSec policy

  • Assign or unassign IPSec policy in Group Policy
  • Assign or unassign IPSec policy on a computer


Assign or unassign IPSec policy in Group Policy

Microsoft Windows graphic

To assign or unassign IPSec policy in Group Policy

  • For Active Directory-based Group Policy:
    1. Open Active Directory Users and Computers.
    2. In the console tree, right-click the domain or organizational unit for which you want to set Group Policy.

      Where?

      • Active Directory Users and Computers [DomainControllerName.DomainName]
      • Domain
      • OrganizationalUnit
      • ChildOrganizationalUnit...
    3. Click Properties, and then click the Group Policy tab.
    4. Click Edit to open the Group Policy object that you want to edit. Or, click New to create a new Group Policy object, and then click Edit.
    5. In the Group Policy console tree, click IP Security Policies on Active Directory.

      Where?

      • PolicyName [ComputerName] Policy
      • Computer Configuration
      • Windows Settings
      • Security Settings
      • IP Security Policies on Active Directory
    6. In the details pane, right-click the policy that you want to assign, and then click Assign. To unassign the currently assigned policy, right-click the policy, and then click Unassign.
  • For local computer policy:
    1. Click Start, click Run, type MMC, and then click OK.
    2. In MMC, click File, click Add/Remove Snap-in, and then click Add.
    3. Click Group Policy Object Editor, and then click Add.
    4. Click Finish, click Close, and then click OK.
    5. In the Group Policy console tree, click IP Security Policies on Local Machine.

      Where?

      • Local Computer Policy
      • Computer Configuration
      • Windows Settings
      • Security Settings
      • IP Security Policies on Local Machine
    6. In the details pane, right-click the policy that you want to assign, and then click Assign. To unassign the currently assigned policy, right-click the policy, and then click Unassign.

Important

  • An IPSec policy might remain active even after the IPSec policy or Group Policy object to which it is assigned has been deleted. Therefore, you should unassign the IPSec policy before you delete either the policy or the Group Policy object. To prevent problems, use the following procedure:
    1. Unassign the IPSec policy in the Group Policy object.
    2. Wait 24 hours to ensure that the change is propagated.
    3. Delete the IPSec policy or Group Policy object.

    If you delete the IPSec policy or Group Policy object without following this procedure, computers in the Active Directory container to which the IPSec policy is assigned might treat the IPSec policy as if it cannot be located and continue to use a cached copy.

Notes

  • To start Active Directory Users and Computers, open a Remote Desktop Connection to either a Windows 2000 domain controller or a member server that has Windows 2000 Administration Tools installed. You must log on to the server as a domain administrator in order to complete this procedure.
  • You cannot administer Active Directory-based IPSec policy from a computer running Windows XP Home Edition.
  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • The settings will take effect the next time Group Policy is refreshed.
  • If a policy is currently assigned and you assign a new policy, the currently assigned policy is automatically unassigned.
  • You cannot assign a policy from the IP Security Policies on Active Directory snap-in, but can only configure policy. To assign policy for Active Directory containers, you must use the IP Security Policies on Active Directory within the Group Policy snap-in.
  • If you need to disable IPSec for only a specific computer, you can disable the IPSEC Services service on that computer.

Related Topics


Assign or unassign IPSec policy on a computer

Microsoft Windows graphic

To assign or unassign IPSec policy on a computer

  1. In IP Security Policies, in the console tree, click IP Security Policies on Name.
  2. Right-click the policy that you want to assign, and then click Assign. Or, to unassign the currently assigned policy, right-click the policy, and then click Unassign.

Notes

  • To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • If a policy is currently assigned and you assign a new policy, the currently assigned policy is automatically unassigned.
  • If you need to disable IP security, you can disable the IPSEC Services service on that computer.
  • If the computer is a member of a Windows 2000 domain and IPSec policy is assigned at the domain level, then the assigned local computer policy is not used.

Related Topics


Manage IPSec policy

  • Restart the IPSec services
  • Restore default IPSec policies
  • Export IPSec policies
  • Import IPSec policies
  • Refresh the list of IPSec policies
  • Delete an IPSec policy
  • Rename an IPSec policy
  • Export the list of IPSec policies


Restart the IPSec services

Microsoft Windows graphic

To restart the IPSec services

  1. Open Services.
  2. Right-click IPSEC Services.
  3. Click Restart.

Notes

  • XOX
  • This might be necessary to clear up old security associations (SAs), soft SAs, or to force a policy download from Active Directory to domain clients.
  • You can also restart the IPSec services from the Command Prompt by typing the following:

    • net stop policyagent

    net start policyagent

    On a computer running Windows 2000 Server and the Routing and Remote Access service, you must use the following sequence when restarting IPSec services:

    1. Stop the Routing and Remote Access service using the net stop remoteaccess command.
    2. Stop the IPSec services using the net stop policyagent command.
    3. Start the IPSec services using the net start policyagent command.
    4. Start the Routing and Remote Access service using the net start remoteaccess command.
  • Use Event Viewer to determine possible causes of failure if the IPSec service will not start.

Related Topics


Restore default IPSec policies

Microsoft Windows graphic

To restore default IPSec policies

  1. In IP Security Policies, in the console tree, click IP Security Policies on Name.
  2. Click Action, point to All Tasks, and then click Restore Default Policies.
  3. When prompted, click Yes.

Important

  • Restoring default IPSec policies will overwrite any changes to the original default policies, even when the names of the default policies have been changed. If you have modified the default policies and you do not want those modifications to be overwritten, do not restore the default policies.

Notes

  • To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • Restoring default IPSec policies is only available for computer-based IPSec policy. You cannot restore the default IPSec policies for Active Directory-based IPSec policy.

Related Topics


Export IPSec policies

Microsoft Windows graphic

To export IPSec policies

  1. In IP Security Policies, in the console tree, click IP Security Policies on Name.
  2. Click Action, point to All Tasks, and then click Export Policies.
  3. Type the path and the name of the file to which you want the policy information saved, and then click Save.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Import IPSec policies

Microsoft Windows graphic

To import IPSec policies

  1. In IP Security Policies, in the console tree, click IP Security Policies on Name.
  2. Click Action, point to All Tasks, and then click Import Policies.
  3. Type the path and the name of the file from which you are importing the IPSec policy information.
  4. Click Open.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • Importing IPSec policies replaces existing IPSec policies with the same name.

Related Topics


Refresh the list of IPSec policies

Microsoft Windows graphic

To refresh the list of IPSec policies

  1. In IP Security Policies, in the console tree, right-click IP Security Policies on Name.
  2. Click Refresh.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • This procedure is only required to update the list of policies for any additions or deletions made by using another management tool.

Related Topics


Delete an IPSec policy

Microsoft Windows graphic

To delete an IPSec policy

  1. In IP Security Policies, right-click the policy that you want to remove, and then click Delete.
  2. When prompted, click Yes.

Important

  • For Active Directory-based IPSec policy, an IPSec policy might remain active even after the IPSec policy has been deleted. Therefore, you should unassign the IPSec policy before you delete the policy. To prevent problems, use the following procedure:
    1. Unassign the IPSec policy in the Group Policy object.
    2. Wait 24 hours to ensure that the change is propagated.
    3. Delete the IPSec policy.

    If you delete the IPSec policy without following this procedure, computers in the Active Directory container to which the IPSec policy is assigned might treat the IPSec policy as if it cannot be located and continue to use a cached copy.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Rename an IPSec policy

Microsoft Windows graphic

To rename an IPSec policy

  1. In IP Security Policies, right-click the policy that you want to rename, and then click Rename.
  2. Type a new name, and then press Enter.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.

Related Topics


Export the list of IPSec policies

Microsoft Windows graphic

To export the list of IPSec policies

  1. In IP Security Policies, in the console tree, right-click IP Security Policies on Name, and then click Export List.
  2. If you only want to save the policy that is selected in the details pane, select the Save Only Selected Rows check box.
  3. Type the file name to which you want to save the selected IPSec policy or the displayed list of IPSec policies, and then click Save.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • To open IP Security Policies, see Related Topics.
  • This procedure does not export the IPSec policies, only a text list of the policies displayed in the details pane.

Related Topics


Monitor IPSec activity

  • Start the IP Security Monitor snap-in
  • Add a computer to the IP Security Monitor snap-in
  • View IP security statistics
  • View main mode details
  • View quick mode details
  • Find a matching specific filter
  • Change the auto update interval
  • Enable DNS name resolution


Start the IP Security Monitor snap-in

Microsoft Windows graphic

To start the IP Security Monitor snap-in

  1. Click Start, click Run, type MMC, and then click OK.
  2. In MMC, click File, click Add/Remove Snap-in, and then click Add.
  3. In the Add Standalone Snap-in dialog box, click IP Security Monitor, and then click Add.
  4. Click Close, and then click OK.

Notes

  • To save the console settings, on the File menu, click Save.
  • To view the saved console, see Related Topics.
  • To revise your initial choice, you can start the Microsoft Management Console, add this snap-in again, and save the console again. If you require multiple configurations, you can save the console with another name.

Related Topics


Add a computer to the IP Security Monitor snap-in

Microsoft Windows graphic

To add a computer to the IP Security Monitor snap-in

  1. Create a console containing IP Security Monitor. Or, open a saved console file containing IP Security Monitor.
  2. In the console tree, right-click IP Security Monitor, and then click Add computer.
  3. In the Add Computer dialog box:
    • For the local computer, click This computer.
    • For a remote computer, click The following computer, and then type the name of the remote computer. Or, click Browse to find it on the network.

Notes

  • To start IP Security Monitor, see Related Topics.
  • If the IPSec services are not started on the computer that is being monitored, the server icon is displayed as a stopped service. To refresh the IP Security Monitor after the IPSec services on that computer have been restarted, right-click the computer and then click Refresh.
  • The IP Security Monitor snap-in can only be used to monitor IPSec on computers running Windows XP. To monitor IPSec on a computer running Windows 2000, use the ipsecmon command at a Windows 2000 command prompt.

Related Topics


View IP security statistics

Microsoft Windows graphic

To view IP security statistics

  1. Create a console containing IP Security Monitor. Or, open a saved console file containing IP Security Monitor.
  2. Right-click a server name.

    Where?

    • IP Security Monitor
    • ServerName
  3. Click Statistics.

Note

  • To start IP Security Monitor, see Related Topics.

Related Topics


View main mode details

Microsoft Windows graphic

To view main mode details

  1. Create a console containing IP Security Monitor. Or, open a saved console file containing IP Security Monitor.
  2. Double-click Main Mode.

    Where?

    • IP Security Monitor
    • ServerName
    • Main Mode
  3. To view main mode details, do one of the following:
    • For generic filter details, in the console tree, click Generic Filters, and then in the details pane, double-click a filter.
    • For specific filter details, in the console tree, click Specific Filters. To find a specific main mode filter, right-click Specific Filters, and then click Find matching filters.
    • For IKE policy details, in the console tree, click IKE Policies, and then in the details pane, double-click a policy.
    • For security association details, in the console tree, click Security Associations, and then in the details pane, double-click a security association.

Note

  • To start IP Security Monitor, see Related Topics.

Related Topics


View quick mode details

Microsoft Windows graphic

To view quick mode details

  1. Create a console containing IP Security Monitor. Or, open a saved console file containing IP Security Monitor.
  2. Double-click Quick Mode.

    Where?

    • IP Security Monitor
    • ServerName
    • Quick Mode
  3. To view quick mode details, do one of the following:
    • For generic filter details, in the console tree, click Generic Filters, and then in the details pane, double-click a filter.
    • For specific filter details, in the console tree, click Specific Filters. To find a specific quick mode filter, right-click Specific Filters, and then click Find matching filters.
    • For negotiation policy details, in the console tree, click Negotiation Policies, and then in the details pane, double-click a policy.
    • For security association details, in the console tree, click Security Associations, and then in the details pane, double-click a security association.

Notes

  • To start IP Security Monitor, see Related Topics.
  • Quick mode details do not display blocking or permit filters.
  • Although there are two separate quick mode SAs established (one for the inbound traffic and one for the outbound traffic), IP Security Monitor only displays a single quick mode SA.

Related Topics


Find a matching specific filter

Microsoft Windows graphic

To find a matching specific filter

  1. Create a console containing IP Security Monitor. Or, open a saved console file containing IP Security Monitor.
  2. For a specific main mode filter:
    1. Double-click Main Mode.

      Where?

      • IP Security Monitor
      • ServerName
      • Main Mode
    2. In the console tree, right-click Specific Filters, and then click Find matching filters.
    3. In the Find Matching Main Mode Filters dialog box, configure the filter settings, and then click Search.
  3. For a specific quick mode filter:
    1. Double-click Quick Mode.

      Where?

      • IP Security Monitor
      • ServerName
      • Quick Mode
    2. In the console tree, right-click Specific Filters, and then click Find matching filters.
    3. In the Find Matching Quick Mode Filters dialog box, configure the filter settings, and then click Search.

Note

  • To start IP Security Monitor, see Related Topics.

Related Topics


Change the auto update interval

Microsoft Windows graphic

To change the auto update interval

  1. Create a console containing IP Security Monitor. Or, open a saved console file containing IP Security Monitor.
  2. Right-click a server name.

    Where?

    • IP Security Monitor
    • ServerName
  3. Click Properties.
  4. Configure the auto update refresh interval, and then click OK.

Note

  • To start IP Security Monitor, see Related Topics.

Related Topics


Enable DNS name resolution

Microsoft Windows graphic

To enable DNS name resolution

  1. Create a console containing IP Security Monitor. Or, open a saved console file containing IP Security Monitor.
  2. Right-click a server name.

    Where?

    • IP Security Monitor
    • ServerName
  3. Click Properties.
  4. Select the Enable DNS check box.

Notes

  • To start IP Security Monitor, see Related Topics.
  • The Enable DNS check box specifies whether you want to use Domain Name System (DNS) reverse queries and NetBIOS Adapter Status messages to resolve the DNS or NetBIOS computer name that is associated with an IP address when displaying sources or destinations. To resolve the DNS computer name from its IP address, your DNS infrastructure must have the appropriate reverse domains and pointer records configured, either manually or with DNS dynamic update. To resolve the NetBIOS computer name from its IP address, NetBIOS over TCP/IP must be enabled on the computer.

Related Topics


Concepts

This section covers:

  • Internet Protocol security overview
  • Understanding Internet Protocol security
  • Deploying Internet Protocol security
  • Administering Internet Protocol security
  • Resources


Internet Protocol security overview

This section covers:

  • Internet Protocol security defined
  • Security issues with IP
  • Introducing IPSec
  • In-depth defense
  • Protection against attacks
  • Layer 3 protection
  • Policy-based security
  • Simplified deployment


Internet Protocol security defined

Internet Protocol security (IPSec) is a framework of open standards for ensuring private, secure communications over Internet Protocol (IP) networks, through the use of cryptographic security services. The Microsoft Windows XP implementation of IPSec is based on standards developed by the Internet Engineering Task Force (IETF) IPSec working group.

IPSec is the long-term direction for secure networking. It provides aggressive protection against private network and Internet attacks through end-to-end security. The only computers that must know about IPSec protection are the sender and receiver in the communication. In Windows XP, IPSec provides the ability to protect communication between workgroups, local area network computers, domain clients and servers, branch offices (which might be physically remote), extranets, and roving clients.


Security issues with IP

Without security, both public and private networks are susceptible to unauthorized monitoring and access. Internal attacks might be a result of minimal or nonexistent intranet security. Risks from outside the private network originate from connections to the Internet and extranets. Password-based user access controls alone do not protect data transmitted across a network.

Common types of network attacks

Without security measures and controls in place, your data might be subjected to an attack. Some attacks are passive in that information is only monitored. Other attacks are active and information is altered with intent to corrupt or destroy the data or the network itself. Your networks and data are vulnerable to any of the following types of attacks if you do not have a security plan in place.

Eavesdropping

In general, the majority of network communications occur in a plaintext (unencrypted) format, which allows an attacker who has gained access to data paths in a network to monitor and interpret (read) the traffic. When an attacker is eavesdropping on communications, it is referred to as sniffing or snooping. The ability of an eavesdropper to monitor the network is generally the biggest security problem that administrators face in an enterprise. Without strong encryption services that are based on cryptography, data can be read by others as it traverses the network.

Data modification

After an attacker has read data, the next logical step is often to modify it. An attacker can modify the data in the packet without the knowledge of the sender or receiver. Even if you do not require confidentiality for all communications, you do not want any of your messages to be modified in transit. For example, if you are exchanging purchase requisitions, you do not want the items, amounts, or billing information to be changed.

Identity spoofing (IP address spoofing)

Most networks and operating systems use the IP address to identify a computer as being valid on a network. In some cases, it is possible for an IP address to be falsely used. This is known as identity spoofing. An attacker might use special programs to construct IP packets that appear to originate from valid addresses inside an organization intranet.

After gaining access to the network with a valid IP address, the attacker can modify, reroute, or delete data. The attacker can also conduct other types of attacks, as described in the following sections.

Password-based attacks

A commonality among most operating systems and network security plans is password-based access control. Access to both a computer and network resources are determined by a user name and password.

Earlier versions of operating system components did not always protect identity information as it was passed through the network for validation. This might allow an eavesdropper to determine a valid user name and password and use it to gain access to the network by posing as a valid user.

When an attacker finds and accesses a valid user account, the attacker has the same rights as the actual user. For example, if the user has administrator rights, the attacker can create additional accounts for access at a later time.

After gaining access to a network with a valid account, an attacker can do any of the following:

  • Obtain lists of valid user and computer names and network information.
  • Modify server and network configurations, including access controls and routing tables.
  • Modify, reroute, or delete data.

Denial-of-service attack

Unlike a password-based attack, the denial-of-service attack prevents normal use of a computer or network by valid users.

After gaining access to a network, an attacker can do any of the following:

  • Distract information systems staff so that they do not immediately detect the intrusion. This gives an attacker the opportunity to make additional attacks.
  • Send invalid data to applications or network services, causing applications or services to close or operate abnormally.
  • Send a flood of traffic until a computer or an entire network is shut down.
  • Block traffic, which results in a loss of access to network resources by authorized users.

Man-in-the-middle attack

As the name indicates, a man-in-the-middle attack occurs when someone between two users who are communicating is actively monitoring, capturing, and controlling the communication without the knowledge of the users. For example, an attacker can negotiate encryption keys with both users. Each user then sends encrypted data to the attacker, who can decrypt the data. When computers are communicating at low levels of the network layer, the computers might not be able to determine with which computers they are exchanging data.

Compromised-key attack

A key is a secret code or number required to encrypt, decrypt, or validate secured information. Although determining a key is a difficult and resource-intensive process for an attacker, it is possible. After an attacker determines a key, that key is referred to as a compromised key.

An attacker uses the compromised key to gain access to a secured communication without the sender or receiver being aware of the attack. With the compromised key, the attacker can decrypt or modify data. The attacker can also attempt to use the compromised key to compute additional keys, which might allow access to other secured communications.

Sniffer attack

A sniffer is an application or device that can read, monitor, and capture network data exchanges and packets. If the packets are not encrypted, a sniffer provides a full view of the data that is inside of the packet. Even encapsulated (tunneled) packets can be opened and read if they are not encrypted.

Using a sniffer, an attacker can do the following:

  • Analyze a network and access information, eventually causing the network to stop responding or become corrupted.
  • Read private communications.

Application-layer attack

An application-layer attack targets application servers by causing a fault in a server's operating system or applications. This results in the attacker gaining the ability to bypass normal access controls. The attacker takes advantage of this situation, gaining control of an application, system, or network, and can do any of the following:

  • Read, add, delete, or modify data or an operating system.
  • Introduce a virus that uses computers and software applications to copy viruses throughout the network.
  • Introduce a sniffer program to analyze the network and gain information that can eventually be used to cause the network to stop responding or become corrupted.
  • Abnormally close data applications or operating systems.
  • Disable other security controls to enable future attacks.


Introducing IPSec

IPSec is the long-term direction for secure networking. It provides a key line of defense against private network and Internet attacks, balancing security with ease of use.

IPSec has two goals:

  1. To protect the contents of IP packets.
  2. To provide a defense against network attacks through packet filtering and the enforcement of trusted communication.

Both goals are met through the use of cryptography-based protection services, security protocols, and dynamic key management. This foundation provides both the strength and flexibility to protect communications between private network computers, domains, sites, remote sites, extranets, and dial-up clients. It can even be used to block receipt or transmission of specific traffic types.

IPSec is based on an end-to-end security model, establishing trust and security from a source IP to a destination IP address. The IP address itself does not necessarily have to be considered an identity, rather the system behind the IP address has an identity that is validated through an authentication process. The only computers that must know about the traffic being secured are the sending and receiving computers. Each computer handles security at its respective end, with the assumption that the medium over which the communication takes place is not secure. Any computers that only route data from source to destination are not required to support IPSec, unless firewall-type packet filtering or network address translation is being done between the two computers. This model allows IPSec to be successfully deployed for the following enterprise scenarios:

  • Local area network (LAN): client/server and peer-to-peer
  • Wide area network (WAN): router-to-router and gateway-to-gateway
  • Remote access: dial-up clients and Internet access from private networks

Typically both sides require IPSec configuration (called an IPSec policy), to set options and security settings that will allow two systems to agree on how to secure traffic between them. The Windows XP implementation of IPSec is based on industry standards developed by the Internet Engineering Task Force (IETF) IPSec working group. Portions of IPSec-related services were jointly developed by Microsoft and Cisco Systems, Inc.


In-depth defense

Data must be protected from unauthorized interception, modification, or access. Network attacks can result in system downtime and public exposure to confidential information.

Network protection strategies generally focus only on preventing attacks from outside the private network by using firewalls, secure routers (security gateways), and user authentication for dial-up access. This is referred to as perimeter security, and it does not protect against attacks from within the network.

User access control security methods (for example, Kerberos V5 authentication), are not adequate to protect against most network-level attacks, because they rely solely on user names and passwords. Many computers are shared by multiple users. As a result, computers are often left in a state where the user name and password have already been validated. If a user name and password or a computer that has already been validated have been acquired by an attacker, user access security cannot stop the attacker's access to network resources.

Physical-level protection strategies, which are not commonly used, protect the physical network wires from being accessed and the network access points from being used. However, this rarely guarantees protection of the entire path that the data must travel through the network from source to destination.

The best level of protection is provided with the IPSec end-to-end model. In this model, the sending computer secures the data prior to transmission (before it reaches the network wires), and the receiving computer verifies the security of the data only after it has been received. For this reason, IPSec should be one of the components in a layered enterprise security plan. It protects your private data in a public environment by providing a strong, cryptography-based defense against attacks. All network traffic is secured at the packet level rather than for an entire communication (that is, a flow of packets). Used in combination with strong user access control, perimeter security, and physical level security, IPSec ensures an in-depth defense for your data.


Protection against attacks

IPSec protects data so that an attacker finds it extremely difficult or impossible to interpret it. The level of protection provided is determined by the strength of the security levels specified in your IPSec policy structure.

IPSec has a number of features that significantly reduce or prevent the attacks discussed in Security issues with IP:

  • Sniffers (lack of confidentiality)

    • The Encapsulating Security Payload (ESP) protocol in IPSec provides data confidentiality by encrypting the payload of IP packets.

  • Data modification

    • IPSec uses cryptography-based keys, shared only by the sending and receiving computers, to create a cryptographic checksum for each IP packet. Any modification to the packet data alters the checksum, which indicates to the receiving computer that the packet was modified in transit.

  • Identity spoofing, password-based, application-layer, and denial-of-service attacks

    • IPSec allows the exchange and verification of identities without exposing that information to interpretation by an attacker. Mutual verification (authentication) is used to establish trust between the communicating systems and only trusted systems can communicate with each other. After identities are established, IPSec uses cryptography-based keys, shared only by the sending and receiving computers, to create a cryptographic checksum for each IP packet. The cryptographic checksum ensures that only the computers that have knowledge of the keys could have sent each packet.

  • Man-in-the-middle attacks

    • IPSec combines mutual authentication with shared, cryptography-based keys.

  • Denial-of-service attacks

    • IPSec uses IP packet filtering methodology as the basis for determining whether communication is allowed, secured, or blocked, according to the IP address ranges, IP protocols, or even specific TCP and UDP ports.


Layer 3 protection

The common level of protection that IPSec provides requires system modification. However, the strategic implementation of IPSec at the IP level (layer 3) enables a high level of protection that is transparent for most applications, services, and upper layer protocols. Deploying IPSec requires no changes to existing applications or operating systems, and policies can be either centrally defined in Active Directory or managed locally on a computer.

The implementation of security at layer 3 can provide protection for all IP and upper layer protocols in the TCP/IP protocol suite, such as TCP, UDP, ICMP, and custom protocols that send traffic at the IP layer. The primary benefit of securing information at a lower layer is that all applications and services using IP for transport of data can be protected with IPSec without any modification to those applications or services.

Other security services that operate above layer 3, such as Secure Sockets Layer (SSL), only provide security to applications that can use SSL (for example, Web browsers). To protect communications for all of the applications on your computer with SSL, you must modify each application. Security services that operate below layer 3, such as link layer encryption, protect only the link, and not necessarily all links along the data path. This makes link layer encryption unsuitable for end-to-end data protection on the Internet or in routed intranet scenarios.


Policy-based security

Although stronger security methods based on cryptography have become necessary to fully protect communication, they can often greatly increase administrative overhead. To reduce overhead, IPSec uses policy-based administration.

IPSec policies, rather than application programming interfaces (APIs), are used to configure IPSec security services. The policies provide variable levels of protection for most traffic types in most existing networks.

You can configure IPSec policies to meet the security requirements of a computer, application, organizational unit, domain, site, or global enterprise. You can use the IP Security Policies snap-in provided in Windows XP to define IPSec policies for computers through Active Directory (for domain members), or on the local computer (for computers that do not belong to domains).


Simplified deployment

To achieve secure communications with a low cost of ownership, Windows XP simplifies the deployment of IPSec with the following features:

  • Integration with the Windows 2000 security framework

    • IPSec uses the Windows 2000 secure domain as a trust model. By default, IPSec policies use the Active Directory default authentication method (Kerberos V5 authentication) to identify and trust communicating computers. Computers that are members of a Windows 2000 domain and are in trusted domains can easily establish IPSec secured communications.

  • Centralized IPSec policy administration through Active Directory

    • IPSec policies can be assigned through Group Policy configuration of Active Directory domains and organizational units. This allows the IPSec policy to be assigned at the domain, site, or organizational unit level, eliminating the administrative overhead of configuring each computer separately.

  • Transparency of IPSec to users and applications

    • Integration at the IP layer (layer 3) provides security for any protocol in the TCP/IP suite of protocols. You do not need separate security for each protocol in the TCP/IP suite of protocols, because applications using TCP/IP pass the data to the IP protocol layer, where it is secured.

  • Flexible security configuration

    • The security services within each policy can be customized to meet the majority of security requirements for the network and data traffic.

  • Automatic key management

    • Internet Key Exchange (IKE) services dynamically exchange and manage cryptographic keys between communicating computers.

  • Automatic security negotiation

    • Internet Key Exchange (IKE) services dynamically negotiate a common set of security settings between communicating computers, eliminating the need for both computers to have identically configured policies.

  • Public key infrastructure support

    • Using public key certificates for authentication is supported. This allows trust and secure communication for computers that do not belong to a Windows 2000 trusted domain, non-Windows-brand operating systems, computers that have membership in untrusted domains, and instances in which computer access must be restricted to a smaller group than domain authentication allows.

  • Preshared key support

    • If authentication using the Kerberos V5 protocol or public key certificates is not possible, a preshared authentication key can be configured.

Note

  • The integration with the Windows 2000 security framework, centralized IPSec policy administration through Active Directory, and use of the Kerberos V5 protocol described here does not apply to computers running Windows XP Home Edition.


Understanding Internet Protocol security

This section introduces the key concepts and terminology you should review before you deploy IPSec. This section covers:

  • IPSec security services
  • IPSec protocol types
  • IPSec components
  • The IPSec process
  • Virtual private networking with IPSec
  • Description of an IPSec policy


IPSec security services

IPSec provides a high level of security by using cryptography-based services. Cryptography allows information to be transmitted securely by hashing (to provide integrity and authentication) and encrypting (to provide confidentiality) the information.

A combination of an algorithm and a key is used to secure information. The algorithm is the mathematical process by which the information is secured. The key is the secret code or number required to secure, read, modify, or verify secured data.

IPSec provides or utilizes the following security services:

  • Security properties
  • Public key certificate-based authentication
  • Preshared key authentication
  • Public key cryptography
  • Data integrity with hash functions
  • Data confidentiality with encryption
  • Key management


Security properties

IPSec provides the following properties for secured communications:

  • Anti-replay

    • Ensures the uniqueness of each IP packet. Anti-replay is also called replay prevention. Anti-replay ensures that data captured by an attacker cannot be reused or replayed to establish a session or gain information illegally. This protects against attempts to intercept a message and then use the identical message to illegally gain access to resources, possibly months later.

  • Integrity

    • Protects data from unauthorized modification in transit, ensuring that the data received is exactly the same as the data sent. Hash functions sign each packet with a cryptographic checksum using a shared, secret key. Only the sender and receiver have the key that is used to calculate the checksum. If the packet contents have changed, the checksum verification fails and the packet is discarded.

  • Confidentiality (encryption)

    • Ensures that data is only disclosed to intended recipients. This is achieved by encrypting the data before transmission, ensuring that the data cannot be read during transmission, even if the packet is monitored or intercepted. Only the communicating computers with the shared, secret key are able to read the data after it has been decrypted. This property is optional and is dependent on IPSec policy settings.

  • Authentication

    • Verifies that a message could only have been sent from a computer that has knowledge of the shared, secret key. The sender includes (with the message) a message authentication code with a calculation that includes the shared, secret key. The receiver performs the same calculation and, if the receiver’s calculation does not match the message authentication code that is included in the message, the message is discarded. The message authentication code is the same as the cryptographic checksum that is used for integrity.


Public key certificate-based authentication

An effectively implemented public key infrastructure, in which security credentials can be presented without compromising them in the process, resolves many security problems. IPSec works with your public key infrastructure to enable certificate-based authentication of computers.

A public key certificate, which ensures that your true identity does not vary from your reported identity, is one type of authentication that reliably provides this verification.

Public key certificates are like digital passports. They are used to verify the identities of non-Windows-brand operating system computers, stand-alone computers, clients that are not members of a trusted domain, or computers that are not running the Kerberos V5 authentication protocol (the default Windows XP authentication method).


Preshared key authentication

IPSec can use preshared keys for authentication. Preshared means that the parties agree on a shared, secret key that becomes part of the IPSec policy. During security negotiation, information is encrypted by using the shared key before transmission, and decrypted using the same key on the receiving end. If the receiver can decrypt the information, identities are considered to be authenticated.

Frequent use of preshared key authentication is not recommended because the authentication key is stored (in an unprotected state) in the IPSec policy. Preshared key methodology is provided for interoperability purposes and to adhere to IPSec standards.


Public key cryptography

IPSec implements public key cryptography for authentication (certificate signing) and key exchange (the Diffie-Hellman algorithm). Public key cryptography has all of the capabilities of secret key cryptography, but is generally more secure because it requires two keys; one for signing and encrypting the data, and one for verifying the signature and decrypting the data. This is called asymmetric cryptography, which indicates that two keys are required for the process.

Each user has a private key (also called a secret key) that is not known to anyone else, and a public key that is widely distributed. For example, if Alice wants to send a secured message to Bob, she uses Bob's public key to encrypt the message. Only Bob can decrypt the message because decryption requires his private key. Although the keys in the pair are related, it is mathematically infeasible to determine the private key from the public key.


Data integrity with hash functions

Hash message authentication codes (HMAC) sign packets to verify that the information received is exactly the same as the information sent. This is called integrity and is critical when data is exchanged over unsecured media.

HMACs provide integrity through a keyed hash, the result of a mathematical calculation on a message using a hash function (algorithm) combined with a shared, secret key. A hash is commonly described as a signature on the packet. However, a hash differs from a digital signature. A hash uses a secret, shared key, and a digital signature uses public key technology and the private key of the sending computer. A digital signature provides nonrepudiation and a hash does not. Hash functions are also called one-way functions. In one-way functions, it is easy to determine the hash from the message but mathematically infeasible to determine the message from the hash. Conversely, in two-way functions, the original message can be determined from its converted form. Encryption and decryption schemes are examples of two-way functions.

The hash is a cryptographic checksum or message integrity code (MIC) that each party must compute to verify the message. For example, the sending computer uses a hash function and shared key to compute the checksum for the message, including it with the packet. The receiving computer must perform the same hash function on the received message and shared key and compare it to the original (included in the packet from the sender). If the message has changed in transit, the hash values are different and the packet is rejected.

For integrity, you can choose between two hash functions when setting policy:

  • MD5

    • Message Digest 5 (MD5) is based on RFC 1321. It was developed in response to a weakness found in MD4. MD5 completes four passes over the data blocks (MD4 completed three passes), using a different numeric constant for each word in the message on each pass. The number of 32-bit constants used during the MD5 computation equates to 64, ultimately producing a 128-bit hash that is used for the integrity check. While MD5 is more resource intensive, it provides stronger integrity than MD4.

  • SHA1

    • Secure Hash Algorithm 1 (SHA1) was developed by the National Institute of Standards and Technology as described in Federal Information Processing Standard (FIPS) PUB 180-1. The SHA process is closely modeled after MD5. The SHA1 computation results in a 160-bit hash that is used for the integrity check. Because longer hash lengths provide greater security, SHA is stronger than MD5.


Data confidentiality with encryption

IPSec in Windows XP uses the US Data Encryption Standard (DES) to provide confidentiality (data encryption). The DES algorithm was published in 1977 by the US National Bureau of Standards. IPSec enables the ability to frequently regenerate keys during a communication. This prevents the entire data set from being compromised if one DES key is broken.

DES is a block cipher that uses a 56-bit key. A block cipher is an encryption algorithm that operates on a fixed size block of data. DES encrypts data in 64-bit blocks using a 64-bit key. The key appears to be a 64-bit key, but one bit in each of the 8 bytes is used for error checking, resulting in 56 bits of usable key.

Cipher block chaining (CBC) is also used to hide patterns of identical blocks of data within a packet. An initialization vector (an initial random number) is used as the first random block to encrypt and decrypt a block of data. Different random blocks are used in conjunction with the secret key to encrypt each successive block. This ensures that identical sets of unsecured data (plaintext) result in unique, encrypted data blocks.

Windows XP IPSec supports the use of:

  • DES

    • Used when the high security and overhead of 3DES are not necessary.

  • 3DES

    • Used when high security is required. 3DES processes each block three times, using a unique 56-bit key each time:

    1. Encryption on the block with key 1
    2. Decryption on the block with key 2
    3. Encryption on the block with key 3

    This process is reversed if the computer is decrypting a packet.


Key management

A key is a secret code or number that is required to read, modify, or verify secured data. Keys are used in conjunction with algorithms (a mathematical process) to secure data. Windows XP automatically handles key generation and implements the following keying properties that maximize protection:

  • Dynamic rekeying

    • IPSec policy controls how often a new key is generated during communication, using a method called dynamic rekeying. The communication is sent in blocks, and each block of data is secured with a different key. This prevents an attacker who has obtained part of a communication, and the corresponding session keys, from obtaining the remainder of the communication. This on-demand security negotiation and automatic key management service is provided using IKE, as defined in RFC 2409, "The Internet Key Exchange (IKE)."

    IPSec policy allows you to control how often a new key is generated. If no values are configured, keys are regenerated automatically at default intervals.

  • Key lengths

    • Every time the length of a key is increased by one bit, the number of possible keys doubles, making it exponentially more difficult to determine the key. IPSec policy provides multiple algorithms to allow for short or long key lengths.

  • Key material generation: The Diffie-Hellman algorithm

    • To enable secure communication, two computers must be able to gain the same shared key (session key), without sending the key across a network and compromising the secret.

    The Diffie-Hellman algorithm (DH) predates Rivest-Shamir-Adleman (RSA) encryption and offers better performance. It is one of the oldest and most secure algorithms used for key exchange. The two parties publicly exchange keying information, which Windows XP additionally protects with a hash function signature. Neither party ever exchanges the actual key; however, after their exchange of keying material, each is able to generate the identical shared key.

    DH keying material exchanged by the two parties can be based on 768 or 1024 bits of keying material, known as DH groups. The strength of the DH group is proportional to the strength of the key computed from the DH exchange. Strong DH groups combined with longer key lengths increase the degree of computational difficulty in trying to determine the key.

    IPSec uses the DH algorithm to provide the keying material for all other encryption keys. DH does not provide authentication. In the Windows XP IPSec implementation, identities are authenticated after the DH exchange takes place, providing protection against man-in-the-middle attacks.


IPSec protocol types

IPSec protocols provide data and identity protection for each IP packet by adding their own security protocol header to each packet.

For information about AH and ESP when used in transport and tunnel mode, see the following topics:

  • Transport mode
  • Tunnel mode


Transport mode

Transport mode provides the protection of an IP payload through an AH or ESP header. Typical IP payloads are TCP segments (containing a TCP header and TCP segment data), a UDP message (containing a UDP header and UDP message data), and an ICMP message (containing an ICMP header and ICMP message data).

Authentication Header transport mode

Authentication Header (AH) provides authentication, integrity, and anti-replay for the entire packet (both the IP header and the data payload carried in the packet). It does not provide confidentiality, which means that it does not encrypt the data. The data is readable, but protected from modification. AH uses keyed hash algorithms to sign the packet for integrity. For more information, see Data integrity with hash functions.

For example, Alice on Computer A sends data to Bob on Computer B. The IP header, the AH header, and the data are protected with integrity. This means Bob can be certain it was really Alice who sent the data and that the data was unmodified.

Integrity and authentication are provided by the placement of the AH header between the IP header and the IP payload, as shown in the following illustration.

Authentication Header

AH is identified in the IP header with an IP protocol ID of 51. AH can be used alone or combined with the Encapsulating Security Payload (ESP) protocol.

The AH header contains the following fields:

  • Next Header

    • Identifies the IP payload by using the IP protocol ID. For example, a value of 6 represents TCP.

  • Length

    • Indicates the length of the AH header.

  • Security Parameters Index (SPI)

    • Used in combination with the destination address and the security protocol (AH or ESP) to identify the correct security association for the communication. The receiver uses this value to determine with which security association the packet is identified.

  • Sequence Number

    • Provides anti-replay protection for the SA. The sequence number is a 32-bit, incrementally increasing number (starting from 1) that indicates the packet number sent over the security association for the communication. The sequence number cannot repeat for the life of the security association. The receiver checks this field to verify that a packet for a security association with this number has not already been received. If one has been received, the packet is rejected.

  • Authentication Data

    • Contains the integrity check value (ICV), also known as the message authentication code, which is used to verify both message authentication and integrity. The receiver calculates the ICV value and checks it against this value (which is calculated by the sender) to verify integrity. The ICV is calculated over the IP header, the AH header, and the IP payload.

Packet signature with the AH header

AH signs the entire packet for integrity, with the exception of some fields in the IP header which might change in transit (for example, the Time to Live and Type of Service fields). If another IPSec header is being used in addition to AH, the AH header is inserted before any other IPSec headers. The AH packet signature is shown in the following illustration.

Packet signature with the AH header

Encapsulating Security Payload transport mode

Encapsulating Security Payload (ESP) provides confidentiality (in addition to authentication, integrity, and anti-replay) for the IP payload. ESP in transport mode does not sign the entire packet. Only the IP payload (not the IP header) is protected. ESP can be used alone or in combination with AH.

For example, Alice on Computer A sends data to Bob on Computer B. The IP payload is encrypted and signed for integrity. Upon receipt, after the integrity verification process is complete, the data payload in the packet is decrypted. Bob can be certain that it was Alice who sent the data, the data is unmodified, and no one else was able to read it.

ESP is identified in the IP header with the IP protocol ID of 50. As shown in the following illustration, the ESP header is placed before the IP payload, and an ESP trailer and ESP authentication trailer is placed after the IP payload.

ESP Header

The ESP header contains the following fields:

  • Security Parameters Index

    • Identifies the correct security association for the communication when used in combination with the destination address and the security protocol (AH or ESP). The receiver uses this value to determine the security association with which this packet should be identified.

  • Sequence Number

    • Provides anti-replay protection for the SA. The sequence number is a 32-bit, incrementally increasing number (starting from 1) that indicates the packet number sent over the security association for the communication. The sequence number cannot repeat for the life of the security association. The receiver checks this field to verify that a packet for a security association with this number has not already been received. If one has been received, the packet is rejected.

The ESP trailer contains the following fields:

  • Padding

    • Padding of 0 to 255 bytes is used to ensure that the encrypted payload with the padding bytes are on byte boundaries required by encryption algorithms.

  • Padding Length

    • Indicates the length of the Padding field in bytes. The receiver uses this field to remove padding bytes after the encrypted payload with the padding bytes has been decrypted.

  • Next Header

    • Identifies the type of data in the payload, such as TCP or UDP.

The ESP authentication trailer contains the following field:

  • Authentication Data

    • Contains the integrity check value (ICV), also known as the message authentication code, which is used to verify both message authentication and integrity. The receiver calculates the ICV value and checks it against this value (which is calculated by the sender) to verify integrity. The ICV is calculated over the ESP header, the payload data, and the ESP trailer.

Packet signature and encryption

As shown in the following illustration, ESP provides protection for IP payloads. The signed portion of the packet indicates where the packet has been signed for integrity and authentication. The encrypted portion of the packet indicates what information is protected with confidentiality.

Encapsulating Security Payload

The IP header is not signed and is not necessarily protected from modification. To provide data integrity and authentication for the IP header, use ESP and AH.


Tunnel mode

Tunnel mode provides the protection of an entire IP packet by treating it as an AH or ESP payload. With tunnel mode, an entire IP packet is encapsulated with an AH or ESP header and an additional IP header. The IP addresses of the outer IP header are the tunnel endpoints, and the IP addresses of the encapsulated IP header are the ultimate source and destination addresses.

AH tunnel mode

As shown in the following illustration, AH tunnel mode encapsulates an IP packet with an AH and IP header and signs the entire packet for integrity and authentication.

AH tunnel mode

ESP tunnel mode

As shown in the following illustration, ESP tunnel mode encapsulates an IP packet with both an ESP and IP header and an ESP authentication trailer.

ESP tunnel mode

The signed portion of the packet indicates where the packet has been signed for integrity and authentication. The encrypted portion of the packet indicates what information is protected with confidentiality.

Because a new header for tunneling is added to the packet, everything that comes after the ESP header is signed (except for the ESP authentication trailer) because it is now encapsulated in the tunneled packet. The original header is placed after the ESP header. The entire packet is appended with an ESP trailer before encryption occurs. Everything that follows the ESP header, except for the ESP authentication trailer, is encrypted. This includes the original header which is now considered to be part of the data portion of the packet.

The entire ESP payload is then encapsulated within the new tunnel header, which is not encrypted. The information in the new tunnel header is used only to route the packet from origin to destination.

If the packet is being sent across a public network, it is routed to the IP address of the tunnel server for the receiving intranet. In most cases, the packet is destined for an intranet computer. The tunnel server decrypts the packet, discards the ESP header, and uses the original IP header to route the packet to the intranet computer.

ESP and AH can be combined when tunneling, providing both confidentiality for the tunneled IP packet and integrity and authentication for the entire packet.

Using IPSec tunnels

IPSec tunnels provide security for IP traffic only. The tunnel is configured to protect traffic between either two IP addresses or two IP subnets. If the tunnel is used between two computers instead of two routers (also known as gateways), the IP address outside the AH or ESP payload is the same as the IP address inside the AH or ESP payload. In Windows XP, IPSec does not support protocol-specific or port-specific tunnels. Configuration is performed using the IP Security Policies and Group Policy snap-ins by configuring and enabling two rules:

  1. A rule for the outbound traffic for the tunnel.

    2. The rule for the outbound traffic is configured with a filter list that describes the traffic to be sent through the tunnel and a tunnel endpoint of an IP address configured on the IPSec tunnel peer (the computer or router on the other side of the tunnel).

  2. A rule for the inbound traffic for the tunnel.

    3. The rule for the inbound traffic is configured with a filter list that describes the traffic to be received through the tunnel and a tunnel endpoint of a local IP address (the computer or router on the local side of the tunnel).

Additionally, filter actions, authentication methods, and other settings need to be specified for each rule.

For conceptual information about IPSec policy tunnel settings, see Tunnel endpoint. For information about configuring an IPSec tunnel, see To specify an IPSec tunnel. For information about how tunneling is used for virtual private networking, see Virtual private networking with IPSec.


IPSec components

The following topics describe IPSec components that are installed with Windows XP:

  • IPSec Policy Agent service
  • Internet Key Exchange
  • Key protection
  • IPSec driver


IPSec Policy Agent service

The purpose of the IPSec Policy Agent is to retrieve policy information and pass it to other IPSec components that require this information to perform security services, as shown in the following illustration.

IPSec Policy Agent

The IPSec Policy Agent is a service that resides on each Windows XP computer, appearing as IPSEC Services in the list of system services in the Services snap-in. The IPSec Policy Agent:

  • Retrieves the appropriate IPSec policy (if one has been assigned) from Active Directory, if the computer is a domain member, or from the local registry, if the computer is not a member of a domain.
  • Polls Active Directory for changes in policy configuration (for domain-based policy).
  • Sends the assigned IPSec policy information to the IPSec driver.

If the computer is a member of a domain, policy retrieval occurs when the system starts, at the interval specified in the IPSec policy, and at the default Winlogon polling interval. You can also manually poll Active Directory for policy using the gpupdate /target:computer command.

The following are additional aspects of IPSec policy behavior for a computer that is a member of a domain:

  • If IPSec policy information is centrally configured for computers that are domain members, the IPSec policy information is stored in Active Directory and cached in the local registry of the computer to which it applies.
  • If the computer is temporarily not connected to the domain and policy is cached, new policy information for that computer replaces old, cached information when the computer reconnects to the domain.
  • If the computer is a stand-alone computer or a member of a domain that is not using Active Directory for policy storage, IPSec policy is stored in the local registry.

The IPSec Policy Agent starts automatically at system start time. If there are no IPSec policies in Active Directory or the registry, or if the IPSec Policy Agent cannot connect to Active Directory, the IPSec Policy Agent waits for policy to be assigned or activated.


Internet Key Exchange

Before secured data can be exchanged, a contract between the two computers must be established. In this contract, called a security association (SA), both agree on how to exchange and protect information, as shown in the following illustration.

ISAKMP/Oakley Service

To build this contract between the two computers, the IETF has established a standard method of security association and key exchange resolution named Internet Key Exchange (IKE) which:

  • Centralizes security association management, reducing connection time.
  • Generates and manages shared, secret keys that are used to secure the information.

This process not only protects communication between computers, it also protects remote computers that request secure access to a corporate network. In addition, this process works whenever the negotiation for the final destination computer (endpoint) is performed by a security gateway.

Security association (SA) defined

A security association (SA) is the combination of a negotiated key, security protocol, and security parameters index (SPI), which together define the security used to protect the communication from sender to receiver. The SPI is a unique, identifying value in the SA that is used to distinguish among multiple security associations that exist at the receiving computer. For example, multiple associations might exist if a computer is securely communicating with multiple computers at the same time. This is a common occurrence when the computer is a file server or a remote access server that serves multiple clients. In these situations, the receiving computer uses the SPI to determine which SA is used to process the incoming packets.

Phase I or main mode SA

In order to ensure successful and secure communication, IKE performs a two-phase operation. Confidentiality and authentication are ensured during each phase by the use of encryption and authentication algorithms that are agreed upon by the two computers during security negotiations. With the duties split between two phases, key creation can be rapidly accomplished.

During the first phase, the two computers establish a secure, authenticated channel. This is called the phase I or main mode SA. IKE automatically provides necessary identity protection during this exchange.

Phase I or main mode negotiation

The following are the steps that comprise a main mode negotiation.

  1. Policy negotiation

    2. The following four mandatory parameters are negotiated as part of the main mode SA:

    • The encryption algorithm (DES, 3DES)
    • The hash algorithm (MD5 or SHA1)
    • The authentication method (Kerberos V5, Certificate, or preshared key authentication)
    • The Diffie-Hellman (DH) group to be used for the base keying material

    If certificates or preshared keys are used for authentication, the computer identity is protected. However, if Kerberos V5 authentication is used, the computer identity is unencrypted until encryption of the entire identity payload takes place during authentication.

  2. DH exchange (of public values)

    3. At no time are actual keys exchanged. Only the base information required by the DH key determination algorithm to generate the shared, secret key is exchanged. After this exchange, the IKE service on each computer generates the master key that is used to protect authentication.

  3. Authentication

    4. The computers attempt to authenticate the DH key exchange. Without authenticating the DH key exchange, the communication is vulnerable to a man-in-the-middle attack. Without successful authentication, communication cannot proceed. The master key is used, in conjunction with the negotiation algorithms and methods, to authenticate identities. The entire identity payload (including the identity type, port, and protocol) is hashed and encrypted using the keys generated from the DH exchange in the second step. The identity payload, regardless of which authentication method is used, is protected from both modification and interpretation.

The sender presents an offer for a potential security association to the receiver. The responder cannot modify the offer. Should the offer be modified, the initiator rejects the responder's message. The responder sends either a reply accepting the offer or a reply with alternatives.

Messages sent during this phase have an automatic retry cycle that is repeated five times. If a response is received before the retry cycle ends, standard SA negotiation begins. If allowed by IPSec policy, unsecured communications will begin after a brief interval. This behavior is known as fall back to clear. Even if the communication falls back to clear, secure communication negotiation is attempted at 5 minute intervals.

There is no limit to the number of exchanges that can take place. The number of SAs established is only limited by system resources.

For more information about fall back to clear, see Filter action.

Phase II or quick mode SA

In this phase, SAs are negotiated on behalf of the IPSec driver.

Phase II or quick mode negotiation

The following are the steps that comprise a quick mode negotiation.

  1. Policy negotiation occurs.

    2. The IPSec computers exchange the following requirements for securing the data transfer:

    • The IPSec protocol (AH or ESP)
    • The hash algorithm for integrity and authentication (MD5 or SHA1)
    • The algorithm for encryption, if requested (3DES or DES)

    A common agreement is reached, and two SAs are established. One SA is for inbound communication and the other is for outbound communication.

  2. Session key material is refreshed or exchanged.

    3. IKE refreshes the keying material and new shared keys are generated for packet integrity, authentication, and encryption (if negotiated). If rekeying is required, either a second DH exchange (as described in main mode negotiation) occurs, or a refresh of the original DH key is used.

  3. The SAs and keys, along with the SPI, are passed to the IPSec driver.

The second negotiation of security settings and keying material (for the purpose of securing data) is protected by the main mode SA. As the first phase provided identity protection, the second phase provides protection by refreshing the keying material prior to sending data. IKE can accommodate a key exchange payload for an additional DH exchange if a rekey is necessary—that is, master key perfect forward secrecy (PFS) is enabled. Otherwise, IKE refreshes the keying material from the DH exchange completed in main mode.

Quick mode results in a pair of security associations, each with its own SPI and key. One SA is used for inbound communication, and the other for outbound communication.

Note

  • Although there are two separate quick mode SAs established, IP Security Monitor only displays a single quick mode SA.

The retry algorithm for a message is similar to the process described in main mode negotiation. However, if this process times out for any reason during the second or higher negotiation off of the same main mode SA, a renegotiation of the main mode SA is attempted. If a message for this phase is received without an established main mode SA, it is rejected.

Using a single main mode SA for multiple quick mode SA negotiations increases the speed of the process. As long as the main mode SA does not expire, renegotiation and reauthentication are not necessary. The number of quick mode SA negotiations that can be performed is determined by IPSec policy settings.

Note

  • Excessive rekeying off of the same main mode SA might make the shared, secret key vulnerable to a known plaintext attack. A known plaintext attack is a sniffer attack in which the attacker attempts to determine the encryption key from encrypted data based on known plaintext.

SA lifetimes

The main mode SA is cached to allow multiple quick mode SA negotiations (unless master key PFS is enabled). When a key lifetime is reached for the master or session key, the SA is renegotiated. In addition, the key is refreshed or regenerated.

When the default time-out period elapses for the main mode SA, or the master or session key lifetime is reached, a delete message is sent to the responder. The IKE delete message tells the responder to expire the main mode SA. This prevents additional new quick mode SAs from being created from the expired main mode SA. IKE does not expire the quick mode SA, because only the IPSec driver contains the number of seconds or bytes that have passed to reach the key lifetime.

Use caution when setting very different key lifetimes for master and session keys. For example, setting a master key lifetime of eight hours and a session key lifetime of two hours might leave a quick mode SA in place for almost two hours after the main mode SA has expired. This occurs when the quick mode SA is generated shortly before main mode SA expiration.


Key protection

The following features enhance the base prime numbers (keying material) and the strength of the keys for the master and session keys.

Key lifetimes

Key lifetimes determine when a new key is generated, rather than how it is generated. Also known as dynamic rekeying or key regeneration, a key lifetime allows you to force a key regeneration after a specific interval. For example, if a communication takes 10,000 seconds and you specify a key lifetime of 1,000 seconds, 10 keys are generated to complete the communication. This ensures that even if an attacker is able to decipher part of a communication, the remainder of the communication is protected. Key lifetimes can be specified for both the master and session keys. Whenever a key lifetime is reached, the SA is also renegotiated. In addition, the key is refreshed or regenerated. The amount of data processed by a single key should not exceed 100 megabytes. You can check current security and encryption guidelines to provide adequate protection for the type of data that is being communicated.

Session key refresh limit

Repeated rekeying off of a session key can compromise the Diffie-Hellman shared secret. Because of this, a session key refresh limit is implemented.

For example, Alice on Computer A sends a message to Bob on Computer B, and then sends another message to Bob a few minutes later. The same session key material might be reused because an SA was recently established. If you want to limit the number of times this occurs, set the session key refresh limit to a low number.

If you have enabled master key perfect forward secrecy (PFS), the session key refresh limit is not used. Setting a session key refresh limit to 1 is identical to enabling master key PFS. If both a master key lifetime and a session key refresh limit are specified, whichever limit is reached first causes the subsequent rekey. By default, IPSec policy does not specify a session key refresh limit.

Diffie-Hellman groups

Diffie-Hellman (DH) groups are used to determine the length of the base prime numbers (key material) for the DH exchange. The strength of any key derived from a DH exchange depends, in part, on the strength of the DH group on which the prime numbers are based.

Each DH group defines the length of the keying material to be used. Group 2 (medium) is stronger than Group 1 (low). Group 1 protects 768 bits of keying material and Group 2 protects 1024 bits. When a larger group is used, the resulting key that is determined from a DH exchange is larger and more difficult to determine by an attacker.

IKE negotiates which specific group to use, ensuring that there are not any negotiation failures that result from a mismatched DH group between the two peers.

If session key PFS is enabled, a new DH key is negotiated in the first quick mode SA negotiation. This new DH key removes the dependency of the session key on the DH exchange that is performed for the master key.

If the initiator is using session key PFS, the responder is not also required to use session key PFS. However, if the initiator is not using session key PFS and the responder is using session key PFS, negotiation fails.

The DH group is the same for both the main mode and quick mode SA negotiations. When session key PFS is enabled, even though the DH group is set as part of the main mode SA negotiation, it affects any rekeys during session key establishment.

Perfect forward secrecy

Unlike key lifetimes, perfect forward secrecy (PFS) determines how a new key is generated, rather than when it is generated. Specifically, PFS ensures that the compromise of a single key permits access only to data that is protected by it, not necessarily to the entire communication. To achieve this, PFS ensures that a key used to protect a transmission, in either mode, cannot be used to generate additional keys. In addition, if the key used was derived from specific keying material, that material cannot be used to generate other keys.

Master key PFS requires a reauthentication and is resource intensive. When it is enabled, IKE must reauthenticate identities, increasing overhead for domain controllers when Kerberos V5 is used for authentication. It requires a new main mode negotiation for every quick mode negotiation that occurs.

Session key PFS can be used without a reauthentication and is less resource intensive. Session key PFS results in a DH exchange to generate new keying material. It requires only four messages and no authentication.

PFS does not have to be enabled on both peers because it is not part of the SA negotiation. If the responder requires PFS and the sender's quick mode SA expires, it simply rejects the sender's message and requires a new negotiation. The sender expires the main mode SA and renegotiates. PFS can be individually set for both the master and session keys.


IPSec driver

The IPSec driver receives the active IP filter list from the IPSec Policy Agent, as shown in the following illustration, and then attempts to match every inbound and outbound packet against the filters in the list.

IPSec driver

When a packet matches a filter, it applies the filter action. When a packet does not match any filters, the packet is passed back without modification to the TCP/IP driver to be received or transmitted.

If the filter action permits transmission, the packet is received or sent with no modifications. If the action blocks transmission, the packet is discarded. If the action requires the negotiation of security, main mode and quick mode SAs are negotiated.

The negotiated quick mode security association (SA) and keys are used with both outbound and inbound processing. The IPSec driver stores all current quick mode SAs in a database. The IPSec driver uses the Security Parameters Index (SPI) field to match the correct SA with the correct packet.

When an outbound IP packet matches the IP filter list with an action to negotiate security, the IPSec driver queues the packet and then notifies Internet Key Exchange (IKE), which begins security negotiations with the destination IP address of that packet. If several outbound packets are going to the same destination and match the same filter before IKE has finished the negotiation, then only the last packet sent is saved.

After a successful negotiation is complete, the IPSec driver on the sending computer:

  1. Receives the SA containing the session key from IKE.
  2. Locates the outbound SA in its database, and inserts the SPI from the SA into the AH or ESP header.
  3. Signs the packets (and encrypts them if confidentiality is required).
  4. Sends the packets to the IP layer to be forwarded to the destination computer.

If the negotiation failed, the IPSec driver discards the packet.

When an IPSec-secured inbound packet matches the IP filter list, the IPSec driver on the receiving computer:

  1. Receives the session key, SA, and SPI from IKE.
  2. Locates the inbound SA in its database by destination address and SPI.
  3. Checks the signature and, if required, decrypts the packets.
  4. Matches the IP packet against the filter to ensure that no traffic, other than what was agreed upon during the negotiation, is received.
  5. Sends packets to the TCP/IP driver to pass to the receiving application.

When an unsecured IP packet is received, the IPSec driver matches the packet against all filters in the filter list. If a match occurs and the filter action for that filter either requires IP security or blocks the packet, then the packet is discarded.

The IPSec driver matches all inbound unsecured packets with the list of filters that specify IPSec tunnels first, and then matches the packet with all filters that specify end-to-end (transport) filters. The IPSec driver does not filter certain types of IP packets. For more information, see To add or edit IPSec filters.


The IPSec process

This topic describes how IPSec works in terms of the IPSec components for two intranet computers, as shown in the following illustration.

The IPSec process

For simplicity, this example is of an intranet in which a computer has an active IPSec policy.

  1. Alice, using a data application on Computer A, sends a message to Bob on Computer B.
  2. The IPSec driver on Computer A checks its stored IP filter lists to determine whether the packets should be secured.
  3. The IPSec driver notifies IKE to begin negotiations.
  4. The IKE service on Computer B receives a message requesting secure negotiation.
  5. The two computers establish a main mode SA and shared master key.

    6. Note

    • If Computer A and Computer B already have a main mode SA from a previous communication (and neither master key PFS is enabled nor have expired key lifetimes), the two computers can begin establishing the quick mode SA.
  6. A pair of quick mode SAs are negotiated. One SA is inbound and one SA is outbound. The SAs include the SPI and the keys that are used to secure the information.
  7. The IPSec driver on Computer A uses the outbound SA to sign and, if required, encrypt the packets.
  8. The driver passes the packets to the IP layer, which forwards the packets to Computer B.
  9. The network adapter driver at Computer B receives the encrypted packets and passes them to the IPSec driver.
  10. The IPSec driver on Computer B uses the inbound SA to validate authentication and integrity and, if required, decrypt the packets.
  11. The driver passes the validated and decrypted packets to the TCP/IP driver, which passes them to the receiving application on Computer B.

Any routers or switches in the path between the communicating computers simply forward the encrypted IP packets to their destination. However, if there is a firewall, security router, or proxy server in the path, IPSec and IKE traffic might not be forwarded.

Security negotiations are not able to pass through a network address translator (NAT). IKE negotiation messages contain IP addresses within the encrypted or signed portion of the message. These addresses cannot be changed by a NAT because the NAT does not have the shared, secret key to either change the encrypted address within the message or change the unencrypted address, without invalidating the integrity check value (ICV).


Virtual private networking with IPSec

Tunneling is the entire process of encapsulation, routing, and decapsulation. Tunneling wraps, or encapsulates, the original packet inside a new packet. This new packet might have new addressing and routing information, which enables it to travel through a network. When tunneling is combined with data confidentiality, the original packet data (as well as the original source and destination) is not revealed to those listening to traffic on the network. The network could be any internetwork: a private intranet or the Internet. After the encapsulated packets reach their destination, the encapsulation is removed and the original packet header is used to route the packet to its final destination.

The tunnel itself is the logical data path through which the encapsulated packets travel. To the original source and destination peer, the tunnel is usually transparent and appears as just another point-to-point connection in the network path. The peers are unaware of any routers, switches, proxy servers, or other security gateways between the tunnel’s beginning point and the tunnel’s end point. When tunneling is combined with data confidentiality, it can be used to provide a virtual private network (VPN).

In Windows XP, two types of tunneling are provided that use IPSec:

  1. The combination of the Layer Two Tunneling Protocol (L2TP) and IPSec (L2TP/IPSec). L2TP is used to tunnel the data across a shared or public network such as the Internet and IPSec Encapsulating Security Payload (ESP) is used to encrypt the data. L2TP/IPSec can be used to tunnel IP or Internetwork Packet Exchange (IPX) traffic.
  2. IPSec in tunnel mode, in which IPSec itself provides encapsulation for IP traffic only.

The encapsulated packets travel through the network inside the tunnel. In this example, the network is the Internet. The gateway might be an edge gateway that stands between the outside Internet and the private network. The edge gateway can be a router, firewall, proxy server, or other security gateway. Also, two gateways can be used inside the private network to protect traffic across untrusted parts of the network.

L2TP and IPSec

L2TP and IPSec are combined to provide both tunneling and security for IP, IPX, and other protocol packets across any IP network. IPSec can also perform tunneling without L2TP, but it is only recommended for interoperability, when one of the gateways does not support L2TP or PPTP.

L2TP encapsulates original packets first inside a PPP frame (performing compression when possible) and then inside a UDP message using port 1701. Because the UDP message is an IP payload, L2TP uses IPSec to secure the tunnel. The IPSec Internet Key Exchange (IKE) protocol negotiates security for the L2TP tunnel using certificate-based or preshared key authentication. If IPSec main mode and quick mode security associations are successfully established, L2TP negotiates the tunnel, including compression and user authentication options, and performs PPP-based user authentication.

Configuration for L2TP/IPSec VPN remote access clients is performed using Network Connections. Configuration for the VPN remote access server and for gateway-to-gateway tunnels is performed using Routing and Remote Access.

Tunnel packet using L2TP and IPSec

The original packet, shown here as the PPP payload, includes the original and ultimate source and destination addresses used on the private network. The outer IP header, shown as IP header, contains the source and destination addresses of the VPN client and server on the public network. The L2TP header includes L2TP tunnel control information. The PPP header identifies the protocol of the original packet (for example, IP or IPX).

For more information about using L2TP and IPSec, see Remote communications and Special IPSec considerations.

Note

  • XOX

IPSec tunneling

The primary reason for using IPSec tunnel mode is interoperability with other routers, gateways, or end-systems that do not support L2TP/IPSec or PPTP VPN tunneling. IPSec tunnel mode is supported only in gateway-to-gateway tunneling scenarios and for certain server-to-server or server-to-gateway configurations as an advanced feature. IPSec tunnel mode is not supported for remote access VPN scenarios. L2TP/IPSec or PPTP should be used for remote access VPN connections.

For information about IPSec tunnel mode and configuring IPSec tunnels, see Tunnel mode.


Description of an IPSec policy

An IPSec policy consists of:

  • General IPSec policy settings

    • Settings that apply regardless of which rules are configured. These settings determine the name of the policy, its description for administrative purposes, key exchange settings, and key exchange methods.

  • Rules

    • One or more IPSec rules that determine which types of traffic IPSec must examine, how traffic is treated, how to authenticate an IPSec peer, and other settings.

After the policies are created, they can be applied at the domain, site, organizational unit, and local level. For more information, see Applying IPSec policies.


General IPSec policy settings

The general settings for an IPSec policy consist of the following types:

  • Name

    • The name for the policy.

  • Description

    • Optional text that describes the purpose of the IPSec policy. It is recommended that you complete and update this description to provide a summary of the settings and rules for the policy.

  • Policy change poll interval

    • The number of minutes between consecutive polls for changes in Active Directory-based IPSec policies. This polling does not detect a change in domain or organizational unit membership, or the assigning or unassigning of a new policy. These events are detected when the Winlogon service polls for changes in Group Policy, which occurs by default every 90 minutes.

  • Key exchange settings

    • The way in which new keys are derived and how often they are renewed.

  • Key exchange methods

    • The ways in which identities are protected during the key exchange.

The default key exchange settings and methods are configured to work for most IPSec deployments. Unless there are special security requirements, default settings should not have to be changed.


Key exchange settings

The strength of the keys which protect the key exchange phase is enhanced by the following features:

Key lifetimes

Lifetime settings determine when a new key is generated. Any time a key lifetime is reached, the associated SA is also renegotiated. The process of generating new keys at intervals is called dynamic rekeying or key regeneration. Lifetimes allow you to force the generation of a new key (regeneration) after a specific interval. For example, if the communication takes 100 minutes and you specify the key lifetime as 10 minutes, 10 keys will be generated (one every 10 minutes) during the exchange. Using multiple keys ensures that if an attacker manages to gain the key to one part of a communication, the entire communication is not compromised. Automatic key regeneration is provided by default. You can override the defaults and either specify a master key lifetime in minutes or by the number of session keys, or enable master key perfect forward secrecy (PFS).

Caution should be taken when setting very different key lifetimes, since they also determine the lifetime of the SA. For example, setting a master key lifetime of 8 hours (480 minutes) and a session key lifetime (set within a filter action) of 2 hours might result in a quick mode SA that is in place for almost 2 hours after the main mode SA has expired. This has the potential to occur if the new quick mode SA is generated just before the main mode SA expires.

  • Session key limit

    • Repeated rekeying off of the same master key might eventually compromise the key. For example, if Alice on Computer A sends a message to Bob on Computer B, and then a few minutes later sends another message to Bob, the same master key material can be reused since an SA was recently established with that computer. If you want to limit the number of times this reuse occurs, you can specify a session key limit.

    If you decide to enable master key perfect forward secrecy (PFS), the session key limit will be set to 1. Master key PFS forces key regeneration each time. If you specify both a master key lifetime in minutes, and a session key limit, whichever interval is reached first will start a new key.

  • Master key perfect forward secrecy (PFS)

    • Master key PFS determines how a new session key is generated. Enabling master key PFS ensures that the master key keying material cannot be used to generate more than one session key.

    Master key PFS should be used with caution because it requires reauthentication and might impact performance. It is not required to be enabled on both peers.

For information about how to configure key exchange settings and methods, see To configure key exchange and To create key exchange security methods.


Key exchange methods

In addition to key properties, you can set security methods for main mode IKE negotiation. It is generally recommended that all of the IKE settings (for example, master key PFS and key lifetime) and security methods remain at their defaults to avoid unnecessary administrative overhead. This provides a standard (medium) level of security. If your security plan calls for a high level of security, you should consider modifying the default security methods.

You can specify which algorithms are used for integrity and confidentiality. The same algorithms that are available for quick mode security methods are available here: MD5 and SHA1 for integrity; DES and 3DES for confidentiality.

Diffie-Hellman groups

Diffie-Hellman (DH) groups are used to determine the length of the base prime numbers used during key exchange. The strength of any key derived depends, in part, on the strength of the Diffie-Hellman group upon which the prime numbers are based.

Group 2 (medium) is stronger than Group 1 (low). Group 1 provides 768 bits of keying material, while Group 2 provides 1024 bits. If mismatched groups are specified on each peer, negotiation fails. The group cannot be switched during the negotiation.

A larger group uses more keying material. Therefore, a key derived from a Diffie-Hellman exchange from a larger group is more secure.

The Diffie-Hellman group is configured as part of the main mode key exchange settings. New session keys generated during quick mode are derived from the Diffie-Hellman main mode master key material, unless master key or session key perfect forward secrecy (PFS) is enabled. If either master key or session key PFS is enabled, a new DH exchange is performed to obtain new master key keying material for each new session key that is required. The difference between master key PFS and session key PFS is that master key PFS requires a reauthentication of the main mode SA in addition to the DH exchange.

For information about how to configure key exchange settings and methods, see To configure key exchange and To create key exchange security methods.


Rules

An IPSec policy consists of one or more rules that determine IPSec behavior. IPSec rules are configured on the Rules tab in the properties of an IPSec policy. Each IPSec rule contains the following configuration items:

  • Filter list

    • A single filter list is selected that contains one or more predefined packet filters that describe the types of traffic to which the configured filter action for this rule is applied. The filter list is configured on the IP Filter List tab in the properties of an IPSec rule within an IPSec policy.

  • Filter action

    • A single filter action is selected that includes the type of action required (permit, block, or secure) for packets that match the filter list. For the secure filter action, the negotiation data contains one or more security methods that are used (in order of preference) during IKE negotiations and other IPSec settings. Each security method determines the security protocol (such as AH or ESP), the specific cryptographic algorithms, and session key regeneration settings used. The negotiation data is configured on the Filter Action tab in the properties of an IPSec rule within an IPSec policy.

  • Authentication methods

    • One or more authentication methods are configured (in order of preference) and used for authentication of IPSec peers during main mode negotiations. The available authentication methods are the Kerberos V5 protocol, use of a certificate issued from a specified certification authority, or a preshared key. The negotiation data is configured on the Authentication Methods tab in the properties of an IPSec rule within an IPSec policy.

  • Tunnel endpoint

    • Specifies whether the traffic is tunneled and, if it is, the IP address of the tunnel endpoint. For outbound traffic, the tunnel endpoint is the IP address of the IPSec tunnel peer. For inbound traffic, the tunnel endpoint is a local IP address. The tunnel endpoint is configured on the Tunnel Setting tab in the properties of an IPSec rule within an IPSec policy. For more information, see Tunnel mode.

  • Connection type

    • Specifies whether the rule applies to local area network (LAN) connections, dial-up connections, or both. The connection type is configured on the Connection Type tab in the properties of an IPSec rule within an IPSec policy.

The rules for a policy are displayed in IP Security Policies in reverse alphabetical order based on the name of the filter list selected for each rule. There is no method for specifying an order in which to apply the rules in a policy. The IPSec driver automatically orders the rules based on the most specific to the least specific filter list. For example, the IPSec driver would apply a rule containing a filter list that specified individual IP addresses and TCP ports before a rule containing a filter list that specified all addresses on a subnet.

Default response rule

The default response rule, which can be used for all policies, has the IP filter list of <Dynamic> and the filter action of Default Response when the list of rules is viewed with the IP Security Policies snap-in. The default response rule cannot be deleted, but it can be deactivated. It is activated for all of the default policies and you have the option of enabling it when you create new IPSec policies with the IP Security Policy Wizard.

The default response rule is used to ensure that the computer responds to requests for secure communication. If an active policy does not have a rule defined for a computer that is requesting secure communication, then the default response rule is applied and security is negotiated. For example, when Computer A communicates securely with Computer B and Computer B does not have an inbound filter defined for Computer A, the default response rule is used.

Authentication methods and the connection type can be configured for the default response rule. The filter list of <Dynamic> indicates that the filter list is not configured, but that filters are created automatically based on the receipt of IKE negotiation packets. The filter action of Default Response indicates that the action of the filter (permit, block, or negotiate security) cannot be configured. However, you can configure:

  • The security methods and their preference order on the Security Methods tab in the properties of the default response rule in the IP Security Policies snap-in.
  • The authentication methods and their preference order on the Authentication Methods tab in the properties of the default response rule in the IP Security Policies snap-in. The default authentication method for the default response rule for new IPSec policies is configured with the IP Security Policy Wizard.


Filter list

An IP filter list triggers security negotiations based on a match with the source, destination, and type of IP traffic. This type of IP packet filtering enables a network administrator to precisely define what IP traffic is secured. Each IP filter list contains one or multiple filters, which define IP addresses and traffic types. One IP filter list can be used for multiple communication scenarios.

IPSec requires both an inbound and outbound filter between the computers specified in the filter list. Inbound filters apply to incoming traffic, enabling the receiving computer to respond to requests for secure communication; or match traffic against the IP filter list. Outbound filters apply to traffic leaving a computer towards a destination, triggering a security negotiation that takes place before traffic is sent. For example, if Computer A wants to securely exchange data with Computer B:

  • The active IPSec policy on Computer A must have a filter for any outbound packets to Computer B. Source=A and Destination=B.
  • The active IPSec policy on Computer A must have a filter for any inbound packets from Computer B. Source=B and Destination=A.

Each peer must also have the reverse filter:

  • The active IPSec policy on Computer B must have a filter for any inbound packets from Computer A. Source=A and Destination=B.
  • The active IPSec policy on Computer B must have a filter for any outbound packets to Computer A. Source=B and Destination=A.

Filter settings

Each filter defines a particular subset of inbound or outbound network traffic that should be secured. You must have a filter to cover any traffic to which the associated rule applies. A filter contains the following settings:

  • The source and destination address of the IP packet. You can configure any IP address assigned to the IPSec peer, a single IP address, IP addresses by DNS name, or groups of addresses to specify IP subnets.
  • The protocol over which the packet is being transferred. This automatically covers all protocols in the TCP/IP protocol suite. However, it can be configured for an individual protocol to meet special requirements, including custom protocols.
  • The source and destination port of the protocol for TCP and UDP. By default, all TCP and UDP ports are covered, but this can be configured to apply to only a specific TCP or UDP port.

For information about how to configure IP filter lists, see To add or edit IP filter lists.

Special filters might be required for certain types of traffic, such as SNMP traffic or security gateways. For more information, see Special IPSec considerations.


Filter action

A filter action defines the security requirements for the data transmission. A filter action can be configured to:

  • Permit traffic (Permit)

    • IPSec passes this traffic to and from the TCP/IP driver without modification or the requirement for security. This is appropriate for traffic from specific computers that are not IPSec-capable. Be sure to limit the IP filter list to a minimal scope when using this type of filter action, so you do not let traffic through which should be secured.

  • Block traffic (Block)

    • IPSec silently discards this traffic. Be sure to use an IP filter list that defines the smallest scope of traffic possible when using a blocking filter action so that you do not block valid computers from communicating.

  • Negotiate IPSec (Negotiate security)

    • IPSec requires the negotiation of security associations and the sending or receiving of IPSec-protected traffic. After you choose to negotiate IPSec, you can also configure the following:

    • Security methods and their order

      • For more information, see IPSec security methods.

    • Allowance of initial incoming unsecured traffic (Accept unsecured communication, but always respond using IPSec)

      • IPSec allows an incoming packet that matches the configured filter list to be unsecured (not protected by IPSec). However, the outgoing response to the incoming packet must be protected. This setting is useful when you are using the default response rule for clients. When a group of servers are configured with a rule that secures communications with any IP address and accepts unsecured communication, responding with only secured communications, enabling the default response rule on client computers ensures that the clients will respond to the server request to negotiate security. This option should be disabled for secure computers connected to the Internet to prevent denial of service attacks.

    • Enabling of communication with non-IPSec-enabled computers (Allow unsecured communication with non-IPSec-aware computer)

      • IPSec falls back to unsecured communication, if necessary. Again, you might need to limit the IP filter list to a minimal scope. Otherwise, if negotiation fails for any reason, any communications affected by the rule in which this filter action resides could result in data being sent without security. If you are concerned about unsecured communication, you might consider disabling these settings. However, communication with computers that cannot initiate IPSec, such as legacy systems, might be blocked. This option should be disabled for secure computers connected to the Internet to prevent denial of service attacks.

    • Generation of session keys from new keying material (Session key perfect forward secrecy (PFS))

      • This determines whether existing master key keying material can be used to derive a new session key. Enabling session key perfect forward secrecy (PFS) ensures that master key keying material cannot be used to derive more than one session key. When session key PFS is enabled, a new Diffie-Hellman key exchange is performed to generate new master key keying material before the new session key is created. Session key PFS does not require main mode reauthentication and uses less resources than master key PFS.

For information about how to configure filter actions, see To add or edit filter actions.

IPSec security methods

Each security method defines the security requirements of any communications to which the associated rule applies. Creating multiple security methods increases the chance that a common method can be found between two computers. The IKE component reads the list of security methods in descending order and sends a list of allowed security methods to the other peer. The first method in common is selected. Typically, the most secure methods are placed at the top of the list and the least secure methods are placed at the bottom of the list.

Predefined security methods

The following security methods are predefined:

  • Encryption and Integrity

    • Uses the ESP protocol to provide data confidentiality (encryption) with the triple Data Encryption Standard (3DES) algorithm, data integrity and authentication with the Secure Hash Algorithm 1 (SHA1) integrity algorithm, and default key lifetimes (100MB, 1 hour). If you require both data and addressing (IP header) protection, you can create a custom security method. If you do not require encryption, use Integrity only.

  • Integrity only

    • Uses the ESP protocol to provide data integrity and authentication with the SHA1 integrity algorithm and default key lifetimes (100MB, 1 hour). In this configuration, ESP does not provide data confidentiality (encryption). This is appropriate when your security plan calls for standard levels of security.

Custom security methods

If the predefined Encryption and Integrity or Integrity only settings do not meet your security requirements, you can specify custom security methods. For example, you can use custom methods when encryption and address integrity, stronger algorithms, or key lifetimes need to be specified. When configuring a custom security method, you can configure the following:

  • Security protocols

    • Both AH (Data and address integrity without encryption) and ESP (Data integrity and encryption) can be enabled in a custom security method when you require IP header integrity and data encryption. If you chose to enable both, you do not need to specify an integrity algorithm for ESP. The algorithm that you select for AH provides integrity.

  • Integrity algorithm
    • Message Digest 5 (MD5), which uses a 128-bit key.
    • Secure Hash Algorithm 1 (SHA1), which uses a 160-bit key. SHA1 is a stronger hash than MD5 and is Federal Information Processing Standard (FIPS) compliant.
  • Encryption algorithm
    • 3DES is the most secure of the DES combinations, and somewhat slower in performance. 3DES processes each block three times, using three unique 56-bit keys.
    • DES is used when the higher security and overhead of 3DES is not necessary. DES uses a single 56-bit key.
  • Session key settings

    • Session key settings determine when a new key is generated, rather than how it is generated. You can specify a lifetime in kilobytes, seconds, or both. For example, if the communication takes 10,000 seconds and you specify the key lifetime as 1000 seconds, 10 keys will be generated to complete the transfer. This ensures that, even if an attacker manages to determine one session key and decipher part of a communication, deciphering the entire communication is not possible. By default, new session keys are generated for every 100 MB of data or every hour. Note that any time a key lifetime is reached, the SA is also renegotiated in addition to the key refresh or regeneration.

Interoperability of encryption settings

When IPSec provides data confidentiality between a Windows XP computer (or a Windows 2000-based computer that has the High Encryption Pack or Service Pack 2 or higher installed) and another computer that does not use 3DES, your list of security methods must include DES. Otherwise, the computer that you are trying to communicate with might not be able to reach a common security agreement with your computer. If confidentiality of the data is not required, you can select ESP format with an integrity algorithm only and set the encryption to <None>. Alternately, you can select an AH format with an integrity algorithm.

For information about how to configure security methods, see To add or edit IPSec security methods.


Authentication methods

Each rule defines a list of authentication methods. Each authentication method defines the requirements for how identities are verified in communications to which the associated rule applies. The two peers must have at least one common authentication method or communication will fail. Creating multiple authentication methods increases the chance that a common method between two computers can be found.

Only one authentication method can be used between a pair of computers, regardless of how many are configured. If you have multiple rules that apply to the same pair of computers, you must configure the authentication methods list in those rules to enable the pair to use the same method. For example, if a rule between a pair of computers specifies only Kerberos for authentication and filters only TCP data and, in another rule, specifies only certificates for authentication and filters only UDP data, authentication will fail.

  • The Kerberos V5 security protocol is the default authentication technology. This method can be used for any clients running the Kerberos V5 protocol (whether or not they are Windows XP Professional or Windows 2000-based clients) that are members of the same or trusted domains.
  • A public key certificate should be used in situations that include Internet access, remote access to corporate resources, external business partner communications, or computers that do not run the Kerberos V5 security protocol. This requires that at least one trusted certification authority (CA) has been configured. Windows XP and Windows 2000 support X.509 Version 3 certificates, including CA certificates generated by commercial certifying authorities.
  • A preshared key can be specified. This is a shared, secret key which is previously agreed upon by two users. It is simple to use and does not require the client to run the Kerberos V5 protocol or have a public key certificate. Both parties must manually configure IPSec to use this preshared key. This is a simple method for authenticating computers that are not running Windows XP or Windows 2000, stand-alone computers, or any computers that are not using the Kerberos V5 protocol. Note that this key is for authentication protection only.

For information about how to configure authentication methods, see To define IPSec authentication methods.

Note

  • The ability to authenticate using Kerberos V5 is not supported on computers running Windows XP Home Edition or computers running any other version of Windows 2000 or Windows XP that are not a member of an Active Directory domain.


Tunnel endpoint

As discussed in Virtual private networking with IPSec, IPSec can perform layer 3 tunneling for scenarios in which Layer Two Tunneling Protocol (L2TP) cannot be used. If you are using L2TP for remote communications, no tunnel configuration is needed because the client and server VPN components of Windows XP automatically create the appropriate rules to secure L2TP traffic.

To create a layer 3 tunnel using IPSec, use the IP Security Policies or Group Policy snap-ins to configure and enable the following two rules for the appropriate policy:

  1. A rule for the outbound traffic for the tunnel.

    2. The rule for the outbound traffic is configured with both a filter list that describes the traffic to be sent across the tunnel and a tunnel endpoint of an IP address configured on the IPSec tunnel peer (the computer or router on the other side of the tunnel).

  2. A rule for the inbound traffic for the tunnel.

    3. The rule for the inbound traffic is configured with both a filter list that describes the traffic to be received across the tunnel and a tunnel endpoint of a local IP address (the computer or router on this side of the tunnel).

For each rule, you must also specify filter actions, authentication methods, and other settings.

For information about how to configure IPSec tunnel endpoints, see To specify an IPSec tunnel. For more information about IPSec tunnel mode, see Tunnel mode.

For advanced information about configuring IPSec tunnels, see article Q252735, "How to Configure IPSec Tunneling in Windows 2000," in the Microsoft Knowledge Base.(http://search.support.microsoft.com/kb/)


Connection type

For each IPSec rule, you must define which connection types on your computer are affected. The connection types apply to all connections in Network Connections on the computer for which you are configuring IPSec policy.

Each rule has one connection type setting:

  • All Network Connections. The rule applies to communications sent through any of the network connections that you have configured on the computer.
  • Local Area Network (LAN). The rule only applies to communications sent through LAN connections that you have configured on the computer.
  • Remote Access. The rule only applies to communications sent through any remote access or dial-up connections that you have configured on the computer.

For information about how to configure connection types, see To specify IPSec connection types.

For information about configuring network connections on a computer, see Configuring a network or dial-up connection.


Applying IPSec policies

IPSec policies can be applied to local computers, domains, sites, organizational units, or any Group Policy object in Active Directory. Your organization's IPSec policies should be based on your organization's written guidelines for secure operations. Policies can store multiple rules, so that one policy can be used to affect multiple types of traffic.

There are two storage locations for IPSec policies:

  1. Active Directory

    2. If the computer is temporarily unconnected to a trusted Windows 2000 domain, the policy information is cached in the local registry. For more information, see Active Directory overview.

  2. Local

    3. Local IPSec policies are defined in the registry for stand-alone computers and computers that are not always part of a trusted Windows 2000 domain.

Windows XP includes default policies that can be used as examples for your own custom policies. Each policy that you define should apply to a scenario in your security plan.

Group Policy

IPSec policies, which are assigned to a Group Policy object in Active Directory, become part of Group Policy, and are transferred to the member computers each time Group Policy propagates. For more information, see Group Policy overview.

When assigning an IPSec policy in Active Directory, consider the following:

  • IPSec policies that are assigned at the domain level override any active, local IPSec policy when that computer is joined to the domain.
  • IPSec policies that are assigned to an organizational unit override an IPSec policy assigned at the domain level, for any member computers of that organizational unit. The IPSec policy assigned to the lowest-level organizational unit in the domain hierarchy overrides an IPSec policy assigned to a higher-level organizational unit, for any member computers of that organizational unit. For more information, see Policies stored in Active Directory.

For information about how to define IPSec policies, see To add or edit IPSec policies.

Notes

  • The integration of IPSec policies with Active Directory described here does not apply to computers running Windows XP Home Edition.
  • You cannot administer Active Directory-based IPSec policy from a computer running Windows XP Home Edition.


Deploying Internet Protocol security

This section covers:

  • IPSec policy design and planning
  • IPSec security considerations
  • IPSec deployment


IPSec policy design and planning

This section provides information about planning and deploying IPSec in your enterprise. You should also review Best practices for general information about IPSec deployment.

For information on specific topics, see:

  • Establishing an IPSec security plan
  • Default IPSec policies
  • Assigning IPSec policy
  • Special IPSec considerations


Establishing an IPSec security plan

Whether for a large domain or a small workgroup, implementing IPSec means finding a balance between making information easily available to the largest number of users, and protecting sensitive information from unauthorized access.

Finding the proper balance requires:

  • Assessing the risk and determining the appropriate level of security for your organization.
  • Identifying valuable information.
  • Defining security policies that use your risk management criteria and protect the identified information.
  • Determining how the policies can best be implemented within the existing organization.
  • Ensuring that management and technology requirements are in place.
  • Providing all users with both secure and efficient access to the appropriate resources, according to their needs.

Security considerations are also influenced by the way the computer will be used. For example, the required security might differ, depending on whether the computer is a domain controller, Web server, remote access server, file server, database server, or intranet or remote client. The Windows 2000 and Windows XP security framework is designed to fulfill the most stringent security requirements. However, software alone might be less effective without careful planning and assessment, effective security guidelines, enforcement, auditing, and sensible security policy design and assignment.

There is no exact definition of the measures that define standard security. These can vary widely, depending on an organization's policies and infrastructures. The following security levels can be considered a general basis for planning your IPSec deployment:

  • Minimal security

    • Computers do not exchange sensitive data. IPSec is not active by default. No administrative action to disable IPSec is required.

  • Standard security

    • Computers, especially file servers, are used to store valuable data. Security must be balanced so it does not become a barrier to users trying to perform their tasks. Windows XP provides default IPSec policies that secure data, but do not necessarily require the highest level of security: Client (Respond Only) and Server (Request Security). These, or similar custom policies, optimize efficiency without compromising security.

  • High security

    • Computers that contain highly sensitive data are at risk for data theft, accidental or malicious disruption of the system (especially in remote dial-up scenarios), or any public network communications. Secure Server (Require Security), a default policy, requires IPSec protection for all traffic being sent or received (except initial inbound communication). The Secure Server (Require Security) policy includes stronger security methods. Unsecured communication with a non-IPSec-aware computer is not allowed.

From planning to policy configuration

After you have identified the types of traffic that need to be secured and the level of security required, you can begin policy configuration. Windows XP IPSec policy configuration is the translation of your security requirements to one or more IPSec policies, only one of which can be assigned at the domain, site, organizational unit, or local level. Each IPSec policy consists of one or more IPSec rules. Each IPSec rule consists of:

  • A selected filter list.
  • A selected filter action.
  • Selected authentication methods.
  • A selected connection type.
  • A selected tunnel setting.

You can configure IPSec policy in the following ways:

  • Create a new policy and define the set of rules for the policy, adding filter lists and filter actions as required.

    • In this method, an IPSec policy is created first and then rules are added and configured. Filter lists (specifying traffic types) and filter actions (specifying how the traffic is treated) are added during rule creation.

    For more information, see To add or edit IPSec policies and To add or edit rules.

  • Create the set of filter lists and filter actions, and then create the policies and add rules that combine the filter lists with filter actions.

    • In this method, the filter lists and the filter actions are configured first. Next, IPSec policies are created and rules are added that combine the appropriate filter list with the appropriate filter action. Additionally, authentication methods, connection types, and tunnel settings are specified.

    For information about how to individually configure IP filter lists and filter actions, see To manage multiple IP filter lists and To manage multiple filter actions.

For either configuration method, after the IPSec policies have been created, they must be assigned. For more information, see Assigning IPSec policy.

Notes

  • To define Active Directory-based IPSec policy, you must have Group Policy administrative permissions. To manage local or remote IPSec policy for a computer, you must be a member of the Administrators group on the local or remote computer.
  • The integration of IPSec policies with Active Directory described here does not apply to computers running Windows XP Home Edition.
  • You cannot administer Active Directory-based IPSec policy from a computer running Windows XP Home Edition.


Default IPSec policies

Windows XP provides a set of predefined IPSec filter lists and filter actions and default policies. This is only intended to provide an example. It is not designed for operational use without modification. A knowledgeable IPSec network administrator or advanced user should design new, custom polices for operational use.

Computers that are connected to the Internet must be carefully protected from network attacks with custom policies. The default policies are defined as examples for intranets because they allow a computer to receive unsecured traffic. The default policies are designed for computers that are members of an Active Directory domain.

Predefined filter lists

The following predefined filter lists are provided as examples for use in the default policies:

  • All ICMP Traffic

    • A filter list for all ICMP traffic (IP protocol 1) sent and received between this computer and all other computers.

  • All IP Traffic

    • A filter list for all IP traffic sent and received between this computer and all other computers. All broadcast, multicast, and Internet Key Exchange (IKE) traffic is excluded. By default, all Kerberos and Resource Reservation Protocol (RSVP) traffic is also excluded. However, you can include Kerberos and RSVP traffic by editing the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPSEC. Add a new value named NoDefaultExempt and assign to it a value of 1. For more information about adding values to registry keys, see To add a value.

    Caution

    • Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

Predefined filter actions

The following predefined filter actions are provided as examples for use in the default policies:

  • Permit

    • Traffic is permitted (the security method is set to Permit).

  • Request Security (Optional)

    • Traffic is secured (the security method is set to Negotiate security), however, the following exceptions apply:

    • Initial incoming unsecured communication is allowed. This behavior is known as inbound passthrough.
    • If security negotiation fails after 3 seconds, unsecured communication to a non-IPSec-aware computer is allowed. This filter action secures traffic to IPSec-aware computers, but allows unsecured communication if it is required. This behavior is known as fallback to clear.

    In order to avoid the potential for denial-of-service attacks, this filter action must not be used on the Internet.

    The following table lists the security methods (in preference order) for the Request Security (Optional) predefined filter action.

    Type | AH integrity | ESP confidentiality | ESP integrity | Key lifetimes (KB/sec)
Custom | None | 3DES | SHA1 | 100000/900
Custom | None | DES | SHA1 | 100000/900
Custom | SHA1 | None | None | 100000/300
Custom | MD5 | None | None | 100000/300

    The security negotiation for traffic using this filter action attempts ESP encryption (first 3DES, and then DES) with SHA1 integrity and then AH integrity (first SHA1, and then MD5).

  • Require Security

    • Traffic is secured (the security method is set to Negotiate security), however, initial incoming unsecured communication is allowed (inbound passthrough is enabled). Unsecured communication with non-IPSec-aware computers is not allowed (fallback to clear is disabled). For example, an incoming connection request packet is allowed. The computer's reply to the connection request matches the outbound filter, causing a request for security to be sent to the sender of the unsecured traffic. If security negotiation is successful, traffic in both directions is required to be secure. If security negotiation is not successful, outgoing traffic is not allowed. The computer might continue to receive incoming unsecured traffic and will continue to attempt to negotiate security.

    In order to avoid the potential for denial-of-service attacks, this filter action must not be used on the Internet.

    The following table lists the security methods (in preference order) for the Require Security predefined filter action.

    Type | AH integrity | ESP confidentiality | ESP integrity | Key lifetimes (KB/sec)
Custom | None | 3DES | SHA1 | 100000/900
Custom | None | 3DES | MD5 | 100000/900
Custom | None | DES | SHA1 | 100000/900
Custom | None | DES | MD5 | 100000/900

    The security negotiation for traffic using this filter action will only attempt to negotiate ESP encryption (first 3DES, and then DES) with differing ESP authentication methods (first SHA1, and then MD5).

The most secure filter action is one that negotiates security and neither receives unsecured traffic (inbound passthrough is disabled) nor allows communication with non-IPSec-aware computers (fallback to clear is disabled). This is known as a lockdown filter action. When a server is using this filter action, clients must initiate secured communication before sending any IP traffic. The IPSec policy for client computers must be configured with additional rules that force the client to negotiate security with the server. A client configured with policy that consists of only the default response rule is not able to communicate with a lockdown server.

Predefined rules

The only predefined rule that is available on all policies is the default response rule. The default response rule is automatically created in a new policy and cannot be created manually. It is used to respond to security requests when no other rule applies. For example, a client policy that enables a client to respond to security requests but does not secure any traffic by default, would have only the default response rule active. In some cases, it is enabled by default. You can disable the default response rule on the Rules tab in properties for a policy. You can also modify the authentication method and security methods of the default response rule for specific needs. For more information, see Default response rule.

Default policies

The following are provided as examples for use in the default policies:

  • Client (Respond Only)

    • This is an example policy for computers that secure communication on request. For example, intranet clients might not require IPSec except when requested by another computer. This policy enables the computer on which it is active to respond to requests for secured communications. This policy contains the default response rule, which creates dynamic IPSec filters for inbound and outbound traffic based on the requested protocol and port traffic for the communication that is being secured.

    This policy has the following settings:

    • First rule (default response rule)
      • IP Filter List: <Dynamic>
      • Filter Action: Default Response
      • Authentication: Kerberos
      • Tunnel Setting: None
      • Connection Type: All
  • Server (Request Security)

    • This is an example policy for computers that should secure communications in most cases, allowing unsecured communication with non-IPSec-aware computers. With this policy, the computer accepts unsecured traffic, but always attempts to secure additional communications by requesting security from the original sender. This policy allows the entire communication to be unsecured if the other computer is not IPSec-enabled. For example, communication to specific servers can be secure while allowing the server to communicate in an unsecured manner to accommodate a mixture of clients (some of which support IPSec and some of which do not).

    To test the use of this policy, assign this policy to a server computer and assign the Client (Respond Only) policy to a client computer. When the client computer attempts to communicate with the server, the server requests secure communications. In addition, attempt to communicate with the server using a computer that either is not IPSec-capable or that has any IPSec policy assigned. When the non-IPSec-capable client attempts to communicate with the server, the server requests secure communications. However, the negotiation fails and the non-IPSec-capable computer connects to the server using unsecured communications.

    This policy has a rule to request security for all IP traffic, a rule to permit ICMP traffic, and the default response rule to respond to requests for security from other computers.

    This policy has the following settings:

    • First rule
      • IP Filter List: All IP Traffic
      • Filter Action: Request Security (Optional)
      • Authentication: Kerberos
      • Tunnel Setting: None
      • Connection Type: All
    • Second rule
      • IP Filter List: All ICMP Traffic
      • Filter Action: Permit
      • Authentication: Kerberos
      • Tunnel Setting: None
      • Connection Type: All
    • Third rule (default response rule)
      • IP Filter List: <Dynamic>
      • Filter Action: Default Response
      • Authentication: Kerberos
      • Tunnel Setting: None
      • Connection Type: All
  • Secure Server (Require Security)

    • This is an example policy for computers on an intranet that require secure communications, such as a server that transmits highly sensitive data. Administrators can use this IPSec policy as an example to create their own custom IPSec policy for production use. The filters used in this policy require all outbound communication to be secured, allowing only the initial inbound communication request to be unsecured.

    To test the use of this policy, assign this policy to a server computer and assign the Client (Respond Only) policy to a client computer. When the client computer attempts to communicate with the server, the server requests secure communications. In addition, attempt to communicate with the server using a computer that either is not IPSec-capable or that has any IPSec policy assigned. When the non-IPSec-capable client attempts to communicate with the server, the server requests secure communications. However, the negotiation fails and the non-IPSec-capable computer does not connect to the server.

    This policy has a rule to require security for all IP traffic, a rule to permit ICMP traffic, and the default response rule to respond to requests for security from other computers.

    • First rule
      • IP Filter List: All IP Traffic
      • Filter Action: Require Security
      • Authentication: Kerberos
      • Tunnel Setting: None
      • Connection Type: All
    • Second rule
      • IP Filter List: All ICMP Traffic
      • Filter Action: Permit
      • Authentication: Kerberos
      • Tunnel Setting: None
      • Connection Type: All
    • Third rule (default response rule)
      • IP Filter List: <Dynamic>
      • Filter Action: Default Response
      • Authentication: Kerberos
      • Tunnel Setting: None
      • Connection Type: All


Assigning IPSec policy

IPSec policy can be created and modified using the IP Security Policies snap-in that is available in Microsoft Management Console (MMC). It can manage policy centrally (for Active Directory clients), locally (on the computer on which you are running the snap-in), or remotely for a computer or domain. The customized console can then be saved so that it is available to you again at any time. For more information, see To start the IP Security Policies snap-in.

Note

  • Although you can assign IPSec policy for a computer, you cannot assign IPSec policy for Active Directory domains. To assign IPSec policy in Active Directory, you must use Group Policy. For more information, see To assign or unassign IPSec policy in Group Policy.

You can use the IP Security Policies snap-in to manage both local computer and domain policy.

Policies stored in Active Directory

A Group Policy object defines access, configuration, and usage settings for accounts and resources. IPSec policies can be assigned to the Group Policy object of a site, domain, or organizational unit. When the IPSec policy is applied to one of the Group Policy objects for the Active Directory object, the IPSec policy is propagated to any computer accounts that are affected by that Group Policy object. For more information, see Active Directory overview.

When assigning an IPSec policy in Active Directory, consider the following:

  • The list of all IPSec policies is available to assign at any level in the Active Directory hierarchy. However, only a single IPSec policy can be assigned at a specific level in Active Directory.
  • IPSec policies that are configured and assigned for the domain take precedence over the local, active IPSec policy when that computer is a member of the domain.
  • IPSec policies that are assigned to organizational units in Active Directory take precedence over domain level policy for any members of that organizational unit.
  • An organizational unit inherits the policy of its parent organizational unit unless either policy inheritance is explicitly blocked or policy is explicitly assigned.
  • IPSec policies from different organizational units are never merged.
  • The highest possible level of the Active Directory hierarchy should be used to assign policies to reduce the amount of configuration and administration required.
  • An IPSec policy might remain active even after the Group Policy object to which it is assigned has been deleted. Because of this, you should unassign the IPSec policy before you delete the policy object. To prevent problems, use the following procedure:
    1. Unassign the IPSec policy in the Group Policy object.
    2. Wait 24 hours to ensure that the change is propagated.
    3. Delete the Group Policy object.

    If you delete the Group Policy object without following this procedure, computers in the Active Directory container to which the IPSec policy is assigned treat the IPSec policy as if it cannot be located and continue to use a cached copy.

  • Group Policy backup and restore in Active Directory must also include IPSec policies to ensure consistency.

The IPSec Policy Agent on a computer running Windows XP Professional polls Active Directory for updates to the assigned IPSec policy. This polling does not detect a change in domain or organizational unit membership or the assigning or unassigning of a new policy. These events are detected when the Winlogon service polls for changes in Group Policy, which occurs by default every 90 minutes. The Winlogon service discovers these changes, notifies the IPSec Policy Agent, and the changes are applied.

For information about how to assign IPSec policy to objects in Active Directory, see To assign or unassign IPSec policy in Group Policy.

Notes

  • The IPSec policy integration with Active Directory described here does not apply to computers running Windows XP Home Edition.
  • You cannot administer Active Directory-based IPSec policy from a computer running Windows XP Home Edition.

Local computer policy

Each computer running Windows XP has exactly one local Group Policy object, often called the local computer policy. In using this local Group Policy object, Group Policy settings can be stored on individual computers regardless of whether they are members of an Active Directory domain. The local Group Policy object can be overwritten by Group Policy objects associated with sites, domains, or organizational units in an Active Directory environment. On a network without an Active Directory domain (lacking a Windows 2000 domain controller), the local Group Policy object settings determine IPSec behavior because they are not overwritten by other Group Policy objects.

For information about how to assign IPSec policy locally, see To assign or unassign IPSec policy on a computer.

For more information about Group Policy, see Group Policy overview.


Special IPSec considerations

The following special IPSec considerations can help simplify the administration of IPSec policies.

IP filter lists

Consider using the following methods when configuring IP filter lists:

  • Use general filters when you want a single filter to apply to a group of computers. For example, when configuring the filter, use My IP address, Any IP Address, or a subnet address instead of specifying a specific computer's source and destination IP addresses.
  • Define filters that allow you to group and secure traffic from logically associated segments or subnets within your network.
  • Be aware that when you are viewing the IPSec policy, the order in which the filters apply is not related to the order displayed. The rule display order is in reverse alphabetic order based on the name of the IP filter list for the rule. All filters are simultaneously retrieved by the IPSec Policy Agent during system startup, and are processed and sorted from most specific to least specific.

Filter actions

Consider the following special requirements when configuring IP filter actions:

  • If there is specific traffic that you always want to block, such as traffic to and from either invalid addresses or specific TCP ports, create a blocking filter action with the Security Method set to Block.
  • If there is specific traffic that you always want to be unsecured, create a permit filter action with the Security Method set to Permit.
  • When you are configuring custom security methods for encrypted traffic, you should only set the ESP confidentiality selection to <None> when a higher layer protocol provides data encryption.
  • When you are planning Internet scenarios, such as IPSec tunneling between branch offices, consider using a list of security methods that specifies high levels of security (for example, 3DES only), short key lifetimes (less than 50 MB), and perfect forward secrecy (PFS) for the master and session keys. This helps protect against known key attacks.

VPN connections using L2TP

When you are using Layer Two Tunneling Protocol (L2TP) for your remote access or router-to-router VPN connections, certificates are the default authentication method for main mode security associations. Computer certificates must be installed either on both the VPN server and remote access client (for remote access connections), or on both the calling router and answering router (for router-to-router VPN connections). The simplest method for installing computer certificates is to configure your Windows 2000 domains to auto-enroll computer certificates for domain members. For more information, see Machine certificates for L2TP over IPSec VPN connections.

When you are using the certificate authentication method for L2TP connections, the list of certification authorities (CAs) is not configurable. Instead, each computer in the L2TP connection sends a list of root CAs to its IPSec peer from which it accepts a certificate for authentication. The root CAs in this list correspond to the root CAs that issued computer certificates to the computer. For example, if Computer A was issued computer certificates by root CAs CertAuth1 and CertAuth2, it notifies its IPSec peer during main mode negotiation that it will accept certificates for authentication from only CertAuth1 and CertAuth2. If the IPSec peer, Computer B, does not have a valid computer certificate issued from either CertAuth1 or CertAuth2, IPSec main mode negotiation fails.

When you are using the certificate authentication method for L2TP connections, the list of certification authorities (CAs) is not configurable. Instead, ensure one of the following before attempting an L2TP connection:

  • Both the VPN client and VPN server were issued computer certificates from the same CA.
  • Both the VPN client and VPN server were issued computer certificates from CAs that follow a certificate chain up to the same root CA.

In general, the VPN client must have a valid computer certificate installed that was issued by a CA that follows a valid certificate chain from the issuing CA up to a root CA that the VPN server trusts. Additionally, the VPN server must have a valid computer certificate installed that was issued by a CA that follows a valid certificate chain from the issuing CA up to a root CA that the VPN client trusts.

For example, if the VPN client was issued computer certificates by root CAs CertAuth1 and CertAuth2, it notifies the VPN server during main mode negotiation that it will accept certificates for authentication from only CertAuth1 and CertAuth2. If the VPN server does not have a valid computer certificate issued from a CA that follows a certificate chain to either CertAuth1 or CertAuth2, IPSec main mode negotiation fails.

A single CA commonly issues computer certificates to all computers in an organization. Because of this, all computers within the organization both have computer certificates from a single CA and request certificates for authentication from the same single CA.

Windows XP VPN clients also support the use of preshared key authentication to create L2TP/IPSec connections. Preshared key is a weak authentication method and is recommended only as an interim measure, when your public key infrastructure (PKI) is being deployed to obtain computer certificates, or when preshared key authentication is required by your VPN server.

For a Windows XP remote access VPN client, you can enable preshared key authentication and configure a preshared key in IPSec settings on the Security tab in the properties of a VPN connection in Network Connections.

Domain controllers and DHCP, DNS, or WINS servers

Before enabling IPSec for a computer that functions as a domain controller or DHCP, DNS, or WINS server, determine whether all of the clients are IPSec-enabled. If the assigned IPSec policy is not configured to allow fall back to clear, which permits unsecured communication with non-IPSec-capable clients, secure negotiation will fail and access to these network services will be blocked.

SNMP

If you are using Simple Network Management Protocol (SNMP) and your SNMP management systems or agents include computers running Windows XP that are configured with IPSec policy, you can add a rule to the appropriate IPSec policy to prevent SNMP messages from being secured. The filter list must specify the source and destination addresses of the SNMP management systems and agents for all UDP traffic that uses ports 161 and 162. The filter action should be set to Permit, which passes through any traffic that matches the IP filter list.


IPSec security considerations

Before deploying IPSec in your organization, consider the following security issues:

  • 3DES and computers running Windows 2000
  • Authentication method
  • Firewall packet filtering
  • Protected traffic (using AH vs. ESP)
  • Unprotected traffic (using the default response rule)

3DES and computers running Windows 2000

IPSec policies allow the choice of a strong encryption algorithm, 3DES, which provides stronger encryption than DES for higher security. All computers to which the policy is assigned will receive this policy. Computers running Windows 2000 must have the High Encryption Pack or Service Pack 2 or later installed in order to perform the 3DES algorithm. If a computer running Windows 2000 receives a 3DES setting, but does not have the High Encryption Pack or Service Pack 2 or later installed, then the 3DES setting is set to the weaker DES.

Authentication method

If the computers in your enterprise are part of an Active Directory domain, IPSec main mode authentication can be accomplished with the default authentication method (Kerberos V5). You do not have to deploy public key certificates for intranet communication. However, computers running Windows XP Home Edition do not support the Kerberos V5 authentication method.

Firewall packet filtering

For a firewall, security gateway, router, or any other server or device that is connected to the Internet and providing packet filtering capabilities to computers on a perimeter network (also known as a demilitarized zone or DMZ), special filtering must be enabled on that computer to ensure that packets secured with IPSec are allowed to be forwarded to computers on the perimeter network. The firewall or other device should normally allow the following types of traffic to pass through it:

  • IP Protocol ID of 50 (0x32) for IPSec Encapsulating Security Protocol (ESP) traffic
  • IP Protocol ID of 51 (0x33) for IPSec Authentication Header (AH) traffic
  • UDP port 500 (0x1F4) for Internet Key Exchange (IKE) negotiation traffic

Most packet filtering software allows you to be more specific. You can define separate packet filters for incoming traffic (input filters), for outbound traffic (output filters), and for each interface. Additionally, you can specify IP addresses for the IPSec computers on the perimeter network. The most restrictive packet filters for IPSec traffic exchanged with a single IPSec computer on the perimeter network are outlined in the following sections.

Filters on the Internet interface

Configure the following input packet filters on the Internet interface of the firewall to allow the specific types of traffic:

  • Destination IP address of the IPSec computer's perimeter network interface and UDP destination port of 500 (0x1F4)

    • This filter allows IKE traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    • This filter allows IPSec ESP traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    • This filter allows IPSec AH traffic to be sent to the IPSec computer on the perimeter network.

Configure the following output packet filters on the Internet interface of the firewall to allow the specific types of traffic:

  • Source IP address of the IPSec computer's perimeter network interface and UDP source port of 500 (0x1F4)

    • This filter allows IKE traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    • This filter allows IPSec ESP traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    • This filter allows IPSec AH traffic to be sent from the IPSec computer on the perimeter network.

Filters on the perimeter network interface

Configure the following input packet filters on the perimeter network interface of the firewall to allow the specific types of traffic:

  • Source IP address of the IPSec computer's perimeter network interface and UDP source port of 500 (0x1F4)

    • This filter allows IKE traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    • This filter allows IPSec ESP traffic to be sent from the IPSec computer on the perimeter network.

  • Source IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    • This filter allows IPSec AH traffic to be sent from the IPSec computer on the perimeter network.

Configure the following output packet filters on the perimeter network interface of the firewall to allow the specific types of traffic:

  • Destination IP address of the IPSec computer's perimeter network interface and UDP destination port of 500 (0x1F4)

    • This filter allows IKE traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 50 (0x32)

    • This filter allows IPSec ESP traffic to be sent to the IPSec computer on the perimeter network.

  • Destination IP address of the IPSec computer's perimeter network interface and IP Protocol ID of 51 (0x33)

    • This filter allows IPSec AH traffic to be sent to the IPSec computer on the perimeter network.

Protected traffic (using AH vs. ESP)

When deciding whether to use AH or ESP to protect your IP traffic, consider the following:

  • AH provides both data authentication and integrity services through the computation and inclusion of a keyed hash for each packet. With AH, the hash calculation includes the entire IP packet and header. Some fields that are allowed to change in transit are excluded. Packet replay services are provided through the inclusion of a sequence number for each packet.
  • ESP provides data authentication and integrity services through the computation and inclusion of a keyed hash for each packet. With ESP, the hash calculation only includes the ESP header, trailer, and payload. The IP header is not protected with the hash. ESP provides data confidentiality services by encrypting the ESP payload with the DES (Data Encryption Standard) or 3DES (triple DES) encryption algorithms. Packet replay services are provided through the inclusion of a sequence number for each packet.

It is easier to calculate and verify the hash for each packet than it is to encrypt and decrypt each packet. If performance is a consideration, you can use AH to protect most of your traffic. When confidentiality is required, you can use ESP instead. For example, you can use AH to protect traffic on your intranet and ESP for traffic that is sent over the Internet.

Note

  • This performance consideration assumes that you are not using IPSec offload network adapters in your organization. Offload network adapters perform cryptographic calculations, such as the calculation and verification of the hash and the encryption and decryption services, on the adapter itself, enhancing performance for IPSec-protected traffic.

Unprotected traffic (using the default response rule)

You can easily implement an IPSec deployment scenario that secures traffic to a specific set of servers, as follows:

  1. Place the servers in an organizational unit and assign an IPSec policy to a Group Policy object of that organizational unit with a rule that requires secured traffic between the servers and any other computer. Within the filter action for the rule, enable the secure servers to accept unsecured communication but always respond using IPSec.
  2. Place the client computers in an organizational unit and assign an IPSec policy to a Group Policy object of that organizational unit that uses only the default response rule. For more information, see Rules.

In this configuration, the following traffic between the client computers and the secure servers is unsecured:

  • The initial request sent by the client computer to establish communication is unsecured because the client does not have a rule in its policy to initiate secured communication with the secure server.
  • Although ongoing communications are secured after secured communication is negotiated between the client and secure server computers, the quick mode security association (SA) that provides security settings for traffic can eventually time out and be removed from both the secure server and the client computer. This occurs when a TCP connection exists and no traffic is sent between the client computer and the secure server for an extended period of time. The dynamic filter created on the client computer by the default response rule is also removed after the dynamic filter has been timed out.

    • If the secure server sends new data on the existing connection, it renegotiates the quick mode SA before sending this data to the client computer because a rule exists on the secure server to secure traffic between itself and all other computers. However, if the client computer sends new data after the quick mode SAs and dynamic filter time out, the new data is sent unsecured because the client does not have a rule to initiate secure communications with the secure server. For TCP connections, one or multiple TCP segments can be sent unsecured by the client computer.

A consequence of using the default response rule is that the client computer can send unsecured data to a secured server after the quick mode SA and dynamic filter have timed out. Client computers rely on the computer with which they are communicating to initiate secure communications. This reliance occurs both when communication is initiated and when it is resumed after a delay that is sufficient to time out a previously established quick mode SA and dynamic filter. To prevent client computers from sending unsecured data to secure servers, you must configure your client computer IPSec policy with additional rules that initiate secured communications to secure servers.

Note

  • The default policies for IPSec permit all Internet Control Message Protocol (ICMP) traffic. For more information, see Default IPSec policies.


IPSec deployment

This section covers:

  • Secured communication in a domain
  • Remote communications


Secured communication in a domain

Before IPSec, in order to provide data confidentiality for departments of an organization that dealt with highly sensitive information, you had to physically separate the department network from the rest of the organization's network. Access controls on sensitive data resources defined which users had permissions to view or modify the data. However, the data was sent across the network in plaintext. For example, to prevent payroll department data from being captured and viewed on the organization network, payroll department computers were placed on a separate physical network that was not connected to the organization network. Because there was no physical way to transfer data between the payroll department network and the rest of the organization's network, there could be no sniffer attacks originating from users on the organization network.

With the combination of strong access controls on data and IPSec providing data confidentiality, departments that deal with sensitive data can be physically connected to the organization network.

The following illustration shows a domain comprised of computers in an enterprise. Most intranet clients do not need to communicate securely. However, a group of servers in the network store highly sensitive information, which some intranet clients will need to access.

Domain security with IPSec

All computers have computer accounts in an Active Directory domain or organizational unit. The computers are grouped into Active Directory organizational units for security. This enables the appropriate assignment of IPSec policies, based on the function of the computers:

  • Servers that store and exchange highly sensitive information belong to the Highest Security Servers organizational unit.
  • Other servers that might use unsecured communication (although secured communication is preferred) to enable data exchange with non-IPSec-aware computers in the domain belong to the Secure Servers organizational unit.
  • Clients that initiate secure communications to the highest security servers and respond when secure communications are required by the secure servers belong to the Clients organizational unit.

When computers are grouped in organizational units, you can assign IPSec policies to only those computers that require IPSec. This allows the appropriate level of security to be assigned, avoiding excess security overhead. In this scenario, Active Directory stores the IPSec policies for all computers.

High security between the clients and the domain controller is unnecessary because Kerberos-related exchanges between the clients and the domain controller are already encrypted, and the IPSec policy transmission from Active Directory to the member computers is protected by Windows LDAP security.

IPSec should be combined with access control security. User permissions are still a necessary part of protecting access to file shares that are available on any of the computers in the Highest Security Servers or Secure Servers organizational units. IPSec secures the network level traffic so that attackers cannot interpret or modify the data.


Remote communications

Secure remote communication is achieved by one of the following:

  • Using the combination of the Layer Two Tunneling Protocol (L2TP) and IPSec (L2TP/IPSec). L2TP is used to tunnel the data across a shared or public network such as the Internet and IPSec Encapsulating Security Payload (ESP) is used to encrypt the data.
  • Using IPSec tunnels.

Remote access clients with L2TP/IPSec

A common requirement is securing communications between remote access clients and the enterprise network across the Internet. This might be a sales consultant who spends most of the time traveling, or an employee working from a home office.

Remote access clients using L2TP and IPSec

In this figure, the remote gateway is a server that provides edge security for the enterprise's intranet. The remote client represents a roving user who requires regular access to network resources and information. An ISP is used here as an example to demonstrate the path of communication when the client uses an ISP to access the Internet. L2TP is combined with IPSec to provide a simple, efficient way to build the tunnel and protect the data across the Internet.

Connecting branch offices with L2TP/IPSec

A large enterprise often has multiple sites which require communication—for example, a corporate office in New York and a sales office in Washington. In this case, L2TP is combined with IPSec to provide the VPN connection and protect the data between the sites.

L2TP/IPSec Router-to-Router VPN Connection

In this figure, the Windows 2000-based routers provide edge security. The routers might have a leased line, dial-up, or other type of Internet connection. The IPSec SA and L2TP tunnel runs between the routers only and provides protected communication through the Internet.

For more information about configuring L2TP-based router-to-router VPN connections, see Deploying router-to-router VPNs.

Securing remote communication for L2TP/IPSec

For L2TP/IPSec connections, you do not configure IPSec policies. Instead, you configure the security properties for VPN connections. When security is configured for either a VPN connection in the Network Connections folder or for a demand-dial interface in Routing and Remote Access:

  • The required filter lists, filter actions, and rules are dynamically created for the duration of the connection by the client or server VPN components in the IPSec Policy Agent.
  • IPSec authentication requires either a computer certificate or a preshared key.
  • Default key exchange settings are in effect.

The level of Internet Protocol security that is used for the duration of the connection is dependent upon the security settings of the VPN client and the VPN server.

On the VPN client, the following are the possible selections for encryption for a virtual private network connection in Network Connections:

  • No encryption. Authentication Header (AH) is used.
  • Optional encryption. If the other computer requests or requires secured communication, IPSec offers security levels ranging from ESP/3DES to AH/MD5.

    • Session key perfect forward secrecy (PFS) is not enabled unless the other computer requests it. If ESP cannot be negotiated, AH is used.

  • Required encryption. The computer requires secured communication. The security that is offered is the same as Optional encryption, except that ESP encryption is required.
  • Maximum strength encryption. The computer requires secured communication and 3DES ESP encryption.

On a Windows 2000 VPN server, the selections for VPN connection encryption (in the profile settings of the remote access policy that matches the VPN connection attempt) are the following:

  • No Encryption. Authentication Header (AH) is used.
  • Basic. ESP and 56-bit DES encryption are used.
  • Strong. ESP and 56-bit DES encryption are used.
  • Strongest. ESP and 3DES (triple DES) encryption are used.

For more information, see Remote access policies.

Using IPSec tunnels

Another method of securely connecting two routers across the Internet is to use IPSec tunneling. IPSec tunneling can be used when one of the routers does not support L2TP/IPSec connections.

For more information, see Tunnel mode and Tunnel endpoint.


Administering Internet Protocol security

This section covers:

  • Managing Internet Protocol security from the command line


Managing Internet Protocol security from the command line

In addition to using the IP Security Policies snap-in, you can use the Ipseccmd command-line tool to manage IPSec policies.

For information about other command-line tools, see Command-line reference. For more information about manageability, see Management and Administration.


Resources

This section covers:

  • IPSec RFCs and Internet drafts
  • IPSec-related Web sites
  • Windows Resource Kits
  • Troubleshooting tools
  • IP Security Monitor statistics


IPSec RFCs and Internet drafts

RFCs are an evolving series of reports, proposals for protocols, and protocol standards used by the Internet community. IPSec standards are defined in RFCs published by the Internet Engineering Task Force (IETF) and other working groups.

IPSec RFCs

  • RFC 1828: IP Authentication using Keyed MD5
  • RFC 1829: The ESP DES-CBC Transform
  • RFC 2085: HMAC-MD5 IP Authentication with Replay Prevention
  • RFC 2104: HMAC: Keyed-Hashing for Message Authentication
  • RFC 2401: Security Architecture for the Internet Protocol
  • RFC 2402: IP Authentication Header
  • RFC 2403: The Use of HMAC-MD5-96 within ESP and AH
  • RFC 2404: The Use of HMAC-SHA-1-96 within ESP and AH
  • RFC 2406: IP Encapsulating Security Payload (ESP)
  • RFC 2407: The Internet IP Security Domain of Interpretation for ISAKMP
  • RFC 2408: Internet Security Association and Key Management Protocol (ISAKMP)
  • RFC 2409: The Internet Key Exchange (IKE)
  • RFC 2410: The NULL Encryption Algorithm and Its Use with IPSec
  • RFC 2411: IP Security Document Roadmap
  • RFC 2451: The ESP CBC-Mode Cipher Algorithms

To see the most current Internet drafts on IPSec architecture and mechanisms, see IPSec-related Web sites.

IPSec Internet drafts

Internet drafts contain detailed, technical information on IPSec architecture, including the algorithms and protocols used. The Internet draft directories provide authors with the ability to distribute and solicit comments on documents that they might eventually submit for publication as an RFC.

To see the most current Internet drafts on IPSec architecture and mechanisms, see IPSec-related Web sites.

The IPSec implementation in Windows XP fully implements the following IETF IPSec architecture document:

  • A GSS-API Authentication Method for IKE (draft-ietf-ipsec-isakmp-gss-auth-0x.txt)

The IPSec implementation in Windows XP adheres to, but does not fully implement the following IETF IPSec architecture document:

  • A PKIX Profile for IKE (draft-ietf-ipsec-pki-req-0x.txt)


IPSec-related Web sites

The following Web sites contain links that relate to IPSec:

  • The IPSec Working Group page contains links to RFCs, Internet Drafts, and other IPSec-related links at the Internet Engineering Task Force Web site.
  • For information and the latest recommendations on security, see Microsoft Security at the Microsoft Web site.(http://www.microsoft.com/)
  • For information about the standards and technology for IPSec, as well as other computer security topics, see the National Institute of Standards and Technology Web site.

Note

  • XOX


Windows Resource Kits

The Windows Resource Kits contain extensive technical information on TCP/IP and IPSec. For more information, see Microsoft Windows Resource Kits.


Troubleshooting tools

You can use the following tools to troubleshoot IPSec in Windows XP:

  • IP Security Policy snap-in
  • Active Directory Users and Computers and Group Policy snap-ins
  • IP Security Monitor snap-in
  • IPSecCMD
  • Audit logging
  • Oakley log
  • Network Monitor

IP Security Policy snap-in

You can use the IP Security Policy snap-in to create, modify, and activate IPSec policies. You can create a console by adding either the IP Security Policies on Local Machine snap-in or the IP Security Policies on Active Directory snap-in. You can also access the IP Security Policies snap-in through Group Policy. For more information, see To start the IP Security Policies snap-in.

Note

  • You cannot administer Active Directory-based IPSec policy from a computer running Windows XP Home Edition.

Active Directory Users and Computers and Group Policy snap-ins

To troubleshoot policy precedence issues and determine the set of policies that are being used by IPSec clients, use the Active Directory Users and Computers and Group Policy snap-ins. Policy precedence is based upon the Group Policy inheritance model. The policy used is the policy assigned at the lowest level of the domain hierarchy for the domain container of which the computer is a member. For example, if there are IPSec policies configured both for the domain and for an organizational unit within the domain, computers that are members of the domain use the domain IPSec policies. The computers that are members of the organizational unit within the domain use the organizational unit IPSec policies. If there are no IPSec policies configured for Active Directory, local policies are used.

For more information about IPSec policy behavior in an Active Directory environment, see Policies stored in Active Directory.

IP Security Monitor snap-in

You can use the IP Security Monitor snap-in to view details on the local computer or remote computers about:

  • Main mode and quick mode generic filters, specific filters, and security associations.
  • IKE policies.
  • Negotiation policies.
  • IKE and IP security statistics.

For more information, see Monitor IPSec activity. For more information about IP security statistics, see IP Security Monitor statistics.

Note

  • The IP Security Monitor snap-in can only be used to monitor IPSec on computers running Windows XP. To monitor IPSec on a computer running Windows 2000, use the ipsecmon command at the Windows 2000 command prompt on the computer that is being monitored.

IPSecCMD

You can use IPSecCMD to configure IPSec policies, filters, and filter actions at the command prompt. For more information, see Ipseccmd. Ipseccmd can only be used on computers running Windows XP. To configure IPSec policies, filters, and filter actions at the command prompt for computers running Windows 2000, use the ipsecpol command that is provided in the Windows 2000 Server Resource Kit.

Audit logging

You can use the Windows XP Event Viewer snap-in to view the following IPSec-related events:

  • IPSec Policy Agent events in the audit log.
  • IPSec driver events in the system log. To enable IPSec driver event logging, set the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\IPSEC\DiagnosticMode registry setting to 1. You must restart the computer for this change to take effect. The IPSec driver only writes events to the system log once an hour. For additional information about IPSec driver event logging, see the Windows Resource Kits.
  • IKE events (SA details) in the audit log. To view these events, enable success or failure auditing for the Audit logon events audit policy for your domain or local computer. For more information, see To establish an audit policy.
  • IPSec policy change events in the audit log. To view these events, enable success or failure auditing for the Audit policy change audit policy for your domain or local computer. For more information, see To establish an audit policy.

Enabling audit logging and viewing the events in Event Viewer is the fastest and simplest way to troubleshoot failed main mode or quick mode negotiations.

Note

  • Audit policies on a computer running Windows XP Home Edition and cannot be configured, however, success and failure auditing for the Audit logon events and Audit policy change audit policies for the local computer are enabled by default.

Oakley log

You can use the Oakley log to view details about the SA establishment process. The Oakley log is enabled in the registry. It is not enabled by default. To enable the Oakley log, set the HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Oakley\EnableLogging registry setting to 1. The Oakley key does not exist by default and must be created. For more information about adding values to registry keys, see To add a value.

After it is enabled, the Oakley log, which is stored in the systemroot\Debug folder, records all ISAKMP main mode or quick mode negotiations. A new Oakley.log file is created each time the IPSec Policy Agent is started and the previous version of the Oakley.log file is saved as Oakley.log.sav.

To activate the new EnableLogging registry setting after modifying its value, stop and start the IPSec Policy Agent and related IPSec services by running the net stop policyagent and net start policyagent commands at the command prompt. If you are restarting the IPSec Policy Agent and related services on a computer running Windows 2000 Server and the Routing and Remote Access service, use the following sequence of commands:

  1. Stop the Routing and Remote Access service using the net stop remoteaccess command.
  2. Stop the IPSec services using the net stop policyagent command.
  3. Start the IPSec services using the net start policyagent command.
  4. Start the Routing and Remote Access service using the net start remoteaccess command.

Caution

  • Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

Network Monitor

You can use Microsoft Network Monitor to troubleshoot IPSec. Network Monitor 2.0, included with both Windows 2000 Server and Systems Management Server 2.0, features parsers for ISAKMP, AH, and ESP. However, Network Monitor does not parse the encrypted portions of IPSec-protected traffic.

For more information, see Network Monitor.


IP Security Monitor statistics

IKE Statistics

The following table describes the items in the list of IKE statistics in the IP Security Monitor snap-in.

IKE Statistic | Description
Active Acquire | An acquire is a request by the IPSec driver to have IKE perform a task. The active acquire statistic includes the outstanding request and the number of any queued requests. Typically, the number of active acquires is 1. Under a heavy load, the number of active acquires is 1 and the number of requests that are queued by IKE for processing.
Active Receive | The number of IKE messages received that are queued for processing.
Acquire Failures | The number of times that an acquire has failed.
Receive Failures | The number of times that the Windows Sockets WSARecvFrom() function has failed when receiving IKE messages.
Send Failures | The number of times that the Windows Sockets WSASendTo() function has failed when sending IKE messages.
Acquire Heap Size | The number of entries in the acquire heap, which stores active acquires. This number increases under a heavy load and then gradually decreases over time, as the acquire heap is cleared.
Receive Heap Size | The number of entries in the IKE receive buffers for incoming IKE messages.
Negotiation Failures | The total number of negotiation failures that occurred during main mode (also known as Phase I) or quick mode (also known as Phase II) negotiation. If you are having difficulty communicating securely, attempt the communication and determine whether the number of Negotiation Failures increases. If it does, check your authentication and security method settings for an unmatched authentication method, an incorrect authentication method configuration (for example, the use of preshared keys that do not match), or unmatched security methods or settings.
Authentication Failures | The total number of identity authentication failures (Kerberos, certificate, and preshared key) that occurred during main mode negotiation. If you are having difficulty communicating securely, attempt the communication and determine whether the number of Authentication Failures increases. If it does, check your authentication settings for either an unmatched authentication method or an incorrect authentication method configuration (for example, the use of preshared keys that do not match).
Invalid Cookies Received | A cookie is a value contained in a received IKE message that is used by IKE to find the state of an active main mode. A cookie in a received IKE message that cannot be matched with an active main mode is invalid.
Total Acquire | The total number of work requests submitted by IKE to the IPSec driver.
Total Get SPI | The total number of requests submitted by IKE to the IPSec driver to obtain a unique Security Parameters Index (SPI).
Key Additions | The number of outbound quick mode security associations (SAs) added by IKE to the IPSec driver.
Key Updates | The number of inbound quick mode SAs added by IKE to the IPSec driver.
Get SPI Failures | The number of requests submitted by IKE to the IPSec driver to obtain a unique SPI that failed.
Key Addition Failures | The number of outbound quick mode SA addition requests submitted by IKE to the IPSec driver that failed.
Key Update Failures | The number of inbound quick mode SA addition requests submitted by IKE to the IPSec driver that failed.
ISADB List Size | The number of main mode state entries, including negotiated main modes, main modes in progress, and main modes that failed and have not been deleted.
Connection List Size | The number of quick mode state entries.
Oakley Main Mode | The total number of successful SAs created during main mode negotiations.
Oakley Quick Mode | The total number of successful SAs created during quick mode negotiations. Because there are typically multiple quick mode SAs created for each main mode SA, this number does not necessarily match the main mode number.
Invalid Packets Received | The number of received IKE messages that are invalid, including IKE messages with invalid header fields, incorrect payload lengths, and incorrect values for the responder cookie (when it should be set to 0). Invalid IKE messages are commonly caused by stale retransmitted IKE messages or an unmatched preshared key between the IPSec peers.
Soft Associations | The total number of negotiations that resulted in the use of plaintext (also known as soft SAs). This typically reflects the number of associations formed with computers that did not respond to main mode negotiation attempts. This can include both non-IPSec-aware computers and IPSec-aware computers that do not have IPSec policy to negotiate security with this IPSec peer. Although soft SAs are not the result of main mode and quick mode negotiations, they are still treated as quick mode SAs.

IPSec Statistics

The following table describes the items in the list of IPSec statistics in the IP Security Monitor snap-in.

IPSec Statistic | Description
Active Security Associations | The number of active IPSec SAs.
Offloaded Security Associations | The number of active IPSec SAs offloaded to hardware.
Pending Key Operations | The number of IPSec key operations in progress.
Key Additions | The total number of successful IPSec SA negotiations.
Key Deletions | The number of key deletions for IPSec SAs.
Re-Keys | The number of rekey operations for IPSec SAs.
Active Tunnels | The number of active IPSec tunnels.
Bad SPI Packets | The total number of packets for which the Security Parameters Index (SPI) was incorrect. The SPI is used to match inbound packets with SAs. If the SPI is incorrect, it might mean that the inbound SA has expired and a packet using the old SPI has recently arrived. This number is likely to increase if rekey intervals are short and there are a large number of SAs. Since SAs expire under normal conditions, a bad SPI packet does not necessarily mean that IP security is failing.
Packets Not Decrypted | The total number of packets that failed decryption. This failure might indicate that a packet arrived for which the SA had expired. If the SA expires, the session key used to decrypt the packet is also deleted. This does not necessarily indicate that IP security is failing.
Packets Not Authenticated | The total number of packets for which data could not be verified. This failure is most likely caused by an expired SA.
Packets With Replay Detection | The total number of packets that contained a valid Sequence Number field.
Confidential Bytes Sent | The total number of bytes sent using the ESP protocol.
Confidential Bytes Received | The total number of bytes received using the ESP protocol.
Authenticated Bytes Sent | The total number of bytes sent using the AH protocol.
Authenticated Bytes Received | The total number of bytes received using the AH protocol.
Transport Bytes Sent | The total number of bytes sent using IPSec transport mode.
Transport Bytes Received | The total number of bytes received using IPSec transport mode.
Bytes Sent in Tunnels | The total number of bytes sent using IPSec tunnel mode.
Bytes Received in Tunnels | The total number of bytes received using IPSec tunnel mode.
Offloaded Bytes Sent | The total number of bytes sent using hardware offload.
Offloaded Bytes Received | The total number of bytes received using hardware offload.


Troubleshooting

What problem are you having?

Traffic is not being secured.

Cause: An IPSec security association (SA) that secures traffic (also known as a hard SA) is not being established.

Solution: This might be caused by either filter list misconfiguration or the enabling of unsecured communication (also known as fallback to clear). Examine the policies that are assigned to both IPSec peers and determine, based on the rules, filter lists, and filter action settings, whether IPSec requires traffic to be secured. Check to determine whether the Allow unsecured communication with non-IPSec-aware computer option on the Security Methods tab of the appropriate filter action is enabled. Use audit logging to determine the cause of the failed negotiation.

See also: Troubleshooting tools; To add or edit IPSec security methods

IPSec SA negotiation is failing.

Cause: An IPSec security association (SA) that secures traffic (also known as a hard SA) is not being established. If fallback to clear is enabled to allow secure computers to communicate with non-IPSec-aware computers, the negotiation failure is expected.

Solution: Negotiation failure between two IPSec-aware computers commonly occurs when the default policies are used on two IPSec peers and one or both are not members of an Active Directory domain. Computers running Windows XP Home Edition do not support Active Directory domain membership or the Kerberos V5 authentication method. Default policies use the Kerberos V5 authentication method, which does not work for computers that are not Active Directory domain members. A negotiation failure between two IPSec-aware computers is most often caused by a mismatched key exchange method, authentication method, or security method. Examine the policies that are assigned to both IPSec peers and determine, based on the advanced settings, rules, filter lists, and filter action settings, whether IPSec should be securing the traffic. If it should, determine whether there is a mismatched key exchange, authentication method, or security method that is preventing a successful negotiation of a hard SA. Use audit logging to determine the cause of the failed negotiation.

See also: Troubleshooting tools; To add or edit IPSec security methods; To define IPSec authentication methods; To create key exchange security methods

IPSec SA negotiation is failing when using certificate authentication.

Cause: The correct computer certificates are not installed on the IPSec peers.

Solution: Install the correct computer certificates on the IPSec peers.

Cause: An incorrect certification authority (CA) is selected in the authentication method.

Solution: For the rules of each IPSec peer, select the CA that issued the computer certificate that is installed on the other IPSec peer.

See also: Troubleshooting tools; Authentication methods; To define IPSec authentication methods

Some computers are unable to communicate with secure computers.

Cause: Secure computers are using a lockdown policy that requires client computers to initiate secure communications. The computers that are unable to communicate are not configured with a rule to initiate secure communication.

Solution: Either allow unsecured incoming connection requests on the secure servers or configure additional rules for the computers that are unable to communicate so that they can initiate secure communication with the secure computers.

See also: To add or edit filter actions

Cause: Client computers are not IPSec-capable and secure computers are not allowing unsecured communications.

Solution: Either install client computers that are IPSec-capable or reconfigure the secure computers to allow communication with computers that are not IPSec-capable.

Cause: Some computers are configured with a policy that has a key exchange, authentication method, or security method that is mismatched with the policy configured for the secure computers.

Solution: Examine the policies that are assigned to both IPSec peers and determine, based on the advanced settings, rules, filter lists, and filter action settings, whether IPSec should be securing the traffic. If it should, determine whether there is a mismatched key exchange, authentication method, or security method that is preventing a successful SA negotiation. Use audit logging to determine the cause of the failed negotiation.

See also: Troubleshooting tools; To add or edit IPSec security methods; To define IPSec authentication methods; To create key exchange security methods

Local computer IPSec policy is not being used.

Cause: The computer is a member of an Active Directory domain and is receiving domain-based IPSec policy.

Solution: All domain members receive domain-based IPSec policy, which overrides local IPSec policy settings. To prevent this, remove the computer from the domain.

Cause: IPSec services are not started.

Solution: Start or restart the IPSec services.

See also: To restart the IPSec services

Unable to assign or unassign an IP Security policy.

Cause: You cannot assign or unassign policies when IP Security Policies are used for Active Directory.

Solution: To assign Active Directory-based policies, use IP Security Policies within Group Policy.

See also: To assign or unassign IPSec policy in Group Policy

DES is being used when 3DES is needed.

Cause: The IPSec peer is a computer running Windows 2000 without either the Windows 2000 High Encryption Pack or Windows 2000 Service Pack 2 (or later) installed.

Solution: Install either the Windows 2000 High Encryption Pack or Windows 2000 Service Pack 2 (or later) on the computer running Windows 2000.

Unsecured traffic is being sent to a secure server.

Cause: The client computer is using the default response rule.

Solution: When the client computer is configured with the default response rule, it sends unsecured packets when it either initiates communication or restarts communication after a delay long enough for the quick mode SA and dynamic filter list to time out. To prevent this, configure additional rules for client computers that always initiate secure communication to secure servers.

See also: IPSec security considerations

Event Viewer is displaying error messages related to IPSec.

Cause: There might be a problem with either the IPSec Policy Agent or the Oakley service.

Solution: View the system and security event logs for messages that might help in determining the origin of the problem.

A policy location error message is displayed.

Cause: This error might occur if the IPSec Policy Agent is attempting to retrieve policy from an incorrect storage location.

Solution: Examine both the IPSec Policy Agent events in Event Viewer and the IPSec Policy Agent log for messages that indicate from what location policy is being retrieved (the local registry or the domain). If HKLM\SOFTWARE\Policies\Microsoft\Windows\IPSec\GPTIPSECPolicy is set to 1, then IPSec policy is obtained from Active Directory. Otherwise, it is obtained from the registry. The local computer warns a user that domain policy is in effect when it starts. If the policy is supposed to be domain-based, but cannot be obtained, you can delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Group Policy\History and then restart the computer.

Caution

  • Incorrectly editing the registry may severely damage your system. Before making changes to the registry, you should back up any valued data on the computer.

See also: View event logs.

Event Viewer is displaying "Bad SPI" error message.

Cause: This error might occur if a key lifetime value is set too low, or if the SA has expired but the sender continues to transmit data to the receiver.

Solution: To determine and correct the problem, use IP Security Monitor to examine the number of rekeys. If the number of rekeys is very large compared to the amount of time the connections have been active, set the key lifetimes in the policy to be longer. Good values for high-traffic Ethernet connections are greater than both 50 megabytes and five minutes. This event is usually logged when an SA has been deleted on one IPSec peer, and the other IPSec peer sends secure traffic on the SA. This is usually a temporary situation that occurs on systems that are busy and fixes itself.

IP Security Monitor snap-in displays the error message "The IPSec server is unavailable or incompatible with the IPSec monitor."

Cause: You are using the IP Security Monitor snap-in to monitor IPSec on a computer running Windows 2000.

Solution: The IP Security Monitor snap-in can only be used to monitor IPSec on computers running Windows XP. To monitor IPSec on a computer running Windows 2000, use the ipsecmon command at the Windows 2000 command prompt on the computer that is being monitored.

See also: Troubleshooting tools

Ipseccmd command cannot configure IPSec or display IPSec statistics on a computer.

Cause: You are using ipseccmd to configure IPSec or display IPSec statistics on a computer running Windows 2000.

Solution: The ipseccmd command can only be used to configure IPSec on computers running Windows XP. To configure IPSec policies, filters, and filter actions at the command prompt for computers running Windows 2000, use the ipsecpol command provided in the Windows 2000 Server Resource Kit.

See also: Troubleshooting tools