IPsec domain isolation means the host accepts inbound traffic only from peers that have authenticated, and drops everything else before any service sees it. It works, and it is the most reliable way to lock down RPC, SMB and WMI on a Windows network without port-level guesswork. It is also the fastest way to make a remote VPS unreachable: the rules that protect the machine are read from a policy that arrives over the network the policy is securing. On a hosted guest this is a boot-time blackhole with no local recovery path. This page covers the rule types, the negotiation ports that must stay open, certificate versus Kerberos on a non-domain-joined VPS, and a rollout order with a tested rollback. The images used for the examples are the Windows Server VPS builds at windows-vps.org.
Connection Security Rules Are Not Firewall Rules
Windows Firewall has two independent engines and mixing them up is the root of most isolation failures. Firewall rules decide whether a packet is allowed. Connection security rules decide whether two hosts authenticate to each other and whether the traffic is encrypted or integrity-protected. Isolation requires both: a connection security rule that demands authentication, and a firewall rule that only matches authenticated traffic.
| Connection security rule | Firewall rule | |
|---|---|---|
| Purpose | Authenticate peers, optionally encrypt | Permit or block traffic |
| Match by | Endpoint pair, protocol, auth method | Program, port, address, user, package |
| Enforcement | IKE negotiation on UDP 500/4500 | Packet filter on all traffic |
| Failure mode | Negotiation timeout, no filter installed | Blocked packet, logged as 5152 |
| Key cmdlets | New-NetIPsecRule |
New-NetFirewallRule |
The firewall rule that makes isolation actually bite uses the -Authentication filter. Without it, an IPsec rule that fails to negotiate simply does not apply, and the traffic flows unauthenticated — which is the difference between a policy that protects the host and one that only decorates the event log.
New-NetFirewallRule -DisplayName 'SMB - authenticated only' -Direction Inbound `
-Protocol TCP -LocalPort 445 -Action Allow `
-Authentication Required -Encryption NotRequired -Profile Domain,PrivateRequest, Require, and the Boot-Time Blackhole
A connection security rule has an authentication mode. The difference between the three is precisely what happens on a host that cannot reach its authentication partner.
- Request outbound / Request inbound — negotiate if the peer offers, fall back to clear text if not. Nothing breaks, nothing is protected. This is the mode to deploy first.
- Request outbound / Require inbound — the host asks for authentication, and inbound connections must present it. Common stepping stone; local outbound still works when the partner is gone.
- Require outbound / Require inbound — the host refuses to send or receive anything unauthenticated. This is the blackhole state: no DHCP reply accepted, no DNS, no RDP, no domain controller.
The blackhole is not a firewall block, so netsh advfirewall show allprofiles shows the profile as ON and healthy while nothing works. Event IDs to look for: 4653 (negotiation failed), 5452 and 5451 (IKE security association established/ended), 4710 (IPsec policy applied), and 5152 (blocked packet). A host that boots and logs 4710 followed by repeated 4653 with no 5451 is negotiating with peers that cannot answer.
Get-WinEvent -FilterHashtable @{LogName='Security'; Id=4653,5451,5452,4710} -MaxEvents 30 |
Select TimeCreated,Id,@{n='Msg';e={$_.Message.Substring(0,[Math]::Min(160,$_.Message.Length))}} | Format-Table -WrapPorts You Must Leave Open
IKE runs outside IPsec — by definition, negotiation cannot be protected by the thing being negotiated. On a NATed cloud network, IPv4 IKE also requires NAT-T on UDP 4500, and the peer must be reachable on both.
| Port | Protocol | Why | Direction |
|---|---|---|---|
| 500 | UDP | IKE main mode | In + Out |
| 4500 | UDP | NAT-T / IKE quick mode | In + Out |
| 88 | TCP/UDP | Kerberos to KDC (if Kerberos auth) | Once, outbound |
| 135 | TCP | RPC to DC / CA for policy and revocation | Once, outbound |
| 80/443 | TCP | CRL/OCSP if using certificate auth | Once, outbound |
| 3389 | TCP | RDP from management subnet, as an exemption | In |
Port 88 and the CRL fetch are the ones that cause delayed failure rather than immediate failure: negotiation succeeds at boot with a cached ticket or certificate, then fails hours later when the ticket expires and the KDC is unreachable under a Require rule.
Why a Remote VPS Is Easier to Lock Out Than a Branch Office
Three structural differences matter. First, there is no out-of-band console — the provider’s web console is a VNC session that presents as a keyboard and mouse, not as a network path, so IPsec policy does not usually block it, but a Require-outbound rule can still break the guest agent that the console depends on. Second, the NIC comes up before any domain services, and if the boot policy is stored in a GPO rather than locally, the machine has to reach the DC to learn that it must authenticate to reach the DC. Third, cloud networks commonly run the management interface in a different subnet from the data interface, and a policy that scopes by “local subnet” will not match it.
Test the boot policy locally before it comes from Group Policy. The local policy store is what gets applied before the network stack is fully up:
# Inspect the boot-time security policy
netsh ipsec static show policy all
# Or via PowerShell, after applying with a local store
Get-NetIPsecRule | Select DisplayName,Enabled,InboundSecurity,OutboundSecurity,Profile
(Get-NetIPsecMainModeCryptoSet).Proposals | Select AuthenticationMethod,EncryptionAlgorithmIf the IPsec policy is delivered by GPO and the GPO is only reachable after IPsec is up, the host deadlocks on the first boot after a rebuild. Set PolicyStore ActiveStore locally first, prove it, then migrate the same rule set into GPO.
Certificate Versus Kerberos on a Non-Domain-Joined VPS
Workgroup machines have no domain to draw a Kerberos ticket from, so isolation on them means certificate authentication. Both sides need a computer certificate with the Client Authentication and Server Authentication EKUs, issued by a CA the other side trusts, and the CA must be reachable for revocation checks at least periodically.
| Kerberos (domain-joined) | Certificate (workgroup VPS) | |
|---|---|---|
| Prerequisite | DC reachable, clock within 5 min | CA, issued cert, trusted root on both sides |
| Ports | 88 TCP/UDP, 135 for policy | 80/443 for CRL/OCSP |
| Renewal failure | Ticket expires in 10 h by default | Cert expires in 1–2 years, easy to forget |
| Clock sensitivity | High — 5 minute skew breaks it | Low, but validity window matters |
| Typical use | Member servers, DC-to-DC | Standalone VPS, DMZ hosts |
The clock point is not theoretical on a VPS. Hypervisor time drift on a suspended guest can exceed the 5-minute Kerberos tolerance, and the resulting symptom is every authenticated connection failing at once with no firewall log entry. Verify before you deploy a Require rule:
w32tm /query /status
w32tm /stripchart /computer:time.windows.com /samples:3 /dataonlySafe Rollout Order and the Rollback Path
Rollback has to be executable from the provider’s console, not from a network share that the policy just cut off. Export the working policy to the local disk before each step.
Rollout Sequence
The order below is designed so that every step is reversible with a single command and no step depends on the network that the previous step might break.
The sequence below assumes one VPS peer and one management workstation, and that you have console access for the whole window.
- Open UDP 500 and 4500 inbound on the host firewall for the management subnet only.
- Create the connection security rule in Request/Request mode, scoped to a single test protocol (SMB on 445, from one workstation).
- Confirm negotiation in the log: 5451/5452 pairs appear and
Get-NetIPsecQuickModeSAlists the association. - Apply
-Authentication Requiredto that one firewall rule. Test file copy, then test a reboot. - Widen to RPC and WMI once SMB is stable. Add a firewall exemption rule for RDP from the management subnet, above the isolation rules, with higher precedence.
- Move to Request/Require. Reboot and confirm the host comes back without console intervention.
- Only then change to Require/Require, and keep a second copy of the policy exported.
# Export before every change so rollback is one command
netsh advfirewall export C:\ipsec\before-require.wfw
netsh advfirewall import C:\ipsec\before-require.wfw
# Fastest safe recovery from a console session
Set-NetIPsecRule -DisplayName 'Isolation - Require' -Enabled FalseThe rollback that matters is the one that works when the network is already gone. Export the working policy before each step, keep the .wfw file on the local disk rather than a share, and confirm that you can import it from the provider console. A policy that requires a working network to repair a broken network is not a policy, it is a scheduled outage. Current Windows Server images and their default IKE and firewall posture are listed on the Windows VPS plans page.
