{"id":945,"date":"2026-10-26T22:11:43","date_gmt":"2026-10-26T22:11:43","guid":{"rendered":"https:\/\/windows-vps.org\/blog\/?p=945"},"modified":"2026-10-26T22:11:43","modified_gmt":"2026-10-26T22:11:43","slug":"windows-server-ipsec-domain-isolation-firewall-requirements","status":"publish","type":"post","link":"https:\/\/windows-vps.org\/blog\/windows-server-ipsec-domain-isolation-firewall-requirements\/","title":{"rendered":"IPsec Domain Isolation: Firewall Requirements and Failure Modes on a Windows VPS"},"content":{"rendered":"
\n

IPsec domain isolation means the host accepts inbound traffic only from peers that have authenticated, and drops everything else before any service sees it. It works, and it is the most reliable way to lock down RPC, SMB and WMI on a Windows network without port-level guesswork. It is also the fastest way to make a remote VPS unreachable: the rules that protect the machine are read from a policy that arrives over the network the policy is securing. On a hosted guest this is a boot-time blackhole with no local recovery path. This page covers the rule types, the negotiation ports that must stay open, certificate versus Kerberos on a non-domain-joined VPS, and a rollout order with a tested rollback. The images used for the examples are the Windows Server VPS builds at windows-vps.org<\/a>.<\/p>\n

Connection Security Rules Are Not Firewall Rules<\/h2>\n

Windows Firewall has two independent engines and mixing them up is the root of most isolation failures. Firewall rules decide whether a packet is allowed. Connection security rules decide whether two hosts authenticate to each other and whether the traffic is encrypted or integrity-protected. Isolation requires both: a connection security rule that demands authentication, and a firewall rule that only matches authenticated traffic.<\/p>\n\n\n\n\n\n\n\n\n\n
<\/th>\nConnection security rule<\/th>\nFirewall rule<\/th>\n<\/tr>\n<\/thead>\n
Purpose<\/td>\nAuthenticate peers, optionally encrypt<\/td>\nPermit or block traffic<\/td>\n<\/tr>\n
Match by<\/td>\nEndpoint pair, protocol, auth method<\/td>\nProgram, port, address, user, package<\/td>\n<\/tr>\n
Enforcement<\/td>\nIKE negotiation on UDP 500\/4500<\/td>\nPacket filter on all traffic<\/td>\n<\/tr>\n
Failure mode<\/td>\nNegotiation timeout, no filter installed<\/td>\nBlocked packet, logged as 5152<\/td>\n<\/tr>\n
Key cmdlets<\/td>\nNew-NetIPsecRule<\/code><\/td>\nNew-NetFirewallRule<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

The firewall rule that makes isolation actually bite uses the -Authentication<\/code> filter. Without it, an IPsec rule that fails to negotiate simply does not apply, and the traffic flows unauthenticated \u2014 which is the difference between a policy that protects the host and one that only decorates the event log.<\/p>\n

New-NetFirewallRule -DisplayName 'SMB - authenticated only' -Direction Inbound `\n  -Protocol TCP -LocalPort 445 -Action Allow `\n  -Authentication Required -Encryption NotRequired -Profile Domain,Private<\/code><\/pre>\n
Request, Require, and the Boot-Time Blackhole<\/h2>\n
A connection security rule has an authentication mode. The difference between the three is precisely what happens on a host that cannot reach its authentication partner.<\/p>\n