Connecting to a Windows VPS from Windows 11: RDP Client Setup and Login Fixes

Windows 11 ships two different Remote Desktop clients and they fail in different ways against a fresh Windows Server host: the classic mstsc.exe console that has been in the box since XP, and the Store-delivered Remote Desktop app. Most login failures are not client bugs — they are an NLA mismatch, an account that is disabled, or a port that the provider’s security group blocks before the guest ever sees the SYN. This page covers the client settings that make the first login succeed, the four error codes you will actually see, and the server-side checks that resolve each one. The host prerequisites for a session are the same on any plan, including the Windows Server 2022 VPS images at windows-vps.org.

Preflight Facts You Need Before Launching the Client

Collect these five values from the provider control panel and write them down. Guessing wastes more time than checking.

• Public IPv4 address, and whether it is static or allocated per boot.
• The administrator account name — Administrator on most images, a custom name on hardened templates.
• The initial password, plus whether the guest forces a change at first logon.
• RDP port, if the image uses something other than 3389.
• Whether a provider-level firewall or security group sits in front of the VM.

Test reachability without launching the GUI, because the error text is far more specific:

Test-NetConnection -ComputerName 203.0.113.10 -Port 3389 -InformationLevel Detailed
# or from cmd
powershell -c "tnc 203.0.113.10 -Port 3389 | Select TcpTestSucceeded,RemoteAddress"

TcpTestSucceeded : False means nothing is listening on that port from your network perspective — stop working on the client. True with a subsequent login failure means the transport is fine and the problem is authentication or NLA.

Client Setup in Windows 11

The classic client is the one to learn first, because every persisted setting lives in a plain-text .rdp file you can diff, copy and version. Launch it and connect to the IP rather than the hostname — a fresh VPS rarely has a matching PTR record and DNS caching on your workstation can point at a recycled address.

mstsc /v:203.0.113.10
mstsc /v:203.0.113.10:3390
mstsc /v:203.0.113.10 /w:1600 /h:900

On the credential prompt, use the local-account form. The .\ prefix forces the client to authenticate against the machine’s local SAM database instead of the workstation’s domain context, which is the correct target for a standalone VPS:

.\Administrator

If the server was renamed or provisioned with a custom account, VPSHOST\myadmin works equally well and is more explicit in logs. A bare Administrator is ambiguous the moment your workstation is domain-joined.

The Four Login Errors and What Each One Means

Error	Text / code	Root cause	Fix
0x800706BA	“The RDP server is unreachable”	TCP blocked or service stopped	Test-NetConnection, check TermService, check provider firewall
0x904 / 0x1104	“An authentication error has occurred”	NLA enabled on host, client sending stored credential with wrong format	Clear Credential Manager entry, retype password
0x4	“The requested session access is denied”	Account not in Remote Desktop Users, or logon locally denied	Add to Remote Desktop Users group
0x3	“The user is logged on already” / black screen	Two admin sessions occupied, or a disconnected session is holding the console	quser then logoff

The 0x904 error is the one people misdiagnose as a firewall problem. It almost always comes from a stale credential. The client retries the entry in Windows Credential Manager before prompting, and if the stored password was captured before the guest forced a first-logon change, every retry fails with the same code. Clear it:

cmdkey /list | findstr /i "203.0.113.10"
cmdkey /delete:TERMSRV/203.0.113.10

The prefix matters: TERMSRV/ followed by the exact host string you typed in the Computer field. Deleting TERMSRV/203.0.113.10:3390 will not remove an entry stored for 203.0.113.10.

Handling NLA and the Certificate Warning

Network Level Authentication authenticates before a session is created, which is why a wrong password on an NLA host produces an error dialog rather than a Windows logon screen. Keep it enabled — it protects against pre-auth resource exhaustion — and fix the client side instead. Only if you are sure the credential is correct should you consider disabling it server-side, and doing so should be temporary:

# On the guest, only as a diagnostic step
Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp' `
  -Name UserAuthentication -Value 0
# Re-enable afterwards (1 = required)
Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp' `
  -Name UserAuthentication -Value 1

The self-signed certificate warning is expected on any host that is not domain-joined or has no trusted certificate bound to the RDP listener. Check that the name or IP in the warning matches the machine you intend to reach, tick “Don’t ask me again”, and accept. The underlying thumbprint is inspectable if you want to pin it:

Get-ChildItem Cert:\LocalMachine\Remote Desktop | Select Subject,Thumbprint,NotAfter

Server-Side Checks When the Client Looks Fine

If the transport test succeeds but the session drops after a couple of seconds, run these on the guest. They cover the four server causes that account for nearly every “RDP connects then closes” report.

Get-Service TermService,UmRdpService | Select Name,Status,StartType
Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server' | Select fDenyTSConnections
Get-NetFirewallRule -DisplayGroup 'Remote Desktop' | Select DisplayName,Enabled,Profile
Get-LocalGroupMember 'Remote Desktop Users'
Get-WinEvent -LogName 'Microsoft-Windows-TerminalServices-LocalSessionManager/Operational' -MaxEvents 20 |
  Select TimeCreated,Id,Message

fDenyTSConnections = 1 means RDP is switched off at the OS level regardless of firewall rules. If the Remote Desktop firewall group shows Enabled: False for the Public profile, and the VPS NIC is classified as Public (common on cloud networks with no domain), the connection is dropped silently. Set the profile explicitly:

Set-NetConnectionProfile -InterfaceAlias 'Ethernet' -NetworkCategory Private
Enable-NetFirewallRule -DisplayGroup 'Remote Desktop'

Using the Windows 11 Remote Desktop App Instead

The Store app is a different codebase. It supports saved PC entries in a workspace, multi-monitor by default, and better scaling on high-DPI laptops, but it hides the low-level options — no /admin console session, no .rdp import, and it does not read Credential Manager entries created by cmdkey. It is the better client for daily use from a laptop with a 4K panel, and the worse client when you need to debug something.

Keep both: the app for routine work, mstsc for console access and diagnostics. A console session (mstsc /admin) bypasses the two-session limit and is the only way in when both administrative slots are occupied by disconnected sessions.

Hardening After the First Successful Login

The first login is also the moment to fix the things that make an RDP host a target. Move the listener off 3389 if your provider allows it, and restrict the port to your own address range rather than the world.

# Change the listener port
Set-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp' -Name PortNumber -Value 3390
New-NetFirewallRule -DisplayName 'RDP 3390 from office' -Direction Inbound -Protocol TCP `
  -LocalPort 3390 -RemoteAddress 198.51.100.0/24 -Action Allow
Disable-NetFirewallRule -DisplayName 'Remote Desktop - User Mode (TCP-In)'

Two follow-ups matter more than the port change: confirm the account lockout policy (a public 3389 listener absorbs thousands of password attempts per day — check Get-WinEvent -LogName Security -FilterXPath "*[System[(EventID=4625)]]" for the volume), and make sure Security log retention is large enough to keep the evidence. Both take ten minutes and eliminate the most common way a working RDP setup becomes an incident. The full set of supported Windows images and their default network behaviour is documented alongside the Windows VPS plans.