IIS SMTP Service on a Windows VPS: Smart Host Relay and Authentication

An application on a Windows VPS that calls SmtpClient.Send() and receives System.Net.Mail.SmtpException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.7.57 Client not authenticated to send mail is a solved problem — but only if the relay path is built correctly. Port 25 outbound is blocked by nearly every hosting provider and by most residential ISPs, so the local SMTP server has to forward through an authenticated smart host. This article covers configuring the IIS SMTP feature, the .NET side that calls it, and the error codes you will hit.

Two facts to establish before changing anything. First, the IIS SMTP feature stopped being installed by default after Windows Server 2012 — it is still shipped as a feature, but it is in maintenance mode and has no UI since Windows Server 2012 R2. Second, if your provider blocks outbound 25, no amount of local configuration solves delivery; you need port 587 or 465 to a smart host, or the provider’s own relay endpoint. Test with Test-NetConnection smtp.yourprovider.net -Port 587 before you write any configuration.

Installing the SMTP Server Feature

On Windows Server 2016 through 2025 the feature is installed with:

Install-WindowsFeature SMTP-Server -IncludeManagementTools

The management console (inetmgr → SMTP Server node) only appears after installing the IIS 6 Management Compatibility component, which is not installed by default. Install it explicitly if you want the GUI: Install-WindowsFeature Web-Metabase, Web-Lgcy-Mgmt-Console. Otherwise, everything below can be driven from C:\Windows\System32\inetsrv\MetaBase.xml or, more safely, from the System.Web.Mail style configuration via %SystemRoot%\System32\inetsrv\smtpsvc ADSI objects.

After installation, the service is SMTPSVC and the default drop directory is C:\inetpub\mailroot\Drop. Verify it is running with Get-Service SMTPSVC; if it is stopped, the failure mode in your application is a connection-refused on 127.0.0.1:25 rather than an authentication error, which is a useful distinction when triaging.

Configuring the Smart Host and Authentication

The default SMTP virtual server accepts mail from any client in the local subnet and attempts direct delivery on port 25. Both defaults are wrong for a hosted VPS. You want it to accept only from 127.0.0.1 and to forward everything to an authenticated smart host.

Through the IIS 6 console: right-click the SMTP virtual server, Properties, Access tab, Relay restrictions, select “Only the list below”, add 127.0.0.1, and tick “Allow all computers which successfully authenticate to relay”. On the Delivery tab, Outbound Security, choose Basic Authentication, enter the smart host credentials, and tick TLS encryption. Then Outbound Connections, set the TCP port to 587, and Advanced, set the smart host name.

Enabling TLS requires importing the smart host’s certificate chain into the local machine store, otherwise outbound connections fail with 454 4.7.0 Temporary authentication failure while the credentials are perfectly valid. Export the root and intermediate certificates from a working machine and import them into Cert:\LocalMachine\Root and Cert:\LocalMachine\CA — not into CurrentUser, because SMTPSVC runs as LocalSystem.

The .NET Side: Why SmtpClient Fails by Default

System.Net.Mail.SmtpClient still exists in .NET 8 and .NET 9 but is marked obsolete for new development — it is not obsoleted, just discouraged in favour of MailKit. If your application already uses it, the minimal working configuration against a local relay is:

var client = new SmtpClient("127.0.0.1", 25) { DeliveryMethod = SmtpDeliveryMethod.Network, EnableSsl = false, UseDefaultCredentials = false };

EnableSsl = false is correct here because TLS terminates at the local SMTP service, between it and the smart host. Setting EnableSsl = true while pointing at 127.0.0.1 on port 25 causes a handshake failure that surfaces as a timeout after 100 seconds. Likewise, leave credentials empty when relaying through the local service — supplying them makes SMTPSVC attempt to use them for outbound auth and can override the configured smart host credentials.

Error Codes and What They Mean

SMTP response codes are more specific than most tutorials admit, and the first three digits tell you which layer failed. Reading them correctly saves hours of guessing.

ResponseMeaningTypical fix
5.7.1 Client host rejectedSmart host does not accept your server IPRegister the VPS IP with the provider, or authenticate
5.7.57 Client not authenticatedCredentials missing or rejectedRecheck Outbound Security settings and app password
550 5.7.1 Relaying deniedRecipient not in allowed domainConfirm the smart host authorises relay for that domain
421 4.7.0 Connection refused by remoteProvider blocks outbound port 25Switch outbound port to 587 with STARTTLS
452 4.2.2 Mailbox fullRecipient-side quotaNot your problem — retry later, honour the bounce

Enable SMTP protocol logging on the virtual server to capture the full exchange: C:\Windows\System32\LogFiles\SMTPSVC1\ holds per-day text logs with the client IP, the command sequence, and the response codes. This is the fastest way to prove whether the failure is local or remote.

SPF, DKIM, and Why Mail Lands in Spam

Delivery succeeding is not the same as delivery being accepted. A VPS IP with no sending history has poor reputation, and a message without SPF alignment is very likely to be filtered. Add an SPF record to the sending domain’s DNS that includes the smart host’s sending infrastructure, and use a subdomain such as mail.example.com for envelope-from addresses so a reputation problem never affects your primary domain.

DKIM signing is not something the IIS SMTP service does itself. Either let the smart host sign on your behalf — most transactional providers do, as long as the MAIL FROM domain is verified — or use a relay service that returns a DKIM key to publish in DNS. If you must sign locally, the SMTP service is the wrong tool and you should send directly from the application through a library like MailKit with DkimSigner.

Check your alignment before blaming the code. nslookup -type=txt example.com 8.8.8.8 should show the SPF record, and a test message to a mailbox you control should carry Authentication-Results headers listing spf=pass and dkim=pass. Failures here are a DNS problem, not an SMTP problem.

Pickup Directory Mode for Offline or Batched Sending

When the application generates mail in bursts — invoice runs, report exports — sending synchronously through SMTP adds latency to the request. Configure the SMTP service to deliver from a pickup directory instead: set the SMTP server’s mode to “Pickup directory from which to send” and point the application at C:\inetpub\mailroot\Pickup. Writing a properly formatted .eml file into that folder is a single file operation, and the service handles retries and batching on its own schedule.

The trade-off is that a bad message sits in mailroot\Queue with a corresponding .bad file, and nobody notices. Monitor queue depth with Get-ChildItem C:\inetpub\mailroot\Queue -Filter *.eml | Measure-Object inside an existing health check, and alert when it exceeds zero for more than ten minutes.

Replacing SMTP Service Entirely: The Modern Path

For new applications, the honest recommendation is to skip the local SMTP service. Send over HTTPS to a transactional email API, or use MailKit to talk directly to the provider’s 587 endpoint with SmtpClient.Connect("smtp.provider.net", 587, SecureSocketOptions.StartTls). This removes an entire locally managed service, works when outbound 25 is blocked, and gives you structured delivery logs from the provider rather than text files on disk.

The one case that still justifies the local relay is a legacy application with a hard-coded localhost:25 that cannot be changed. In that situation the IIS SMTP feature is a lightweight shim, and the configuration above is the minimum safe version of it.

Checklist Before Going Live

  • Outbound 587 confirmed reachable from the VPS with Test-NetConnection.
  • Relay restriction limited to 127.0.0.1; open relay verified closed with a test from an external host (expect 550 5.7.1).
  • Smart host certificate chain present in Cert:\LocalMachine\Root.
  • SPF record published, DKIM signing in place at the relay.
  • Queue depth and Drop folder monitored by an existing scheduled task.
  • Application configured with EnableSsl = false for localhost and no credentials.

Outbound mail is one of the few services where a misconfiguration is invisible until a customer says they never received the invoice. Our Windows VPS plans leave outbound 587 and 465 open by default, as the article on Windows VPS port blocking rules explains.

Leave a Comment