Securing TCP traffic and certificate pinning?

We develop a Java application that serves requests over TCP. We also develop client libraries for the application, where each library supports a different language/platform (Java, .NET, etc.).

Until recently, TCP traffic was confined to a secure network. To support usage on an insecure network we implemented TLS using the recipes in java-plain-and-tls-socket-examples. There are recipes here for both server and client, and a script to generate an X.509 certificate. Below is a summary of the recipe that we are using for TLS with server-only authentication:

• Create an X.509 root certificate that is self-signed.
• Configure the server with a keystore file that contains the certificate's identifying data plus the public and private keys.
• Configure the client with a trust store file that contains the same identifying data plus only the public key.

This looks like a setup for certificate pinning, as the client has a copy of the server certificate before connecting. Apparently, when connecting, the client will use this data somehow to validate the certificate that the server sends.

We assume for now that this approach is valid for securing TCP traffic. Signing by a certificate authority seems unnecessary because we control both server and client.

Initial testing shows that the implementation is working in our Java server and Java client (both running locally):

• Client accepts a server certificate that matches data in the client's trust store.
• Client rejects a server certificate that does not matches data in the client's trust store.
• tcpdump shows that TCP packets contain encrypted data.

.NET Client

We use SslStream to encrypt TCP traffic. As the documentation suggests, we do not specify a TLS version; instead we throw an exception if the version is below 1.2.

We're not confident about how to use X509Chain.ChainPolicy.CustomTrustStore correctly, because the documentation omits information like use cases for this type, and for option types like X509KeyStorageFlags and X509VerificationFlags.

The code below aims to mimic the recipe outlined above, i.e. configure a trust store data structure for the client to use when validating a server certificate. This approach seems equivalent to importing the certificate into the operating system's trust store.

// Import the trust store.
private X509Certificate2Collection GetCertificates(string storePath, string storePassword)
{
    byte[] bytes = File.ReadAllBytes(storePath);

    var result = new X509Certificate2Collection();
    result.Import(bytes, storePassword, X509KeyStorageFlags.EphemeralKeySet);
    return result;
}

// Callback function to validate a certificate received from the server.
// fCertificates stores the result of function GetCertificates.
private bool ValidateServerCertificate(
    object sender,
    X509Certificate certificate,
    X509Chain chain,
    SslPolicyErrors sslPolicyErrors)
{
    // Do not allow this client to communicate with unauthenticated servers.
    //
    // With a self-signed certficate, sslPolicyErrors should be always equal to
    // SslPolicyErrors.RemoteCertificateChainErrors.
    var result = (SslPolicyErrors.RemoteCertificateChainErrors == sslPolicyErrors);
    if (result)
    {
        // The values below are default values: set them to be explicit.
        chain.ChainPolicy.VerificationFlags = X509VerificationFlags.NoFlag;
        chain.ChainPolicy.RevocationMode = X509RevocationMode.NoCheck;
    
        chain.ChainPolicy.TrustMode = X509ChainTrustMode.CustomRootTrust;
        chain.ChainPolicy.CustomTrustStore.AddRange(fCertificates);
        result = chain.Build((X509Certificate2)certificate);
    }

    return result;
}

// Initialize SslStream.
private SslStream GetStream(TcpClient tcpClient, string targetHost)
{
    SslStream sslStream = new SslStream(
        tcpClient.GetStream(),
        false,
        new RemoteCertificateValidationCallback(ValidateServerCertificate),
        null
    );

    try
    {
        sslStream.AuthenticateAsClient(targetHost);

        // require TLS 1.2 or higher
        if (sslStream.SslProtocol < SslProtocols.Tls12)
        {
            throw new AuthenticationException($"The SSL protocol ({sslStream.SslProtocol}) must be {SslProtocols.Tls12} or higher.");
        }
    }
    catch (AuthenticationException caught)
    {
        sslStream.Dispose();
        throw caught;
    }

    return sslStream;
}

Initial testing has yielded results that vary depending on the operating system:

• .NET 6 ASP.NET web client on Ubuntu on WSL2:
  • Accepts a valid server certificate:
    • The callback function argument sslPolicyErrors is equal to SslPolicyErrors.RemoteCertificateChainErrors (expected).
    • Encrypts TCP packets (verified using tcpdump).
    • Automatically uses TLS 1.3.
  • Rejects an invalid server certificate: X509Chain.Build returns false and the only chain status item is "UntrustedRoot, self signed certificate".
• .NET 6 MAUI client on MacOS:
  1. Rejects a valid server certificate: the callback function argument sslPolicyErrors includes these values:
     • SslPolicyErrors.RemoteCertificateNameMismatch (unexpected).
     • SslPolicyErrors.RemoteCertificateChainErrors (expected).
  2. If we change the code to ignore sslPolicyErrors then it:
     • Accepts a valid server certificate and automatically uses TLS 1.2.
     • Rejects an invalid server certificate (as above).

Questions

1. In what ways, if any, could security be compromised with this .NET code?
2. Upon reviewing discussions about "certificate name mismatch" (see SslPolicyErrors.RemoteCertificateNameMismatch above), it seems that our server certificate should include a subjectAltName field to specify allowed DNS names. Is this necessary, or would it be reasonable, as we are using certificate pinning, to ignore sslPolicyErrors when validating the server certificate?

I can't answer your specific questions but here is some thoughts:

You have not mentioned anything about how the server authenticates clients. So you might consider implementing something like client certificates. If you control both you probably want some way to ensure random attackers cannot connect.

You might consider creating a threat model. In many cases it is the things that you haven't thought about that cause problems.

• If you are handling national security data or financial data you might want an external audit. Such might even be required in some cases.
• If there is no way an attacker could sell, use, or ransom the data, then you will probably not be directly targeted. So you might worry more about mass attacks against known vulnerabilities, i.e. keep all your software up to date.
• Consider other ways to mitigate risks. Are your server/client running in least privilege possible? Are you using a DMZ? Are firewalls correctly configured? Are credentials for support etc well managed?