TLS everything on MariaDB Galera Cluster

A short how-to secure all traffic inside a galera cluster. This will consist of securing the mysql/3306, galera/4567-4568 and my non-default port SST/4569 (default 4444). CentOS 8.2.2004 with MariaDB 10.5.5 was used but these steps should apply to most other distros as well.

Some prep before starting. To make sure unavailable DNS servers don't break the cluster, as suggested by RHEL knowledgebase on "How should the /etc/hosts file be set up on RHEL cluster nodes?", add all the nodes to each others /etc/hosts file.

10.0.0.81 node1.muhnetwork
10.0.0.82 node2.muhnetwork

As for the certificates. In the 3306-world host verification looks like to a mixed bag of using CommonName(CN) field and/or SAN (X509v3 Subject Alternative Name) IP/DNS fields. For example 10.3 client on Debian didn't match CN for hostname verification, 10.5 client on CentOS 8 matched CN and SAN, Galera SST doesn't match SAN IP Address fields. As everything seems to use SAN DNS fields, it's the best bet. I made a single certificate for both nodes with SAN as following.

X509v3 Subject Alternative Name: 
  IP Address:10.0.0.81, IP Address:10.0.0.82, DNS:node2.muhnetwork, DNS:node1.muhnetwork

CA and server certificates and key files are copied to each node into /etc/my.cnf.d/tls/ directory. Key files must be readable by the database process. SELinux wise /etc/my.cnf.d/ has it's own label and therefore smaller set of possible source labels accessing it compared to /etc/pki/tls/.

Securing mysql/3306

Smack this into yer config! But do ensure client combatibility for TLSv1.2. Also we're going to reject non-TLS connections with require_secure_transport which was added in MariaDB version 10.5.2.

[server]
ssl_cert = /etc/my.cnf.d/tls/node1.muhnetwork.crt
ssl_key = /etc/my.cnf.d/tls/node1.muhnetwork.key
ssl_ca = /etc/my.cnf.d/tls/ca.crt

ssl_cipher = TLSv1.2
require_secure_transport = 1

For client configuration to verify the server hostname/IP address and certificate.

[client]
ssl_ca = /path/to/the/ca.crt
ssl-verify-server-cert

Or while launching client

mysql -h node1.muhnetwork --ssl-ca=/path/to/the/ca.crt --ssl-verify-server-cert

Securing galera/4567-4568

SSL configuration is set in the wsrep_provider_options parameter.

[galera]
wsrep_provider_options = "socket.ssl_key=/etc/my.cnf.d/tls/node1.muhnetwork.key;socket.ssl_cert=/etc/my.cnf.d/tls/node1.muhnetwork.crt;socket.ssl_ca=/etc/my.cnf.d/tls/ca.crt"

Which is a beauty as it doesn't allow newline within the value. :) And the logs should contain "WSREP: SSL handshake succesful"

WSREP: SSL handshake successful, remote endpoint ssl://10.0.0.82:4567 local endpoint ssl://10.0.0.82:58132 cipher: TLS_AES_256_GCM_SHA384 compression: none

Galera IST uses the same configuration as shown in the the logs.

WSREP: IST receiver addr using ssl://10.0.0.81:4568
WSREP: IST receiver using ssl

Securing galera-sst/4569

Add a new block for SST in your server configuration. encrypt 3 with tca ensure the hostname verification.

[sst]
encrypt = 3
tcert = /etc/my.cnf.d/tls/node1.muhnetwork.crt
tkey = /etc/my.cnf.d/tls/node1.muhnetwork.key
tca = /etc/my.cnf.d/tls/ca.crt

And announce receiver address with a hostname included in the certificate.

[server]
wsrep_sst_receive_address = "node1.muhnetwork:4569"

openssl-connect should be visible on your logs at the donor:

WSREP_SST: [INFO] Evaluating /usr/bin/mariabackup  --backup --no-version-check  $tmpopts $INNOEXTRA --galera-info --stream=$sfmt --target-dir=$itmpdir --mysqld-args $WSREP_SST_OPT_MYSQLD 2> /var/lib/mysql//mariabackup.backup.log | socat -u stdio openssl-connect:node2.muhnetwork:4569,cert=/etc/my.cnf.d/tls/node1.muhnetwork.crt,key=/etc/my.cnf.d/tls/node1.muhnetwork.key,cafile=/etc/my.cnf.d/tls/ca.crt;

and openssl-listen on the receiver:

WSREP_SST: [INFO] Evaluating socat -u openssl-listen:4569,reuseaddr,cert=/etc/my.cnf.d/tls/node2.muhnetwork.crt,key=/etc/my.cnf.d/tls/node2.muhnetwork.key,cafile=/etc/my.cnf.d/tls/ca.crt stdio | mbstream -x; RC=( ${PIPESTATUS[@]} ) (20200827 15:31:26.943)

Possible errors

Certificate or key files are not readable or missing for SST. Verify db process has access (file permissions, owner) and selinux labeling in case files have been moved around.

socat[1986] E SSL_CTX_use_certificate_file(): error:02001002:system library:fopen:No such file or directory

Certificate verification fails during SST. In this case verify the wsrep_receive_address matches either CN or DNS field of SAN of the certificate.

socat: E certificate is valid but its commonName does not match hostname

Happy clustering!