Problems renewing Let's Encrypt certificates

Custom Deployment
1

Hi guys,
suddenly I have no access to Openremote and it is because the Let’s Encrypt certificates have not been renewed. I’ve already checked that I have port 80 open.
I also restarted the proxy container (even the server), and looking at the log I can see the messages below.
Any ideas? I need access to the application urgently.

Alternatively, let’s say I have the renewed .pem files, where do they go?
Any help is appreciated.

2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] DOMAINNAMES: example.com
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] HAPROXY_CONFIG: /etc/haproxy/haproxy.cfg
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] HAPROXY_CMD: haproxy -W -db -f /etc/haproxy/haproxy.cfg 
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] HAPROXY_USER_PARAMS: 
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] PROXY_LOGLEVEL: info
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] LUA_PATH: 
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] CERT_DIR: /deployment/certs
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] LE_DIR: /deployment/letsencrypt
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] Checking HAProxy configuration: /etc/haproxy/haproxy.cfg
2025-04-01 10:36:15 Configuration file is valid
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] Starting crond
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] Starting monitoring process
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] cert_init...waiting 10s for haproxy to be ready
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] HAProxy starting
2025-04-01 10:36:15 [INFO][2025-04-01 08:36:15] Monitoring config file '/etc/haproxy/haproxy.cfg' and certs in '/deployment/certs' for changes...
2025-04-01 10:36:15 [acme] http-01 plugin v0.1.1
2025-04-01 10:36:15 keycloak_backend/keycloak changed its IP from (none) to 172.18.0.2 by docker_resolver/dns.
2025-04-01 10:36:15 Server keycloak_backend/keycloak ('keycloak') is UP/READY (resolves again).
2025-04-01 10:36:15 Server keycloak_backend/keycloak administratively READY thanks to valid DNS answer.
2025-04-01 10:36:15 mqtt/manager changed its IP from (none) to 172.18.0.5 by docker_resolver/dns.
2025-04-01 10:36:15 Server mqtt/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:15 Server mqtt/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:15 manager_backend/manager changed its IP from (none) to 172.18.0.5 by DNS cache.
2025-04-01 10:36:15 Server manager_backend/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:15 Server manager_backend/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:05 [WARNING]  (1) : Exiting Master process...
2025-04-01 10:36:05 [WARNING]  (105) : Proxy stats stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:05 [WARNING]  (105) : Proxy http stopped (cumulated conns: FE: 86409, BE: 0).
2025-04-01 10:36:05 [WARNING]  (105) : Proxy https stopped (cumulated conns: FE: 186429, BE: 0).
2025-04-01 10:36:05 [WARNING]  (105) : Proxy mqtt stopped (cumulated conns: FE: 2231, BE: 502).
2025-04-01 10:36:05 [WARNING]  (105) : Proxy manager_backend stopped (cumulated conns: FE: 0, BE: 154514).
2025-04-01 10:36:05 [WARNING]  (105) : Proxy keycloak_backend stopped (cumulated conns: FE: 0, BE: 182450).
2025-04-01 10:36:15 [info] 090/083615 (1) : [acme] http-01 plugin v0.1.1
2025-04-01 10:36:15 [NOTICE]   (1) : New worker (31) forked
2025-04-01 10:36:15 [NOTICE]   (1) : Loading success.
2025-04-01 10:36:15 [WARNING]  (31) : keycloak_backend/keycloak changed its IP from (none) to 172.18.0.2 by docker_resolver/dns.
2025-04-01 10:36:15 [WARNING]  (31) : Server keycloak_backend/keycloak ('keycloak') is UP/READY (resolves again).
2025-04-01 10:36:15 [WARNING]  (31) : Server keycloak_backend/keycloak administratively READY thanks to valid DNS answer.
2025-04-01 10:36:15 [WARNING]  (31) : mqtt/manager changed its IP from (none) to 172.18.0.5 by docker_resolver/dns.
2025-04-01 10:36:15 [WARNING]  (31) : Server mqtt/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:15 [WARNING]  (31) : Server mqtt/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:15 [WARNING]  (31) : manager_backend/manager changed its IP from (none) to 172.18.0.5 by DNS cache.
2025-04-01 10:36:15 [WARNING]  (31) : Server manager_backend/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:15 [WARNING]  (31) : Server manager_backend/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:18 172.18.0.1:56434 [01/Apr/2025:08:36:18.907] https/1: SSL handshake failure (error:0A000416:SSL routines::sslv3 alert certificate unknown)
2025-04-01 10:36:25 [INFO][2025-04-01 08:36:25] Executing cert_init at Tue, 01 Apr 2025 08:36:25 +0000
2025-04-01 10:36:25 [INFO][2025-04-01 08:36:25] Symlinking first domain to built in cert directory to take precedence over self signed cert
2025-04-01 10:36:25 [INFO][2025-04-01 08:36:25] Updating haproxy cert chain for 'example.com'
2025-04-01 10:36:25 /deployment/certs/ DELETE example.com
2025-04-01 10:36:25 [INFO][2025-04-01 08:36:25] Executing auto renew at Tue, 01 Apr 2025 08:36:25 +0000
2025-04-01 10:36:25 [INFO][2025-04-01 08:36:25] Change detected...
2025-04-01 10:36:25 172.18.0.1:56440 [01/Apr/2025:08:36:25.918] https/1: SSL handshake failure (error:0A000416:SSL routines::sslv3 alert certificate unknown)
2025-04-01 10:36:25 Saving debug log to /var/log/letsencrypt/letsencrypt.log
2025-04-01 10:36:25 
2025-04-01 10:36:25 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-01 10:36:25 Processing /deployment/letsencrypt/renewal/example.com.conf
2025-04-01 10:36:25 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-01 10:36:27 172.18.0.1:56446 [01/Apr/2025:08:36:27.915] https/1: SSL handshake failure (error:0A000416:SSL routines::sslv3 alert certificate unknown)
2025-04-01 10:36:30 [INFO][2025-04-01 08:36:30] HAProxy restart required...
2025-04-01 10:36:30 [INFO][2025-04-01 08:36:30] Checking HAProxy configuration: /etc/haproxy/haproxy.cfg
2025-04-01 10:36:30 Configuration file is valid
2025-04-01 10:36:30 [INFO][2025-04-01 08:36:30] Config is valid so requesting restart...
2025-04-01 10:36:30 [NOTICE]   (1) : Reloading HAProxy
2025-04-01 10:36:30 [INFO][2025-04-01 08:36:30] Monitoring config file '/etc/haproxy/haproxy.cfg' and certs in '/deployment/certs' for changes...
2025-04-01 10:36:30 [acme] http-01 plugin v0.1.1
2025-04-01 10:36:30 [info] 090/083630 (1) : [acme] http-01 plugin v0.1.1
2025-04-01 10:36:30 Proxy stats stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 Proxy http stopped (cumulated conns: FE: 1, BE: 0).
2025-04-01 10:36:30 Proxy https stopped (cumulated conns: FE: 3, BE: 0).
2025-04-01 10:36:30 Proxy mqtt stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 Proxy manager_backend stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 Proxy keycloak_backend stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 [WARNING]  (31) : Proxy stats stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 [WARNING]  (31) : Proxy http stopped (cumulated conns: FE: 1, BE: 0).
2025-04-01 10:36:30 [WARNING]  (31) : Proxy https stopped (cumulated conns: FE: 3, BE: 0).
2025-04-01 10:36:30 [WARNING]  (31) : Proxy mqtt stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 [WARNING]  (31) : Proxy manager_backend stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 [WARNING]  (31) : Proxy keycloak_backend stopped (cumulated conns: FE: 0, BE: 0).
2025-04-01 10:36:30 [NOTICE]   (1) : New worker (100) forked
2025-04-01 10:36:30 [NOTICE]   (1) : Loading success.
2025-04-01 10:36:30 [NOTICE]   (1) : haproxy version is 2.8.5-aaba8d0
2025-04-01 10:36:30 [WARNING]  (1) : Former worker (31) exited with code 0 (Exit)
2025-04-01 10:36:31 [WARNING]  (100) : keycloak_backend/keycloak changed its IP from (none) to 172.18.0.2 by docker_resolver/dns.
2025-04-01 10:36:31 [WARNING]  (100) : Server keycloak_backend/keycloak ('keycloak') is UP/READY (resolves again).
2025-04-01 10:36:31 [WARNING]  (100) : Server keycloak_backend/keycloak administratively READY thanks to valid DNS answer.
2025-04-01 10:36:31 [WARNING]  (100) : mqtt/manager changed its IP from (none) to 172.18.0.5 by docker_resolver/dns.
2025-04-01 10:36:31 [WARNING]  (100) : Server mqtt/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:31 [WARNING]  (100) : Server mqtt/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:31 [WARNING]  (100) : manager_backend/manager changed its IP from (none) to 172.18.0.5 by DNS cache.
2025-04-01 10:36:31 [WARNING]  (100) : Server manager_backend/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:31 [WARNING]  (100) : Server manager_backend/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:31 keycloak_backend/keycloak changed its IP from (none) to 172.18.0.2 by docker_resolver/dns.
2025-04-01 10:36:31 Server keycloak_backend/keycloak ('keycloak') is UP/READY (resolves again).
2025-04-01 10:36:31 Server keycloak_backend/keycloak administratively READY thanks to valid DNS answer.
2025-04-01 10:36:31 mqtt/manager changed its IP from (none) to 172.18.0.5 by docker_resolver/dns.
2025-04-01 10:36:31 Server mqtt/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:31 Server mqtt/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:31 manager_backend/manager changed its IP from (none) to 172.18.0.5 by DNS cache.
2025-04-01 10:36:31 Server manager_backend/manager ('manager') is UP/READY (resolves again).
2025-04-01 10:36:31 Server manager_backend/manager administratively READY thanks to valid DNS answer.
2025-04-01 10:36:35 172.18.0.1:58122 [01/Apr/2025:08:36:35.921] https/1: SSL handshake failure (error:0A000416:SSL routines::sslv3 alert certificate unknown)

2025-04-01 10:42:54 Renewing an existing certificate for example.com
2025-04-01 10:42:56 
2025-04-01 10:42:56 Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2025-04-01 10:42:56   Domain: example.com
2025-04-01 10:42:56   Type:   unauthorized
2025-04-01 10:42:56   Detail: xx.xx.xx.xx: Invalid response from http://example.com/.well-known/acme-challenge/o33zeEO7niVpSGhWWtAo6eG4K9vWa6eSZh8d3xTWvGU: 404
2025-04-01 10:42:56 
2025-04-01 10:42:56 Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2025-04-01 10:42:56 
2025-04-01 10:42:56 
2025-04-01 10:42:56 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-01 10:42:56 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-01 10:42:58 01/Apr/2025:08:42:58 +0000 https~ 172.18.0.1:60230 <NOSRV> -1/-1/-1/-1/1 "<BADREQ>" 0 2/2/0/0/0 0/0


2025-04-01 14:06:37 Renewing an existing certificate for example.com
2025-04-01 14:06:39 
2025-04-01 14:06:39 Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2025-04-01 14:06:39   Domain: example.com
2025-04-01 14:06:39   Type:   unauthorized
2025-04-01 14:06:39   Detail: xx.xx.xx.xx: Invalid response from http://example.com/.well-known/acme-challenge/CThWqVNNrTFi2j8tfRzy1G2a8SUAEG3bwi-n-bAmZQI: 404
2025-04-01 14:06:39 
2025-04-01 14:06:39 Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2025-04-01 14:06:39 
2025-04-01 14:06:39 Failed to renew certificate example.com with error: Some challenges have failed.
2025-04-01 14:06:39 
2025-04-01 14:06:39 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-01 14:06:39 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2025-04-01 14:06:39 All renewals failed. The following certificates could not be renewed:
2025-04-01 14:06:39   /deployment/letsencrypt/live/example.com/fullchain.pem (failure)
2025-04-01 14:06:39 1 renew failure(s), 0 parse failure(s)
2025-04-01 14:06:39 Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
2025-04-01 14:06:39 [WARNING]  (1) : Process 23 exited with code 0 (Exit)
2025-04-01 14:06:56 172.18.0.1:59776 [01/Apr/2025:12:06:56.860] https/1: SSL handshake failure (error:0A000416:SSL routines::sslv3 alert certificate unknown)
2

Hi,

Looks like you’re trying to generate certs for example.com which I assume is incorrect. Ensure you have the correct DOMAINNAME and/or DOMAINNAMES set on your proxy container.

You should always be able to accept the unsafe cert and proceed to the app in your browser, you cannot do this?

3

Hi Rich.
No, I changed the real url to ‘example.com’ just for this post. In fact, the application has been running for months until now.
Regarding the unsafe cert, I can’t, I have a notification like this:
"example.com" has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site.

4

OK that makes sense, worth checking the obvious.

I guess the proxy container won’t start due to this cert renewal issue; you could try setting DOMAINNAME to localhost so it uses the built in self signed cert to get it running at least and then you can at least use the system.

5

Feel free to email me your real domain name and we can check accessibility.

1 Like
6

Just to be sure: are the DNS settings correct for the url you’re using? Does the A record point to your server’s IP address?

7

HSTS can be disabled for certain websites in your browser unless it is a .app domain then it is always enabled in most browsers.

8

Honestly, I don’t know how to check that, the application was working fine until the certificate expired. In fact, I thought I had been closed to the port 80 needed for renewal, but it was not.

9

Yes, for a while I tried to do it, but I think it is pointless to make the effort as other project partners are accessing data via the API and HTTPS.

10

OK, it works now, I don’t know exactly what the solution was as I was touching several things. I have a suspicion though, maybe another application was using port 80 needed for Let’s Encrypt renewal. We’ll see next time… anyway, thanks for the ideas provided, I now have a better understanding of how OR works, thanks a million guys