Migrating OpenRemote to a New AWS Instance — Need Advice

I’m currently in the process of restoring our OpenRemote deployment to a new AWS EC2 instance, following these steps:

  1. Pulled the latest OpenRemote code using the Quick Start guide
  2. Modified the docker-compose.yml to reflect the public IP address of the new instance.
  3. Followed the Backup/Restore OpenRemote DB procedure from the [Developer Guide: Useful Commands and Queries

I also updated the following values in docker-compose.yml to match the values used in the previous production system:

KEYCLOAK_ADMIN_PASSWORD
OR_ADMIN_PASSWORD

After restoring the database and starting the stack, the openremote-manager-1 container remains unhealthy and fails with the message:

‘dependency failed to start: container openremote-manager-1 is unhealthy’

I’d appreciate advice on the best practice for migrating OpenRemote to a new instance.

I have seen this error in the logs depending on the password settings in docker-compose.yml:
manager-1 | 2025-04-06 07:36:14.316 SEVERE [main ] org.openremote.container.Container : >>> Runtime container startup failed
manager-1 | java.io.IOException: Integrity check failed: java.security.UnrecoverableKeyException: Failed PKCS12 integrity checking
manager-1 | at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2162)
manager-1 | at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:228)
manager-1 | at java.base/java.security.KeyStore.load(KeyStore.java:1500)
manager-1 | at java.base/java.security.KeyStore.getInstance(KeyStore.java:1828)
manager-1 | at java.base/java.security.KeyStore.getInstance(KeyStore.java:1709)
manager-1 | at org.openremote.manager.security.KeyStoreServiceImpl.start(KeyStoreServiceImpl.java:107)
manager-1 | at org.openremote.container.Container.start(Container.java:179)
manager-1 | at org.openremote.container.Container.startBackground(Container.java:223)
manager-1 | at org.openremote.manager.Main.main(Main.java:36)
manager-1 | Caused by: java.security.UnrecoverableKeyException: Failed PKCS12 integrity checking
manager-1 | at java.base/sun.security.pkcs12.PKCS12KeyStore.lambda$engineLoad$2(PKCS12KeyStore.java:2156)
manager-1 | at java.base/sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run(PKCS12KeyStore.java:257)
manager-1 | at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2140)
manager-1 | … 8 more
manager-1 exited with code 0

Here is my docker-compose.yml:

OpenRemote v3

Profile that runs the stack by default on https://localhost using a self-signed SSL certificate,

but optionally on https://$OR_HOSTNAME with an auto generated SSL certificate from Letsencrypt.

It is configured to use the AWS logging driver.

volumes:
proxy-data:
manager-data:
postgresql-data:

services:

proxy:
image: openremote/proxy:${PROXY_VERSION:-latest}
restart: always
depends_on:
manager:
condition: service_healthy
ports:
- “80:80” # Needed for SSL generation using letsencrypt
- “${OR_SSL_PORT:-443}:443”
- “8883:8883”
- “127.0.0.1:8404:8404” # Localhost metrics access
volumes:
- proxy-data:/deployment
environment:
LE_EMAIL: ${OR_EMAIL_ADMIN:-}
DOMAINNAME: ${OR_HOSTNAME:-0.123.456.789}
DOMAINNAMES: ${OR_ADDITIONAL_HOSTNAMES:-}
# USE A CUSTOM PROXY CONFIG - COPY FROM https://raw.githubusercontent.com/openremote/proxy/main/haproxy.cfg
#HAPROXY_CONFIG: ‘/data/proxy/haproxy.cfg’

postgresql:
restart: always
image: openremote/postgresql:${POSTGRESQL_VERSION:-latest}
shm_size: 128mb
volumes:
- postgresql-data:/var/lib/postgresql/data
- manager-data:/storage

keycloak:
restart: always
image: openremote/keycloak:${KEYCLOAK_VERSION:-latest}
depends_on:
postgresql:
condition: service_healthy
volumes:
- ./deployment:/deployment
environment:
KEYCLOAK_ADMIN_PASSWORD: ${OR_ADMIN_PASSWORD:-password}
KC_HOSTNAME: ${OR_HOSTNAME:-0.123.456.789}
KC_HOSTNAME_PORT: ${OR_SSL_PORT:–1}

manager:
privileged: true
restart: always
image: openremote/manager:${MANAGER_VERSION:-latest}
depends_on:
keycloak:
condition: service_healthy
ports:
- “127.0.0.1:8405:8405” # Localhost metrics access
environment:
OR_SETUP_TYPE:
OR_ADMIN_PASSWORD: ${OR_ADMIN_PASSWORD:-password}
OR_SETUP_RUN_ON_RESTART:
OR_EMAIL_HOST:
OR_EMAIL_USER:
OR_EMAIL_PASSWORD:
OR_EMAIL_X_HEADERS:
OR_EMAIL_FROM:
OR_EMAIL_ADMIN:
OR_METRICS_ENABLED: ${OR_METRICS_ENABLED:-true}
OR_HOSTNAME: ${OR_HOSTNAME:-0.123.456.789}
OR_ADDITIONAL_HOSTNAMES:
OR_SSL_PORT: ${OR_SSL_PORT:–1}
OR_DEV_MODE: ${OR_DEV_MODE:-false}

  # The following variables will configure the demo
  OR_FORECAST_SOLAR_API_KEY:
  OR_OPEN_WEATHER_API_APP_ID:
  OR_SETUP_IMPORT_DEMO_AGENT_KNX:
  OR_SETUP_IMPORT_DEMO_AGENT_VELBUS:
volumes:
  - manager-data:/storage
  - ./deployment:/deployment

Hey @Clint ,

This happens because the KeystoreService doesn’t have the correct password to unlock the Keystore that OpenRemote creates. If you had changed the keystore’s password using OR_KEYSTORE_PASSWORD, use that.

Basically, the keystore’s password is going to be OR_KEYSTORE_PASSWORD, if that doesn’t exist then it’s going to be OR_ADMIN_PASSWORD, and if that also does not exist, it’s going to be secret. So the one that the Keystore uses should be the same as with the previous deployment.

If you haven’t used x509 authentication in the MQTT client, you can also delete the keystores in the deployment or tmp directory, they’re called client_keystore.p12 or client_truststore.p12. This will recreate the files.

Let me know if that works!

Thanks Panos

I deleted the following and it seemed to then work very quickly with all the containers healthy and I could log in.

rm /var/lib/docker/volumes/openremote_manager-data/_data/manager/keycloak-credentials.json
rm /var/lib/docker/volumes/openremote_manager-data/_data/keystores/client_keystore.p12
rm /var/lib/docker/volumes/openremote_manager-data/_data/keystores/client_truststore.p12

I then tried to repeat the process and was back to the same problem with container openremote-manager-1 waiting and then failing as unhealthy.
I have tried repeating this process a number of times since but no luck.
I am pretty sure I have the passwords in docker-compose.yml as per the previous deployment.

Any assistance much appreciated.

Hey @Clint ,

Not sure what you mean by repeating the process? Since you deleted the keystores, then they were recreated, and they currently use either OR_KEYSTORE_PASSWORD or OR_ADMIN_PASSWORD.

Hi @panosp

As per below, I seem to be getting through the credential validation process but the openremote-manager-1 container never gets to ‘healthy’ so that I can log in.

I can see rules etc in the logs being uotput to the screen but the manager container seems to be waiting on something.

manager-1 | 2025-04-07 07:18:53.448 INFO [main ] curity.keycloak.KeycloakIdentityProvider : No stored credentials so using OR_ADMIN_PASSWORD
manager-1 | 2025-04-07 07:18:53.454 INFO [main ] curity.keycloak.KeycloakIdentityProvider : Keycloak proxy URI set to: http://keycloak:8080/auth
manager-1 | 2025-04-07 07:18:53.454 INFO [main ] curity.keycloak.KeycloakIdentityProvider : Validating keycloak credentials
manager-1 | 2025-04-07 07:18:57.284 INFO [main ] curity.keycloak.KeycloakIdentityProvider : Credentials are valid
manager-1 | 2025-04-07 07:18:57.288 INFO [main ] curity.keycloak.KeycloakIdentityProvider : OR_ADMIN_PASSWORD credentials are valid so creating/recreating stored credentials
manager-1 | 2025-04-07 07:19:00.125 INFO [main ] curity.keycloak.KeycloakIdentityProvider : Stored credentials successfully generated so using them
manager-1 | 2025-04-07 07:19:00.127 INFO [main ] curity.keycloak.KeycloakIdentityProvider : Keycloak proxy URI set to: http://keycloak:8080/auth
manager-1 | 2025-04-07 07:19:00.130 INFO [main ] curity.keycloak.KeycloakIdentityProvider : Validating keycloak credentials
manager-1 | 2025-04-07 07:19:00.500 INFO [main ] curity.keycloak.KeycloakIdentityProvider : Credentials are valid