Haproxy in docker stack won't start

Custom Deployment
1

I wanted to spin up a local test instance of OpenRemote today. I was following the quickstart from GitHub - openremote/openremote: 100% open-source IoT Platform - Integrate your devices, create rules, and analyse and visualise your data using this (unmodified) docker-compose.yml: https://raw.githubusercontent.com/openremote/openremote/master/docker-compose.yml

I tried this on several hosts, in our local lab and a datacenter. All were running some kind of RHEL-ish distro, most AlmaLinux 8 oder 9.
No other applications / containers are running on those machines.

The haproxy from the docker stack is stuck in a start loop with the following log entries:

openremote-proxy-1       | [INFO][2024-01-11 13:27:34] HAPROXY_USER_PARAMS:
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] PROXY_LOGLEVEL: info
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] LUA_PATH:
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] CERT_DIR: /deployment/certs
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] LE_DIR: /deployment/letsencrypt
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] Checking HAProxy configuration: /etc/haproxy/haproxy.cfg
openremote-proxy-1       | Configuration file is valid
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] Starting crond
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] cert_init...waiting 10s for haproxy to be ready
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] Starting monitoring process
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] HAProxy starting
openremote-proxy-1       | [INFO][2024-01-11 13:27:34] Monitoring config file '/etc/haproxy/haproxy.cfg' and certs in '/deployment/certs' for changes...
openremote-proxy-1       | [NOTICE]   (1) : haproxy version is 2.7.8-58c657f
openremote-proxy-1       | [ALERT]    (1) : Not enough memory to allocate 1073741815 entries for fdtab!
openremote-proxy-1       | [ALERT]    (1) : No polling mechanism available.
openremote-proxy-1       |   This may happen when using thread-groups with old pollers (poll/select), or
openremote-proxy-1       |   it is possible that haproxy was built with TARGET=generic and that FD_SETSIZE
openremote-proxy-1       |   is too low on this platform to support maxconn and the number of listeners
openremote-proxy-1       |   and servers. You should rebuild haproxy specifying your system using TARGET=
openremote-proxy-1       |   in order to support other polling systems (poll, epoll, kqueue) or reduce the
openremote-proxy-1       |   global maxconn setting to accommodate the system's limitation. For reference,
openremote-proxy-1       |   FD_SETSIZE=1024 on this system, global.maxconn=536870889 resulting in a maximum of
openremote-proxy-1       |   1073741815 file descriptors. You should thus reduce global.maxconn by 536870396. Also,
openremote-proxy-1       |   check build settings using 'haproxy -vv'.
openremote-proxy-1       |
openremote-proxy-1 exited with code 1

Has anybody seen this and has some quick pointers as to where / how to fix this?
Or do I have to do some deep diving myself?

Best regards,
Christian

2

Oh, and I forgot to mention this before: As there’s the obvious line “Not enough memory” - all test machines had at least 32 GB of RAM free at the time of testing. I suppose this should be enough for a simple test setup?

Christian

3

Looks like this is causing the issue;

Seems like adding maxconn to the HAProxy config is necessary.
@Rich any idea or suggestions?

4

Yeah, that looks like it. Docker is on version 24 on all our systems.
I suppose there’s no easy way to change the haproxy config with rebuilding the docker image myself?

5

As mentioned in the docker-compose.yml file you sent,
you can provide a custom config usingthe HAPROXY_CONFIG environment variable. :wink:
The config file we use can be found here.

Our proxy image is available on GitHub here, but it’s basically only a wrapper around HAProxy.
Feel free to use a different proxy, or use the repo and build a local image if you need to.

6

Hi Martin,

thanks for taking your time to answer here. I loaded a custom config with a global maxconn for haproxy, which let’s it start correctly.
After that I cannot even get any webpage, I just get an empty page. dev console shows requests for manager_config.json, keycloak.min.js, or.json run into a 404. A request for /register gives a 403.

I found some posts about putting a default manager_config.json to the deployment folder, but that doesn’t get picked up there as it seems (still 404).

All in all it seems that’s not a way to quickly fire up a basic local test instance of OpenRemote (which is what I wanted to to basically, while fiddling around with some ideos for usage).

I’ll have to drop it here and perhaps revisit this if I can make some more free time to experiment.

Thanks for now and have a nice weekend!
Christian

1 Like
7

Thanks for bringing this to our attention. I have now created a new openremote/proxy:latest image which should solve the problem, just pull this latest image:

docker compose -p or pull proxy

NOTE: if you are using a custom haproxy.cfg then you might need to make changes as haproxy seems to have made breaking changes to how they process " within redirect statements.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.