Deploy OpenRemote 3 stack on AWS with CloudFormation

bobster888 · July 24, 2022, 3:39pm

OK … after much climbing the AWS, EC2, IAM etc curve and stumbling in the dark, I have tried to follow the instructions as previously kindly provided but unfortunately and numerous attempts unable to deploy OR so far …

17422 / 17422ERROR:root:stderr: b"\nAn error occurred (ValidationError) when calling the CreateStack operation: AccessDenied. User doesn’t have permission to call ssm:GetParameters\n"

Any clues as to where I’m going wrong very appreciated.

I’m not convinced I have done the following correctly …

Is this my ‘instance’ key pair?

cheers

bobster888 · July 24, 2022, 4:39pm

Quick update … I redid user adding all the permissions listed in or guide and this thread, IAM, EC2, SNS etc … and tried again and got the following …

[ec2-user@ip-172-31-90-79 ~]$ openremote-cli deploy --provider aws --dnsname host.domain --region us-east-1
To see commands use -v switch (-vvv for debug)

Deploying OR… This usually takes less than 15 minutes.

Generated password: 1JUDxBMU

100% […] 17422 / 17422
{

“StackId”: “arn:aws:cloudformation:us-east-1:946269502467:stack/host-e04c24ae-d942-42c7-9b75-481d78c5924b/267d08f0-0b6e-11ed-92f6-0a7e2d21602f”*
}

Waiting for CloudFormation…

After approx. 10mins it fails … with following errors …

ERROR:root:stderr: b’\nWaiter StackCreateComplete failed: Waiter encountered a terminal failure state: For expression “Stacks[].StackStatus” we matched expected path: “CREATE_FAILED” at least once\n’
ERROR:root:Exiting main because of uncached: (255, ‘\nWaiter StackCreateComplete failed: Waiter encountered a terminal failure state: For expression “Stacks[].StackStatus” we matched expected path: “CREATE_FAILED” at least once\n’)
Traceback (most recent call last):

File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/cli.py”, line 624, in main*
OpenRemote(sys.argv[1:], parser=parser)*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/cli.py”, line 71, in init*
getattr(self, command)(arguments)*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/cli.py”, line 161, in deploy*
args.password, args.dnsname, args.region*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/scripts.py”, line 199, in deploy_aws*
f’aws cloudformation wait stack-create-complete '*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/shell.py”, line 37, in execute*
output.returncode, output.stdout.decode(‘utf-8’)*
Exception: (255, ‘\nWaiter StackCreateComplete failed: Waiter encountered a terminal failure state: For expression “Stacks[].StackStatus” we matched expected path: “CREATE_FAILED” at least once\n’)
[ec2-user@ip-172-31-90-79 ~]$

Does this help?

michal · July 25, 2022, 7:18am

This is probably point where there are problems. You should own domain and add hosted zone in route 53 for it. The cloud formation tries insert A record with host to it. For example we own openremote.io domain and I’ve created Route 53 hosted zone mvp.openremote.io on our AWS account. Then I use -dnsname test.mvp.openremote.io and after creation I can access the openremote host on https://test.mvp.openremote.io.

michal · July 25, 2022, 8:40am

Secondly, it is not necessary to run openremote-cli --provider aws from EC2 instance. This option is meant to run from local machine as it creates a new EC2 instance.

bobster888 · July 25, 2022, 11:40am

ok … thanks for this … think we are getting closer (I hope)

I did the following where microgro.co is in the r53 hosted zone … and got the following error(s) …

[ec2-user@ip-172-31-90-79 ~]$ or deploy --dnsname microgro.co --region us-east-1 -v
If you need help go to https://forum.openremote.io/

Deploying OR… This usually takes less than 15 minutes.

> docker swarm init
> docker volume rm openremote_postgresql-data
> wget -nc https://github.com/openremote/openremote-cli/raw/main/cloudformation/mvp-docker-compose.yml
100% […] 3539 / 3539
Generated password: 6wrkb0Tjoc

> DOMAINNAME=microgro.co IDENTITY_NETWORK_HOST=microgro.co PASSWORD=6wrkb0Tjoc docker-compose -f mvp-docker-compose.yml -p openremote up -d
ERROR:root:stderr: b’/bin/sh: docker-compose: command not found\n’
ERROR:root:Exiting main because of uncached: (127, ‘/bin/sh: docker-compose: command not found\n’)
Traceback (most recent call last):

File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/cli.py”, line 624, in main*
OpenRemote(sys.argv[1:], parser=parser)*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/cli.py”, line 71, in init*
getattr(self, command)(arguments)*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/cli.py”, line 187, in deploy*
args.dnsname,*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/scripts.py”, line 68, in deploy*
f’{env}docker-compose -f mvp-docker-compose.yml -p openremote up -d’*
File “/home/ec2-user/.local/lib/python3.7/site-packages/openremote_cli/shell.py”, line 37, in execute*
output.returncode, output.stdout.decode(‘utf-8’)*
Exception: (127, ‘/bin/sh: docker-compose: command not found\n’)
[ec2-user@ip-172-31-90-79 ~]$

Any tips appreciated.

cheers

michal · July 25, 2022, 12:48pm

This is only domain, you should add host to it - e.g. --dnsname test.microgro.co.

docker-compose: command not found is self explaining. You don’t have docker-compose installed on your EC2 box. You can either use --provider aws which spins a new EC2 instance with docker-compose preinstalled or install docker-compose on your EC2 host:

curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose

bobster888 · July 27, 2022, 3:05pm

thx for last … OK …

created iot.microgro.co hosted zone
installed docker compose
ran following … with same error as before …

[ec2-user@ip-172-31-90-79 ~]$ or deploy --dnsname test.iot.microgro.co --region us-east-1 -v
If you need help go to https://forum.openremote.io/

Deploying OR… This usually takes less than 15 minutes.

> docker swarm init
> docker volume rm openremote_postgresql-data
> wget -nc https://github.com/openremote/openremote-cli/raw/main/cloudformation/mvp-docker-compose.yml
100% […] 3539 / 3539
Generated password: TqoDAPLtL

> DOMAINNAME=test.iot.microgro.co IDENTITY_NETWORK_HOST=test.iot.microgro.co PASSWORD=TqoDAPLtL docker-compose -f mvp-docker-compose.yml -p openremote up -d
ERROR:root:stderr: b’/usr/local/bin/docker-compose: line 1: Not: command not found\n

and just to confirm docker-compose is installed …

[ec2-user@ip-172-31-90-79 ~]$ ls -l
total 45168
drwxr-xr-x 3 ec2-user ec2-user 78 Jul 20 17:58 aws
-rw-rw-r-- 1 ec2-user ec2-user 46244650 Jul 22 16:02 awscliv2.zip
-rw-rw-r-- 1 ec2-user ec2-user 3539 Jul 27 14:52 mvp-docker-compose.yml
[ec2-user@ip-172-31-90-79 ~]$ which docker-compose
/usr/local/bin/docker-compose
[ec2-user@ip-172-31-90-79 ~]$

Please note the mvp-docker-compose.yml file appeared in the directory … does this mean I’m getting closer?

Any thoughts on why the error?

cheers

michal · July 27, 2022, 5:44pm

This is in fact a different error message. The previous one was from /bin/sh: docker-compose: command not found and now it is from docker-compose itself. It is complaining about Not, I suspect that docker-compose isn’t installed properly. Try to run

> docker-compose --version
Docker Compose version v2.5.1

Check what is inside your /usr/local/bin/docker-compose file

bobster888 · July 28, 2022, 10:53am

good catch! … I need to read these errors better.

so … >docker-compose --version … doesn’t work …

however >sudo docker-compose --version gives …

Docker Compose version v2.7.0
i’m logged in as root …

I tried uninstalling using >yum remove docker-compose (and sudo) … but to no avail

I think I installed docker-compose while logged in as root and not as IAM

any advice appreciated.

michal · July 28, 2022, 1:11pm

If sudo docker-compose works than it might be permissions issue. What do you see?

> ls -l `which docker-compose`

on my mac I see

> ls -l `which docker-compose` 
lrwxr-xr-x  1 root  staff  62 Jun 24  2021 /usr/local/bin/docker-compose -> /Applications/Docker.app/Contents/Resources/bin/docker-compose

Important is that it has x bit set for all users. You can also try

sudo chmod 755 `which docker-compose`

bobster888 · July 28, 2022, 4:20pm

many thanks for all this help but to no avail … i still need to use sudo to get docker-compose version … and still getting same errors when trying to install. (sigh)

I obviously messed up somewhere in my initial setup … hardly surprising as the whole aws ec2 thing is totally new to me.

I’m going to terminate this instance and start again … hopefully more carefully and armed with a little bit more experience and familiarity.

laters

michal · July 30, 2022, 8:28am

I think that below one would work

or deploy --provider aws --dnsname test.iot.microgro.co --region us-east-1 -v

assuming that you have hosting zone iot.microgro.co configured on account accessible with the openremote-cli profile.

michal · July 31, 2022, 10:36am

also, for this to work you hosted zone must be publicly registered and correctly configured. As for now this is not the case

> nslookup iot.microgro.co 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8#53

** server can't find iot.microgro.co: NXDOMAIN

bobster888 · August 5, 2022, 9:55am

Hi … I tried your link …

I can access the openremote host on https://test.mvp.openremote.io.

but no access?

bobster888 · August 5, 2022, 12:50pm

ok … thx for this and got it deployed and pull aok

however the or manager login does not appear on the browser page just a blank page with or logo on the tab.

Also did following …

$ nslookup iot.microgro.co
Server: 172.31.0.2
Address: 172.31.0.2#53

** server can’t find iot.microgro.co: NXDOMAIN

any tips or links to next step(s) appreciated.

many thanks for your help so far to this point.

michal · August 5, 2022, 1:42pm

This is correct. As hosting costs money I usually destroy a test stack as soon as I don’t need it anymore.

michal · August 5, 2022, 1:46pm

Some logs would be helpful. Also, as you performed a lot of different things a description what you’ve done this time would help too.

bobster888 · August 8, 2022, 3:41pm

I believe the main thing was that I created a new or-user but this time it was programmatic …

then ran …

$aws configure --profile openremote-cli

and inputted the new id and secret keys …

and then …

$or deploy --provider aws --dnsname test.iot.microgro.co --region us-east-1 -v

and it produced no errors and deployed (huzzar!)

Next ran the following …

docker-compose pull

then
docker-compose -p openremote up

and it seemed to load up aok … however the following (log?) errors started to appear after 10 mins …

Also, there is no access on http://test.iot.microgro.co/

Do I need to add something in route53 … Type A record or?

However, the aws public ip of the instance does bring up a blank browser page with the logo OR logo on the tab … but no sign in dialogue …

Are there any other logs I can look at?

I tried the following …

$ or deploy -a health -v
If you need help go to https://forum.openremote.io/

0.1.0
[ec2-user@ip-172-31-91-11 ~]$

Is this ok … normal … or?

cheers

michal · August 9, 2022, 8:52am

You are mixing 2 things. Deploying openremote stack on a fresh EC2 instance with deploying it locally. Anyway, if you say that or deploy --provider aws --dnsname test.iot.microgro.co --region us-east-1 -v runs without errors this means that it already created A record in route53 and was able to perform health check of the deployed stack. Just look at the command output and try to understand what this feedback means. However, you are saying that there is no access to test.iot.microgro.co, which is strange as it should be reported as error during or deploy... command. By the way is your hosting zone public? With private hosting zone you will be able to access this host only within VPC.

bobster888 · August 10, 2022, 2:25pm

ah … yes … you are right … confused about local and newly created instance …

and yes … the A record was created for test.iot.microgro.co in r53 … but still no joy in accessing the OR login page

Soo … I’ve gone back and used my original microgro.co instance to deploy a new stack to new instance

test1.iot.microgro.co … then ssh to this instance and tried following and got the following warnings …

$docker-compose pull
WARN[0000] The “OR_HOSTNAME” variable is not set. Defaulting to a blank string.
WARN[0000] The “OR_EMAIL_USER” variable is not set. Defaulting to a blank string.
WARN[0000] The “OR_EMAIL_PASSWORD” variable is not set. Defaulting to a blank string.
WARN[0000] The “OR_HOSTNAME” variable is not set. Defaulting to a blank string.
WARN[0000] The “OR_HOSTNAME” variable is not set. Defaulting to a blank string.

Do I need to edit the yml file?

thx