Setup Machine:

This is step 1 of 5 major steps:

Base Machine to run Docker Containers Treafik and Portainer:

This is what will be needed to create a production server

base virtual machine for containers
ability to request DNS,

SUMMARY

These are a summary of the steps, The Step Details are below.

DOCKER REQUIREMENT

If you are running on Ubuntu, you need to remove the provided docker.com version. Official docker package We suggest that for others, confirm that you can run

docker compose version
Docker Compose version v2.13.0

If you cannot run docker compose then update to the docker.com version This is the version we are presently running.

Client: Docker Engine - Community
     Version:           20.10.21
     API version:       1.41

DOCKER SWARM

Docker swarm needs to be init'd with the public ip address.

Step Overview:

create a machine in openstack (if production)
- select size
- associate floating IP
  - ask for DNS for that ip to be configured with needed names
ssh to machine. You do not need to have the DNS's to install the software. But it will be needed.
- update apt
  - sudo apt update
- update base software
  - sudo apt upgrade
- install docker

Use Official Docker for Ubuntu

use these docker install instructions
add ubuntu (or other users) to docker group
- sudo groupadd docker
- sudo usermod -aG docker ubuntu
reboot
sudo reboot now

create a directory for geocodes, set up permissions and groups
- sudo mkdir /data/decoder
- ln -s /data/decoder/ decoder
- ln -s /data/decoder/ geocodes
- sudo addgroup geocodes
- usermod -a -G geocodes {user}
- sudo chgrp geocodes /data/decoder
- sudo chmod g+rwx /data/decoder
init docker swarm
- DOCKER SWARM
  Docker swarm needs to be init'd with the public ip address.
- nslookup {HOSTNAME}
- sudo docker swarm init --advertise-addr {PUBLIC_IP}
- save the token to a file (I use NOTES)
verify proper base configuration
- docker compose --help shows a -p flag
SNAPSHOT and creaate an image
clone geocodes
- cd decoder or cd /data/decoder
- git clone https://github.com/earthcube/geocodes.git
configure a base server
base-machine-compose.yaml is the full stack with a portainer, treafik
base-swarm-compose.yaml is just a treakfit. connect with your existing portainer.
take a break and wait for the DNS entries.
- if you cannot wait for the DNS, you can go to the no cert port
  - https://{HOST}}:9443/
  - use chrome, click advanced, and go to the port.

Step Details:

create a machine in openstack

Suggested size:

SDSC Openstack:

ubuntu 22
100 gig
- m1.2xlarge (8 CPU, 32 gig)
- network: earthcube
Security groups:
- remote ssh (22)
- geocodes (http/https; 80:443)
- portainer (temporary need: 9443)
- minio (optional: 9000/9001)
Keypair: earthcube (or any)

Ports Pre-DNS

minio ports do not need to be open, we are proxying on 80 and 443 Portainer port (9443) can be opended temporarily if you want to play a bit pre-DNS.

Associate a Public IP

After the machine is created, we can change the IP to the one associated with geocodes.earthcube.org

setup domain names

ESSENTIAL for PRODUCTION

It is ESSENTIAL for PRODUCTION that the names are defined in a DNS. This allows for https for all services and some services (aka s3/minio) do not play well with a proxy.

You might be able to run production stack using localhost, with these DNS... but that mucks with the lets encrypt HTTPS certs... if you control your own DNS, these are the entries needed. Name for local DNS

Local testing and development can be using the local compose configuration. This use http, and local ports for services that cannot be proxied

ssh to machine and verify

ssh -i ~/.ssh/earthcube.pem ubuntu@{public IP}

add your ssh key so you can log in as main user (eg. ubuntu)

SSH Keys

for production, we recommend that you use a group account/main account

to do this you will need to create and copy a public/private key

Generate an ssh-key:

ssh-keygen -t rsa -b 4096 -C "comment"

copy it to your remote server:

ssh-copy-id user@ip

or you can manually copy the

~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys.

Edit

It can be done through ssh command as mentioned @chepner:

ssh user@ip 'mkdir ~/.ssh'
ssh user@ip 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

(Above based on: stackexchange)

configure a base server

update OS

update apt
- sudo apt update
update base software
- `sudo apt upgrade

add docker, git

Offical Docker for Ubuntu

use these docker install instruction

add ubuntu (or other users) to docker group
sudo groupadd docker
sudo usermod -aG docker ubuntu
reboot
sudo reboot now

create a directory for geocodes, set up permissions and groups

* `sudo mkdir /data/decoder`
* `ln -s /data/decoder/ decoder`
* `sudo addgroup geocodes`
* `usermod -a -G geocodes {user}`
* `sudo chgrp geocodes /data/decoder`
* `sudo chmod g+rwx /data/decoder`

clone geocodes stack

cd decoder or cd /data/decoder
git clone https://github.com/earthcube/geocodes.git
cd geocodes/deployment

copy base_machine.example.env, to .env

Option 1. production server use .env

cp base_machine.example.env .env
modify the file
- note: you can also copy the full portainer.env.

Option 2. testing, playing, developer

cp base_machine.example.env {myproject}.env
modify the file
- note: you can also copy the full portainer.env.

modify the treafik-data/traefik.yml

treafik-data/traefik.yml

acme:
# using staging for testing/development
#     caServer: https://acme-staging-v02.api.letsencrypt.org/directory
    email: example@earthcube.org
    storage: acme.json
    httpChallenge:
        entryPoint: http

If production, comment the line as shown. Developers see Lets Encypt Notes

Let Encrypt Notes

lets encrypt,

(developers) set to use staging environment server while testing If you are doing development, then leave the caServer uncommented.

start the base containers

new machine or developer
./run_base.sh -e {myproject}.env
production: this uses the default .env (cp portainer.env .env)

./run_base.sh

      ubuntu@geocodes-dev:~/geocodes/deployment$ ./run_base.sh -e geocodes-1.env
      Error response from daemon: network with name traefik_proxy already exists
      NETWORK ID     NAME              DRIVER    SCOPE
      ad6cbce4ec60   bridge            bridge    local
      2f618fa7da6d   docker_gwbridge   bridge    local
      f8048bc7a3d9   host              host      local
      kibdi510bt0x   ingress           overlay   swarm
      12c01a2186b0   none              null      local
      u4d4oxfy7olc   traefik_proxy     overlay   swarm
      Verify that the traefik_proxy network SCOPE is swarm
      traefik_data
      portainer_data
      true
      [+] Running 2/2
      ⠿ Container portainer  Started                                           13.7s
      ⠿ Container traefik    Started

Testing Setup

Are containers running

docker ps

    * ubuntu@geocodes-dev:~/geocodes/deployment$ docker ps
      CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS         PORTS                                                                      NAMES
      09a5d8683cce   traefik:v2.4                    "/entrypoint.sh trae…"   2 minutes ago   Up 2 minutes   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   traefik
      d3e2333ade6f   portainer/portainer-ce:latest   "/portainer"             2 minutes ago   Up 2 minutes   8000/tcp, 9000/tcp, 9443/tcp                                               portainer

Is network setup correctly?

docker network ls

docker network ls
      NETWORK ID     NAME              DRIVER    SCOPE
      ad6cbce4ec60   bridge            bridge    local
      2f618fa7da6d   docker_gwbridge   bridge    local
      f8048bc7a3d9   host              host      local
      kibdi510bt0x   ingress           overlay   swarm
      12c01a2186b0   none              null      local
      u4d4oxfy7olc   traefik_proxy     overlay   swarm

Note

NAME:traefik_proxy needs to exist, and be DRIVER:overlay, SCOPE:swarm

Are volumes available

docker volumes

ubuntu@geocodes-dev:~$ docker volume ls
      DRIVER    VOLUME NAME
      local     graph
      local     minio
      local     portainer_data
      local     traefik_data

are Traefik and Portainer available via the web?

Treafik https://admin.{host}
- login is admin:iforget

image

Portainer https://portainer.{host}/
- this will ask you to setup and admin password

image

Go to step 2.

How to/Troubleshooting

updating Portainer, or treafik

the latest image needs to be pulled

docker pull portainer/portainer-ce:latest

then ./run_base.sh

How tos needed:

LOCAL DNS SETUP
editing your local machine /etc/hosts file does not work with letsencrypt.
If user has a local name server they control, that might work.
setup a new password for traefik
lets encrypt