Setup Machine:

This is step 1 of 5 major steps:

1. Install base containers on a server
2. Setup services containers
3. Setup Gleaner containers
4. Initial setup of services and loading of data
5. Setup Geocodes UI using datastores defined in Initial Setup

Base Machine to run Docker Containers Treafik and Portainer:

This is what will be needed to create a production server

• base virtual machine for containers
• ability to request DNS,

SUMMARY

These are a summary of the steps, The Step Details are below.

DOCKER REQUIREMENT

If you are running on Ubuntu, you need to remove the provided docker.com version. Official docker package We suggest that for others, confirm that you can run

docker compose version
Docker Compose version v2.13.0

If you cannot run docker compose then update to the docker.com version This is the version we are presently running.

Client: Docker Engine - Community
     Version:           20.10.21
     API version:       1.41

DOCKER SWARM

Docker swarm needs to be init'd with the public ip address.

---

Step Overview:

• create a machine in openstack (if production)
  • select size
  • associate floating IP
    • ask for DNS for that ip to be configured with needed names

• ssh to machine. You do not need to have the DNS's to install the software. But it will be needed.

  • update apt
    • sudo apt update

  • update base software

    • sudo apt upgrade

  • install docker

Use Official Docker for Ubuntu

• use these docker install instructions

• add ubuntu (or other users) to docker group

  • sudo groupadd docker
  • sudo usermod -aG docker ubuntu
• reboot
• sudo reboot now

• create a directory for geocodes, set up permissions and groups
  • sudo mkdir /data/decoder
  • ln -s /data/decoder/ decoder
  • ln -s /data/decoder/ geocodes
  • sudo addgroup geocodes
  • usermod -a -G geocodes {user}
  • sudo chgrp geocodes /data/decoder
  • sudo chmod g+rwx /data/decoder
• init docker swarm

  • DOCKER SWARM

    Docker swarm needs to be init'd with the public ip address.
  • nslookup {HOSTNAME}
  • sudo docker swarm init --advertise-addr {PUBLIC_IP}
  • save the token to a file (I use NOTES)
• verify proper base configuration
  • docker compose --help shows a -p flag
• SNAPSHOT and creaate an image
• clone geocodes
  • cd decoder or cd /data/decoder
  • git clone https://github.com/earthcube/geocodes.git
• configure a base server
• base-machine-compose.yaml is the full stack with a portainer, treafik
• base-swarm-compose.yaml is just a treakfit. connect with your existing portainer.
• take a break and wait for the DNS entries.
  • if you cannot wait for the DNS, you can go to the no cert port
    • https://{HOST}}:9443/
    • use chrome, click advanced, and go to the port.

---

Step Details:

create a machine in openstack

Suggested size:

SDSC Openstack:

• ubuntu 22
• 100 gig
  • m1.2xlarge (8 CPU, 32 gig)
  • network: earthcube
• Security groups:
  • remote ssh (22)
  • geocodes (http/https; 80:443)
  • portainer (temporary need: 9443)
  • minio (optional: 9000/9001)
• Keypair: earthcube (or any)

Ports Pre-DNS

minio ports do not need to be open, we are proxying on 80 and 443 Portainer port (9443) can be opended temporarily if you want to play a bit pre-DNS.

Associate a Public IP

After the machine is created, we can change the IP to the one associated with geocodes.earthcube.org

---

setup domain names

• Machines
• Name for remote DNS

ESSENTIAL for PRODUCTION

It is ESSENTIAL for PRODUCTION that the names are defined in a DNS. This allows for https for all services and some services (aka s3/minio) do not play well with a proxy.

You might be able to run production stack using localhost, with these DNS... but that mucks with the lets encrypt HTTPS certs... if you control your own DNS, these are the entries needed. Name for local DNS

Local testing and development can be using the local compose configuration. This use http, and local ports for services that cannot be proxied

---

ssh to machine and verify

ssh -i ~/.ssh/earthcube.pem ubuntu@{public IP}

add your ssh key so you can log in as main user (eg. ubuntu)

SSH Keys

for production, we recommend that you use a group account/main account

to do this you will need to create and copy a public/private key

Generate an ssh-key:

ssh-keygen -t rsa -b 4096 -C "comment"

copy it to your remote server:

ssh-copy-id user@ip

or you can manually copy the

~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys.

Edit

It can be done through ssh command as mentioned @chepner:

ssh user@ip 'mkdir ~/.ssh'
ssh user@ip 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub

(Above based on: stackexchange)

---

configure a base server

update OS

• update apt
  • sudo apt update
• update base software
  • `sudo apt upgrade

add docker, git

Offical Docker for Ubuntu

use these docker install instruction

• add ubuntu (or other users) to docker group
• sudo groupadd docker
• sudo usermod -aG docker ubuntu
• reboot
• sudo reboot now

create a directory for geocodes, set up permissions and groups

* `sudo mkdir /data/decoder`
* `ln -s /data/decoder/ decoder`
* `sudo addgroup geocodes`
* `usermod -a -G geocodes {user}`
* `sudo chgrp geocodes /data/decoder`
* `sudo chmod g+rwx /data/decoder`

clone geocodes stack

• cd decoder or cd /data/decoder
• git clone https://github.com/earthcube/geocodes.git
• cd geocodes/deployment

copy base_machine.example.env, to .env

Option 1. production server use .env

• cp base_machine.example.env .env
• modify the file
  • note: you can also copy the full portainer.env.

Option 2. testing, playing, developer

• cp base_machine.example.env {myproject}.env
• modify the file
  • note: you can also copy the full portainer.env.

modify the treafik-data/traefik.yml

treafik-data/traefik.yml

acme:
# using staging for testing/development
#     caServer: https://acme-staging-v02.api.letsencrypt.org/directory
    email: example@earthcube.org
    storage: acme.json
    httpChallenge:
        entryPoint: http

If production, comment the line as shown. Developers see Lets Encypt Notes

Let Encrypt Notes

lets encrypt,

(developers) set to use staging environment server while testing If you are doing development, then leave the caServer uncommented.

start the base containers

• new machine or developer

• ./run_base.sh -e {myproject}.env

• production: this uses the default .env (cp portainer.env .env)

./run_base.sh

      ubuntu@geocodes-dev:~/geocodes/deployment$ ./run_base.sh -e geocodes-1.env
      Error response from daemon: network with name traefik_proxy already exists
      NETWORK ID     NAME              DRIVER    SCOPE
      ad6cbce4ec60   bridge            bridge    local
      2f618fa7da6d   docker_gwbridge   bridge    local
      f8048bc7a3d9   host              host      local
      kibdi510bt0x   ingress           overlay   swarm
      12c01a2186b0   none              null      local
      u4d4oxfy7olc   traefik_proxy     overlay   swarm
      Verify that the traefik_proxy network SCOPE is swarm
      traefik_data
      portainer_data
      true
      [+] Running 2/2
      ⠿ Container portainer  Started                                           13.7s
      ⠿ Container traefik    Started

Testing Setup

---

Are containers running

docker ps

    * ubuntu@geocodes-dev:~/geocodes/deployment$ docker ps
      CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS         PORTS                                                                      NAMES
      09a5d8683cce   traefik:v2.4                    "/entrypoint.sh trae…"   2 minutes ago   Up 2 minutes   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   traefik
      d3e2333ade6f   portainer/portainer-ce:latest   "/portainer"             2 minutes ago   Up 2 minutes   8000/tcp, 9000/tcp, 9443/tcp                                               portainer

---

Is network setup correctly?

docker network ls

docker network ls
      NETWORK ID     NAME              DRIVER    SCOPE
      ad6cbce4ec60   bridge            bridge    local
      2f618fa7da6d   docker_gwbridge   bridge    local
      f8048bc7a3d9   host              host      local
      kibdi510bt0x   ingress           overlay   swarm
      12c01a2186b0   none              null      local
      u4d4oxfy7olc   traefik_proxy     overlay   swarm

Note

NAME:traefik_proxy needs to exist, and be DRIVER:overlay, SCOPE:swarm

Are volumes available

docker volumes

ubuntu@geocodes-dev:~$ docker volume ls
      DRIVER    VOLUME NAME
      local     graph
      local     minio
      local     portainer_data
      local     traefik_data

---

are Traefik and Portainer available via the web?

• Treafik https://admin.{host}
  • login is admin:iforget

image

• Portainer https://portainer.{host}/
  • this will ask you to setup and admin password

image

Go to step 2.

1. Install base containers on a server
2. Setup services containers
3. Setup Gleaner containers
4. Initial setup of services and loading of data
5. Setup Geocodes UI using datastores defined in Initial Setup

---

How to/Troubleshooting

updating Portainer, or treafik

the latest image needs to be pulled

docker pull portainer/portainer-ce:latest

then ./run_base.sh

How tos needed:

• LOCAL DNS SETUP
• editing your local machine /etc/hosts file does not work with letsencrypt.
• If user has a local name server they control, that might work.
• setup a new password for traefik
• lets encrypt

---