Project

KO | EN

gitlite — search

#python #hacktoberfest #react #ai #android #rust #macos #java #windows #awesome #cli #awesome-list #security #mcp

(★ 48)

Infrastructure

Project Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Kartta Labs Suite

This project contains everything you need to spin up a complete suite of Kartta
Labs web applications for development purposes on your local system in docker.  It also
contains configuration files and scripts needed to launch the applications in
GKE (Google Kubernetes Engine).

This project does not contain the code for the individual applications -- it
just contains the overall scripts and configuration files for launching and
managing the applications together.  The code for each application is stored in
its own repository in http://github.com/kartta-labs.  The scripts in this
project will take care of cloning the applications repos for you.

## Initial Setup

Do these things just once to get set up.

1. Install Docker

2. Clone this repo and cd into it

```
   git clone https://github.com/kartta-labs/Project
   cd Project
   ```

3. Generate a secrets file by running (optionally in an isolated python environment).

```
   pip install -r requirements.txt
   ./makesecrets
   ```
   This will create the file `./container/secrets/secrets.env` by copying `example-secrets.env`.
   For running locally with docker for development, you don't need to edit the generated file,
   although you may customize the configuration by editing it if you want.

You can provide a set of intial values to `makesecrets` by passing it a YAML file if you want:
   ```
   ./makesecrets values.yml
   ```
   This will cause `makesecrets` to substitute values found in `values.yml` for the corresponding
   variable when generating `./container/secrets/secrets.env`.  Note: you can't use `values.yml` to
   add additional variables -- the substitution only happens for variables already present in
   `example-secrets.env`.

If you want to use code from a repository other than the offical http://github.com/kartta-labs
   repository for one or more of the applications, change the value of the appropriate *_REPO
   variable(s) in ./container/secrets/secrets.env before continuing.

4. Run the bootstrap script
   ```
   ./bootstrap.sh
   ```
   This will take 45-60 minutes or so but only has to be done once.  It might ask you to
   enter your password at various points (because it uses sudo, since many docker commands
   need to run as root), so check on it occasionally.

## Running

Once `bootstrap.sh` finishes, to run the suite of applications, do

```
./start
```
Leave this command running in one terminal; use a separate terminal for everything else below.  To shut everything down
when you are done, or so you can restart everything, hit Ctrl-C.

Once all the containers have started (usually 1 or 2 minutes), visit http://localhost/e/ in your browser
for editor, or http://localhost/w/ for warper.

### Things to note

1. See the file ARCHITECTURE.md in this directory for a general overview of
   how these applications are configure.

1. As currently configured this will only work on port 80 on localhost, which
   unfortunately means you can't port-forward to your local machine if you're
   accessing the host remotely, since only root can forward privileged ports.
   That means that in order to view this in a browser you'll have to
   remotedesktop to the host machine and run the browser there.  It's possible
   to reconfigure everything to use a public port other than 80, though, and/or
   a hostname other than 'localhost', but that will involve some careful
   tweaking of the various nginx docker configs.

1. The apps currently run on port 8888 using http (not https).  If you want
   to view using a local browser, use ssh to tunnel port 8888 from your remote
   host to your local machine: `ssh -L 8888:localhost:8888 remote.machine.net`.

1. Run `docker ps` to see a list of all the running containers; the output will look similar to this:
   ```
   CONTAINER ID   IMAGE                  COMMAND                  PORTS                                            NAMES
   cd904223a9d1   project_fe             "nginx -g 'daemon of…"   0.0.0.0:80->80/tcp                               project_fe_1
   3539235734cd   project_editor         "/editor-container-s…"   0.0.0.0:32936->80/tcp                            project_editor_1
   34d109561bca   project_warper      "/warper-containe…"   0.0.0.0:32938->80/tcp, 0.0.0.0:32937->3000/tcp   project_warper_1
   c2916d2bb811   project_cgimap         "/cgimap-container-s…"   0.0.0.0:32935->8000/tcp                          project_cgimap_1
   1d039d94f71d   project_oauth-proxy    "/root/go/bin/oauth2…"   0.0.0.0:4180->4180/tcp                           project_oauth-proxy_1
   afdeaf8b1429   mdillon/postgis        "docker-entrypoint.s…"   5432/tcp                                         project_warper-db_1
   750b55e3ae1f   redis                  "docker-entrypoint.s…"   6379/tcp                                         project_redis_1
   707c200a13e8   postgres:11            "docker-entrypoint.s…"   5432/tcp                                         project_editor-db_1
   ```

1. Each container corresponds to a "service" entry in the `docker-compose.yml` file.
   Each one is like a single server, or VM.  The ones whose IMAGE name start with 'project_' correspond
   to services Kartta runs in production:

* fe: frontend nginx server, receives all incoming requests and routes them
       internally to the relevant other service
     * oauth-proxy: handles all authentication
     * editor: editor-website rails app
     * warper: warper rails app

The other containers are local instances of postgres/postgis/redis servers which are needed by
   the project_ services.  In production we use GCP managed versions of these services.

You can use the `./dsh` script to start a bash shell in any of the
   containers; it takes a single arg which is the name of the service (without
   the 'project_' prefix or '_1' suffix).  For example `./dsh editor`.  This is like
   "ssh-ing" to the VM; once in the shell, you can poke around with the
   filesystem and running processes for the container.  In particular, you can
   use this to view the nginx config, application log files, or running processes.

1. All the config files and application code for the editor and warper apps,
   as well as all nginx conf files, are "mounted" inside the containers from
   their corresponding location in the 'Project' directory on your workstation (the
   docker "host").  You can edit these files with an editor on the host, and the
   changes are immediately visible from inside the running containers.  The mappings
   that determine which files/dirs get mounted in each container, and where, are
   given by the "volumes" directives in the `docker-compose.yml` file.

1. Inside any of the nginx containers (editor, warper, fe), run
   `nginx -s reload`
   to restart nginx after changing either the nginx conf or any of the
   application files.

1. `docker-compose` creates a local IP network in which each container is known
   by its service name from the `docker-compose.yml` file.  For example, the
   editor container can be referred to by the name `editor` when constructing
   hostnames and/or urls for use in any of the containers.  These names are not
   available on the host -- only inside the containers.

1. The `docker-compose.yml` file in the top level `Project` directory references various
   environment variables which must be set in order for it to work.  These variables
   are set in the secrets file `./container/secrets/ecrets.env`, and the `./start`
   script takes care of loading these values into the environment automatically.
   You should always use `./start` to launch the containers -- it runs `docker-compose up`
   for you, after loading the environment variables from the secrets file.
   If you need to otherwise run `docker-compose` (e.g. to re-build an image for an
   application you're developing), run the `./dcwrapper` script rather than running
   `docker-compose` directly.

## Misc helpful commands

* Shut down any remaining running containers:
  ```
  sudo docker-compose down
  ```

* Rebuild the docker images before starting the containers:
  ```
  docker-compose up --build
  ```
  This uses docker's build cache so it usually runs much faster than the 30 min
  or so needed to run the bootstrap script initially.  Docker is pretty good
  (but not perfect) about knowing what parts of each build need to be re-done
  based on which files have changed.

* Completely remove all docker artifacts, including built images and cache data:
  ```
  sudo docker system prune -a
  ```
  Do this if you want to force docker to rebuild EVERYTHING the next time you
  run `docker-compose up` (this is like a much stronger version of
  `docker-compose up --build`).

* The database containers store their data in files in the `tmp` dir (the one at the top level of the `Project` dir).
  You can clear the data from a database by removing the corresponding file from `tmp`.  Note that if you do that,
  the database will need to be re-initialized by re-running the relevant commands from `bootstrap.sh`.
  If you are at all unsure about what's needed to do this, it's probably safer to start over by doing
  `sudo docker system prune -a` to clear everything out of docker, clone a new copy of this `Project` repo,
  and re-run `bootstrap.sh`.

## Kubernetes Deployments

The 'k8s' directory contains configuration files and scripts for deploying the
suite of applications in GKE (Google Kubernetes Engine).  This is intended for
running production servers, and/or for testing the production deployment
process.  If you just want to run the suite so you can work on one or more of
the applications, use the docker process described above.

1. Create a new GCP project, or decide on an existing one to use.

2. Clone a fresh copy of this 'Project' repo.  This should be a new copy -- not one that you
   have previously used with docker for local development.

3. Generate a secrets file by running

```
   ./makesecrets
   ```

4. Edit `./container/secrets/secrets.env` and set all the required values at the top of the file;
   see the comments in the file for details.

5. Warper has an optional feature that causes it to run OCR on uploaded maps in an attempt to initially
   place them in an approximate location.  This feature is disabled by default.  If you want to
   enable this feature:
     * Go to https://pantheon.corp.google.com/apis/credentials and create a new API key
     * Edit `./container/secrets/secrets.env` to insert the value of that API key for
       the variable MAPWARPER_GOOGLE_MAPS_KEY.  Also change the value of MAPWARPER_ENABLE_OCR_JOB
       to "true".

6. Run `./k8s/kbootstrap.sh`.  It may take up to an hour to run.  When it finishes, it will
   print out a message with the IP address of the running server (this is the IP of the load
   balancer).

7. Create a DNS entry which associates the generated IP address with the SERVER_NAME
   you chose above.

8. The vector tile server deployment is handled by a separate script.  If you also want to
   create a vector tile server deployment, run `./k8s/kvector-bootstrap.sh`.  Do this _after_
   running `./k8s/kbootstrap.sh`; `./k8s/kvector-bootstrap.sh` will not work correctly if
   `./k8s/kbootstrap.sh` has not been run first.

9. Note that both `./k8s/kbootstrap.sh` and `./k8s/kvector-bootstrap.sh` edit the secrets file
   (container/secrets/secrets.env) to add information such as IP addresses and names of
   resources they create, or values for passwords they generate.  You should guard that secrets
   file carefully -- it will be needed when making any changes to the deployment.

Project

Project Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Project

Project Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README