{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "![Docker](https://www.docker.com/sites/default/files/d8/styles/role_icon/public/2019-07/horizontal-logo-monochromatic-white.png?itok=SBlK2TGU)\n", "\n", "Some notes about using Docker to host various applications like Heptapod, Discourse, CoCalc etc. on Linux and AWS.\n", "" ] }, { "cell_type": "markdown", "metadata": { "toc": true }, "source": [ "

Table of Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Docker\n", "\n", "Several packages (such as Heptapod and CoCalc) require a rather complete system, so are\n", "easiest to install using [Docker](https://docs.docker.com) containers. Here we discuss\n", "how to set these up. Where possible, we try to use [Rootless]\n", "mode, which prevents the need for\n", "providing docker with root access, but has limitations. Currently, [Rootless] works for\n", "Heptapod, but causes [problems with CoCalc](https://github.com/sagemathinc/cocalc-docker/issues/118).\n", "\n", "[Rootless]: \n", "\n", "## Docker Account\n", "\n", "We generally work with a `docker` user, and keep all docker-specific files for this\n", "account:\n", "\n", "```bash\n", "sudo useradd -m docker\n", "sudo usermod -aG sudo docker # Enable sudo for docker.\n", "sudo su docker\n", "chsh -s /bin/bash\n", "bash\n", "curl -fsSL https://get.docker.com/rootless | sh\n", "```\n", "\n", "On my machine, I then add a `swandocker` alias to my `~/.ssh/config` file, e.g.:\n", "\n", "```bash\n", "# ~/.ssh/config\n", "\n", "Host *docker\n", " User docker\n", " ForwardAgent yes\n", " SetEnv LC_EDITOR=vi\n", "\n", "Host swan*\n", " HostName swan.physics.wsu.edu\n", "\n", "Host penguin*\n", " HostName penguin.physics.wsu.edu\n", "\n", "Host *\n", " ForwardAgent yes\n", " SendEnv LC_HG_USERNAME\n", " SendEnv LC_GIT_USERNAME\n", " SendEnv LC_GIT_USEREMAIL\n", " AddressFamily inet\n", " # Force IPv4\n", " # https://www.electricmonk.nl/log/2014/09/24/\n", " # ssh-port-forwarding-bind-cannot-assign-requested-address/ \n", "```\n", "\n", "All of the following is done in this account:\n", "\n", "```bash\n", "ssh ssh-copy-id swandocker\n", "ssh swandocker\n", "...\n", "```\n", "\n", "## Installing Docker (With Root Access)\n", "\n", "```bash\n", "ssh swandocker\n", "sudo apt-get update\n", "sudo apt-get upgrade # Optional\n", "sudo apt-get install uidmap dbus-user-session\n", "sudo apt-get purge docker docker.io # Remove old root-version of docker.\n", "sudo apt-get autoremove --purge\n", "sudo apt-get install docker docker.io nvidia-docker2\n", "sudo systemctl start docker\n", "```\n", "\n", "```bash\n", "sudo systemctl stop docker docker.sock\n", "```\n", "\n", "\n", "## Installing Docker Rootless\n", "\n", "*Note: Be sure to completely purge any previous root-enabled version of Docker before proceeding.*\n", "\n", "```bash\n", "ssh swandocker\n", "sudo apt-get update\n", "sudo apt-get upgrade\n", "sudo apt-get install uidmap dbus-user-session\n", "sudo apt-get purge docker docker.io # Remove old root-version of docker.\n", "sudo apt-get autoremove --purge\n", "```\n", "\n", "To actually install [rootless], run the following script, downloaded with curl:\n", "\n", "```bash\n", "curl -fsSL https://get.docker.com/rootless | sh\n", "```\n", "\n", "This will put everything in `~/bin`\n", "\n", "\n", "This allows the `docker` user to add processes to start services that will start at login.\n", "\n", "```bash\n", "sudo loginctl enable-linger docker\n", "systemctl --user start docker\n", "```\n", "\n", "Add the appropriate environmental variables to `~docker/.bashrc`:\n", "\n", "```bash\n", "...\n", "# Docker install (rootless)\n", "export PATH=/home/docker/bin:$PATH\n", "export DOCKER_HOST=unix:///run/user/1017/docker.sock\n", "```\n", "\n", "[rootless]: \n", "\n", "## Docker Cheatsheet\n", "\n", "Here are some useful commands:\n", "\n", "* `docker pull`: Pulls an image.\n", "* `docker create`: Creates a container from an image.\n", "* `docker start`: Starts running a container.\n", "* `docker stop`: Stops ...\n", "* `docker attach`: Attach to a running container.\n", "* `docker log`: See logs for specified container.\n", "* `docker ps -a`: List all containers (both running and stopped).\n", "* `docker images`: List all images.\n", "* `docker rm`: Remove a container.\n", "* `docker rmi`: Remove an image.\n", "* `docker inspect`: Lots of information about a container.\n", "* `docker exec -it /bin/bash`: Connect to the specified container and run `bash` (like ssh-ing into the VM).\n", "* `docker info`: Information about the install. You can use this to see if the\n", " `overlay2` storage driver is used, for example.\n", " \n", "These appear in documentation, but I do not use them:\n", "\n", "* `docker run`: This is [equivalent to `docker create` + `docker start` + `docker\n", " attach`](https://stackoverflow.com/questions/37744961/docker-run-vs-create). This can\n", " only be executed once. After the container is created, one cannot use subsequent\n", " calls to `run` to change, for example, port assignments. It is probably most useful\n", " for short foreground processes in conjunction with the `--rm` option." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Issues:**\n", "I originally had a bunch of errors because of interference with the previously installed\n", "docker version (not rootless). These went away once I did `sudo apt-get purge docker\n", "docker.io`.\n", "\n", "* `Aborting because rootful Docker is running and accessible. Set FORCE_ROOTLESS_INSTALL=1 to ignore.`\n", "* `Failed to start docker.service: Unit docker.socket failed to load: No such file or directory.`\n", "\n", "So I stopped the root docker service (from a previous install) and removed this file:\n", "\n", "```bash\n", "sudo service docker stop\n", "sudo systemctl disable docker\n", "sudo apt-get remove docker \n", "#sudo rm /var/run/docker.sock /lib/systemd/system/docker.service\n", "```\n", "\n", "After resolving these issues, I was having the following issue when trying to run the server with `systemctl`:\n", "\n", "```bash\n", "$ systemctl --user start docker\n", "$ docker run hello-world\n", "docker: Cannot connect to the Docker daemon at unix:///tmp/docker-1017/docker.sock. Is the docker daemon running?.\n", "See 'docker run --help'.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Limiting Resouces" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It can be useful to limit the resources available to Docker containers. To do this, you must enable cgroup swapping. This can be done by adding the following to `/etc/default/grub`:\n", "\n", "```bash\n", "# /etc/default/grub\n", "...\n", "GRUB_CMDLINE_LINUX_DEFAULT=\"swapaccount=1\"\n", "...\n", "```\n", "\n", "then running `sudo update-grub`. Once this is done, you can limit the resources used by Docker containers with flags like:\n", "\n", "```\n", "--memory=4g --cpus=1\n", "```\n", "\n", "etc.\n", "\n", "* [How to limit memory and CPU for Docker Containers](https://www.serverlab.ca/tutorials/containers/docker/how-to-limit-memory-and-cpu-for-docker-containers/)\n", "* [What does `swapaccount=1` in GRUB_CMDLINE_LINUX_DEFAULT do?](https://unix.stackexchange.com/questions/531480/what-does-swapaccount-1-in-grub-cmdline-linux-default-do)\n", "* [Docker Resource Constraints](https://docs.docker.com/config/containers/resource_constraints/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Docker Compose" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sometimes you must run several processes and have them interact. For example, heptapod generally requires a mail-server like `sendmail`, but [the omnibus docker image does not contain sendmail](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2630). To combine two containers in a single environment, one typically uses [docker compose](https://docs.docker.com/compose/)... but this does not seem to work with rootless." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Heptapod" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Heptapod](https://heptapod.net) is a service providing Mercurial access to [GitLab CE](https://about.gitlab.com/install/?version=ce). When running the public server, we host it here:\n", "\n", "* http://swan.physics.wsu.edu:11080\n", "\n", "Here we describe how to run [Heptapod](https://heptapod.net) in a Docker container.\n", "This is a service based on [GitLab CE](https://about.gitlab.com/install/?version=ce)\n", "that provides a backend with issue tracking etc. for Mercurial. \n", "As above, I have created a `docker` user account on `swan`. First I login to this, then\n", "make some directories for the server data in `/data2/docker/heptapod`. \n", "Then I pull the docker image.\n", "\n", "```bash\n", "ssh docker@swan # Login to swan as docker\n", "echo 'export GITLAB_HOME=\"${HOME}/srv/heptapod' >> ~/.bashrc\n", "sudo mkdir -p /data2/docker # Make the data directory for docker\n", "sudo chown docker /data2/docker # Change owner...\n", "sudo chmod a-wrx,u+rwxs /data2/docker # ...and give appropriate permissions\n", "mkdir -p /data2/docker/heptapod # Now create the heptapod directory.\n", "sudo ln -s /data2/docker/heptapod /srv/ # Link it to /srv/heptapod...\n", "ln -s /data2/docker ~docker/srv # ...and to the docker home directory.\n", "```\n", "\n", "Now we pull the heptapod image and start a couple of containers:\n", "\n", "* `heptapod-local`: Only listens on local ports. To use this, users must login with ssh and forward ports appropriately so they can connect (see below).\n", "* `heptapod-public`: Listens on public ports. This exposes Heptapod to the world, which may be a security risk. We do this to allow \"weak\" collaborators access, or to enable transferring repositories from Bitbucket.\n", "\n", "```bash\n", "docker pull octobus/heptapod\n", "docker create \\\n", " --name heptapod-local \\\n", " --restart always \\\n", " --hostname localhost \\\n", " --publish 127.0.0.1:11080:80 \\\n", " --publish 127.0.0.1:11443:443 \\\n", " --publish 127.0.0.1:11022:22 \\\n", " --volume ${GITLAB_HOME}/config:/etc/gitlab \\\n", " --volume ${GITLAB_HOME}/logs:/var/log/gitlab \\\n", " --volume ${GITLAB_HOME}/data:/var/opt/gitlab \\\n", " octobus/heptapod\n", "\n", "docker create \\\n", " --name heptapod-public \\\n", " --restart always \\\n", " --hostname swan.physics.wsu.edu \\\n", " --publish 11080:80 \\\n", " --publish 11443:443 \\\n", " --publish 11022:22 \\\n", " --volume ${GITLAB_HOME}/config:/etc/gitlab \\\n", " --volume ${GITLAB_HOME}/logs:/var/log/gitlab \\\n", " --volume ${GITLAB_HOME}/data:/var/opt/gitlab \\\n", " octobus/heptapod\n", "```\n", "\n", "Now we can run whichever one we want:\n", "\n", "```bash\n", "docker start heptapod-local # Use this in general.\n", "#docker start heptapod-public # Use this when needed.\n", "```\n", "\n", "Once started, I initialized a mercurial repository in the configuration directory so I\n", "can keep track of configuration changes:\n", "\n", "```bash\n", "cd ~/srv/heptapod\n", "# For some reason, the following file is given rw permission only\n", "# for the user in the docker image, so we can't back it up...\n", "sudo chgrp docker config/heptapod.hgrc\n", "sudo chmod g+rw config/heptapod.hgrc\n", "hg init\n", "cat > .hgignore <\n", "$ ssh swandocker\n", "\n", "docker@swan:~$ docker ps\n", "CONTAINER ID IMAGE ... PORTS NAMES\n", "638b8e2d6535 octobus/heptapod 0.0.0.0:11022->22/tcp, 0.0.0.0:11080->80/tcp, 0.0.0.0:11443->443/tcp heptapod-public\n", "docker@swan:~$ export heptapod_id=heptapod-public\n", "docker@swan:~$ docker exec -t ${heptapod_id} gitlab-rake gitlab:env:info\n", "\n", "System information\n", "System:\t\t\n", "Current User:\tgit\n", "Using RVM:\tno\n", "Ruby Version:\t2.6.6p146\n", "Gem Version:\t2.7.10\n", "Bundler Version:1.17.3\n", "Rake Version:\t12.3.3\n", "Redis Version:\t5.0.9\n", "Git Version:\t2.28.0\n", "Sidekiq Version:5.2.9\n", "Go Version:\tunknown\n", "\n", "GitLab information\n", "Version:\t13.4.6\n", "Revision:\t7ea571d2b76a\n", "Directory:\t/opt/gitlab/embedded/service/gitlab-rails\n", "DB Adapter:\tPostgreSQL\n", "DB Version:\t11.9\n", "URL:\t\thttp://swan.physics.wsu.edu\n", "HTTP Clone URL:\thttp://swan.physics.wsu.edu/some-group/some-project.git\n", "SSH Clone URL:\tssh://git@swan.physics.wsu.edu:11022/some-group/some-project.git\n", "Using LDAP:\tno\n", "Using Omniauth:\tyes\n", "Omniauth Providers: bitbucket\n", "\n", "GitLab Shell\n", "Version:\t13.7.0\n", "Repository storage paths:\n", "- default: \t/var/opt/gitlab/git-data/repositories\n", "GitLab Shell path:\t\t/opt/gitlab/embedded/service/gitlab-shell\n", "Git:\t\t/opt/gitlab/embedded/bin/git\n", "docker@swan:~$ docker exec -t ${heptapod_id} gitlab-backup create\n", "docker@swan:~$ docker exec -t ${heptapod_id} gitlab-backup create\n", "2020-12-04 08:17:52 +0000 -- Dumping database ... \n", "Dumping PostgreSQL database gitlabhq_production ... [DONE]\n", "2020-12-04 08:17:57 +0000 -- done\n", "2020-12-04 08:17:57 +0000 -- Dumping namespaces (Mercurial specific) ...\n", "Calling tar czf /var/opt/gitlab/backups/namespaces.tar.gz -C /var/opt/gitlab/backups/namespaces .\n", "output=, status=0\n", "2020-12-04 08:17:59 +0000 -- done\n", "2020-12-04 08:17:59 +0000 -- Dumping repositories ...\n", " * mforbes/pymmf (@hashed/6b/...) ... \n", " * mforbes/pymmf (@hashed/6b/...) ... [DONE]\n", " * mforbes/pymmf (@hashed/6b/...) ... [DONE] Wiki\n", " * mforbes/kzsolve (@hashed/d4/...) ... \n", "...\n", " * mforbes/zdk_phys_final_eos_fast (@hashed/52/...) ... \n", " * mforbes/zdk_phys_final_eos_fast (@hashed/52/...) ... [DONE]\n", " * mforbes/zdk_phys_final_eos_fast (@hashed/52/...) ... [SKIPPED] Wiki\n", "2020-12-04 08:25:57 +0000 -- done\n", "2020-12-04 08:25:57 +0000 -- Dumping uploads ... \n", "2020-12-04 08:25:58 +0000 -- done\n", "2020-12-04 08:25:58 +0000 -- Dumping builds ... \n", "2020-12-04 08:25:58 +0000 -- done\n", "2020-12-04 08:25:58 +0000 -- Dumping artifacts ... \n", "2020-12-04 08:25:58 +0000 -- done\n", "2020-12-04 08:25:58 +0000 -- Dumping pages ... \n", "2020-12-04 08:25:58 +0000 -- done\n", "2020-12-04 08:25:58 +0000 -- Dumping lfs objects ... \n", "2020-12-04 08:25:58 +0000 -- done\n", "2020-12-04 08:25:58 +0000 -- Dumping container registry images ... \n", "2020-12-04 08:25:58 +0000 -- [DISABLED]\n", "Creating backup archive: 1607070358_2020_12_04_13.4.6_gitlab_backup.tar ... done\n", "Uploading backup archive to remote storage ... skipped\n", "Deleting tmp directories ... done\n", "done\n", "...\n", "Deleting old backups ... skipping\n", "Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data \n", "and are not included in this backup. You will need these files to restore a backup.\n", "Please back them up manually.\n", "Backup task is done.\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n",
    "$ ssh aws-hg\n",
    "[ec2-user@ip-10-147-105-73 ~]$ docker ps\n",
    "CONTAINER ID ... PORTS                                             NAMES\n",
    "650311e25626 ... 0.0.0.0:80->80/tcp, 22/tcp, 0.0.0.0:443->443/tcp  ecs-heptapod-11-heptapod-f0effff7e3cddbc38001\n",
    "...\n",
    "[ec2-user@... ~]$ export heptapod_id=\"ecs-heptapod-11-heptapod-f0effff7e3cddbc38001\"\n",
    "[ec2-user@... ~]$ docker exec -it ${heptapod_id} gitlab-rake gitlab:env:info\n",
    "\n",
    "System information\n",
    "System:\t\t\n",
    "Current User:\tgit\n",
    "Using RVM:\tno\n",
    "Ruby Version:\t2.6.6p146\n",
    "Gem Version:\t2.7.10\n",
    "Bundler Version:1.17.3\n",
    "Rake Version:\t12.3.3\n",
    "Redis Version:\t5.0.9\n",
    "Git Version:\t2.28.0\n",
    "Sidekiq Version:5.2.9\n",
    "Go Version:\tunknown\n",
    "\n",
    "GitLab information\n",
    "Version:\t13.3.5\n",
    "Revision:\t11c5a0a148c9\n",
    "Directory:\t/opt/gitlab/embedded/service/gitlab-rails\n",
    "DB Adapter:\tPostgreSQL\n",
    "DB Version:\t11.7\n",
    "URL:\t\thttp://650311e25626\n",
    "HTTP Clone URL:\thttp://650311e25626/some-group/some-project.git\n",
    "SSH Clone URL:\tssh://git@650311e25626/some-group/some-project.git\n",
    "Using LDAP:\tno\n",
    "Using Omniauth:\tyes\n",
    "Omniauth Providers: \n",
    "\n",
    "GitLab Shell\n",
    "Version:\t13.6.0\n",
    "Repository storage paths:\n",
    "- default: \t/var/opt/gitlab/git-data/repositories\n",
    "GitLab Shell path:\t\t/opt/gitlab/embedded/service/gitlab-shell\n",
    "Git:\t\t/opt/gitlab/embedded/bin/git\n",
    "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### HTTP Redirect\n", "\n", "**Note: [SSL does not yet work with non-standard ports](https://docs.gitlab.com/omnibus/settings/ssl.html)... so I am using HTTP only.**\n", "I have randomly chosen ports 11080, 11443 and 11022 for HTTP, HTTPS, and SSH access. These are not very memorable, so it would be nice to redirect `https://swan.physics.wsu.edu/heptapod` to `https://swan.physics.wsu.edu:11443`. To do this, we simply add a `Redirect /heptapod https://swan.physics.wsu.edu:11443/` statement to one of the Apache config files:\n", "\n", "```bash\n", "#/etc/apache2/sites-enabled/000-default.conf \n", "\n", " ...\n", " Redirect /heptapod http://swan.physics.wsu.edu:11080/\n", " Redirect /discourse http://swan.physics.wsu.edu:10080/ \n", "\n", "```\n", "```bash\n", "#/etc/apache2/sites-enabled/000-default-le-ssl.conf \n", "\n", " ...\n", " Redirect /heptapod https://swan.physics.wsu.edu:11443/\n", " Redirect /discourse https://swan.physics.wsu.edu:10443/ \n", "\n", "```\n", "\n", "Don't forget to restart the server:\n", "\n", "```bash\n", "sudo service apache2 restart\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Bitbucket Import" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. [Enable OAuth2 integration on Bitbucket](https://docs.gitlab.com/ce/integration/bitbucket.html). I used my public settings `http://swan.physics.wsu.edu/heptapod/users/sign_in`.\n", "\n", " *Note: I had issues resulting in `redirect_uri` issue because I was using my alias `http://swan.physics.wsu.edu/heptapod/users/auth` but in my configuration, I used the `http://swan.physics.wsu.edu:9080/users/auth` form. If you look at the URL sent, it includes the `redirect_uri` which must match.*\n", " \n", "2. Edit the `/etc/gitlab/gitlab.rb` file on the server. Since we mapped `/etc/gitlab` to `~docker/srv/config`, we can edit it there without connecting.\n", " ```bash\n", " vi ~docker/srv/heptapod/config/gitlab.rb\n", " ```\n", " ```bash\n", " #/etc/gitlab/gitlab.rb\n", " gitlab_rails['omniauth_enabled'] = true\n", " ...\n", " gitlab_rails['omniauth_providers'] = [\n", " {\n", " \"name\" => \"bitbucket\",\n", " \"app_id\" => \"Rc...\",\n", " \"app_secret\" => \"hA...\",\n", " \"url\" => \"https://bitbucket.org/\",\n", " }\n", " ]\n", " ```\n", "3. Start the public server, or reconfigure GitLab:\n", "\n", " ```bash\n", " ssh docker@swan\n", " docker start heptapod-public\n", " ```\n", " \n", " or\n", " \n", " ```bash\n", " docker exec -it heptapod-public gitlab-ctl reconfigure\n", " ```\n", "4. Register for an account on our Heptapod instance.\n", "5. Login.\n", "6. Import new project from Bitbucket Cloud." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### References\n", "\n", "* [Omnibus GitLab Instructions](https://docs.gitlab.com/omnibus/docker/): These are the instructions for running the GitLab Docker container. The Heptapod container is based on this, so should function similarly.\n", "* [Heptapod as a Bitbucket replacement](https://heptapod.net/heptapod-as-a-bitbucket-replacement.html): Instructions on how to import projects from bitbucket.\n", "* [Backup/Restore](https://docs.gitlab.com/ee/raketasks/backup_restore.html): Also the way to migrate from one server to another." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Issues" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Some imports are broken.\n", "* Cloning links are incorrect: `http://swan.physics.wsu.edu/mforbes/mmfutils mmfutils_heptapod`. Probably need to update the hostname to include the port and/or the `/heptapode` alias.\n", "* Cloning from `http://swan.physics.wsu.edu:9080/mforbes/mmfutils mmfutils_heptapod` does not work.\n", "* Cloning from `ssh://git@swan.physics.wsu.edu:9022/mforbes/mmfutils` works on `swan` but not from outside.\n", "* Cloning from `ssh://git@localhost:9022/mforbes/mmfutils` works with SSH tunnel.\n", "\n", "#### Password Changes\n", "On swan, we do not have a mail server setup, so if you want to recover a password, then you need to [follow this procedure](https://docs.gitlab.com/ee/security/reset_user_password.html):\n", "\n", "```bash\n", "ssh swandocker\n", "docker exec -it heptapod-local /bin/bash\n", "gitlab-rails console -e production\n", "user = User.where(id: 2).first # Play with the number to find the user you want\n", "user.password = 'secret_pass'\n", "user.password_confirmation = 'secret_pass'\n", "user.admin = true # If you want user to be admin.\n", "user.save!\n", "```\n", "\n", "#### Upgrades/Migration\n", "\n", "GitLab does not allow you to jump to much between versions, so you might need to specify an upgrade path. This means you should recreate the container with a sequence of images, then let the backend migration take place. Here is an example:\n", "\n", "1. First [find out which tags are available](https://fordodone.com/2015/10/02/docker-get-list-of-tags-in-repository/):\n", "\n", "```bash\n", "$ wget -q https://registry.hub.docker.com/v1/repositories/octobus/heptapod/tags -O - | sed -e 's/[][]//g' -e 's/\"//g' -e 's/ //g' | tr '}' '\\n' | awk -F: '{print $3}'\n", "latest\n", "...\n", "0.16.0 # <<<<<<<<<< This matched our aws instance\n", "0.16.0-py2\n", "0.16.0-py3\n", "0.16.0rc1\n", "0.16.0rc1-py2\n", "0.16.0rc1-py3\n", "0.16.1\n", "0.16.1-py2\n", "0.16.1-py3\n", "0.16.2\n", "0.16.2-py2\n", "0.16.2-py3\n", "0.17.0\n", "0.17.0rc1\n", "0.17.1\n", "0.17.2\n", "...\n", "```\n", "\n", "\n", "To [see if the migration is done](https://docs.gitlab.com/ce/update/#checking-for-background-migrations-before-upgrading):\n", "\n", "```bash\n", "ssh swandocker\n", "docker exec -it heptapod-local gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Discourse" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "ssh docker@swan\n", "mkdir repositories\n", "cd repositories\n", "git clone https://github.com/discourse/discourse_docker.git\n", "cd discourse_docker\n", "```\n", "\n", "Edit the generated `containers/app.yml` file. I am trying to use Gmail with a custom alias `michael.forbes+discourse@gmail.com` which I registered under my [Gmail account settings](https://mail.google.com/mail/u/0/#settings/accounts).\n", "\n", "```yaml\n", " DISCOURSE_DEVELOPER_EMAILS: 'michael.forbes+discourse@gmail.com'\n", " DISCOURSE_SMTP_ADDRESS: in-v3.mailjet.com\n", " DISCOURSE_SMTP_USER_NAME: ******\n", " DISCOURSE_SMTP_PASSWORD: *********\n", " \n", "## The Docker container is stateless; all data is stored in /shared\n", "volumes:\n", " - volume:\n", " host: /home/docker/srv/discourse/shared/standalone\n", " guest: /shared\n", " - volume:\n", " host: /home/docker/srv/discourse/shared/standalone/log/var-log\n", " guest: /var/log\n", " \n", "...\n", " ## If you want to set the 'From' email address for your first registration, uncomment and change:\n", " ## After getting the first signup email, re-comment the line. It only needs to run once.\n", " - exec: rails r \"SiteSetting.notification_email='m.forbes@wsu.edu'\"\n", "```\n", "\n", "**Mailjet**: I managed to \n", "\n", "*Notes:*\n", "* *I could not use `michael.forbes+discourse@gmail.com` as the SMTP user here since I cannot login to gmail with this.*\n", "* *Gmail did not work: probably have to use an App password since two-factor authentication is enabled.*\n", "* *I had to use an absolute path for the host `/home/docker`: `~` did not work.*\n", "* *I also tried using Mailjet with ` - exec: rails r \"SiteSetting.notification_email='m.forbes@wsu.edu'\"` but this did not seem to activate either (I was expecting Mailjet to send me an activation email to make sure...)*\n", "\n", "After editing these, I was able to continue after making these directories:\n", "\n", "```bash\n", "rm -rf ~/srv/discourse\n", "mkdir ~/srv/discourse\n", "```\n", "\n", "```bash\n", "./launcher rebuild app\n", "```\n", "\n", "**Not Working:** Discourse is running, but not able to send emails." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### HTTP Redirect\n", "\n", "I have randomly chosen ports 10080 and 10433 for HTTP and HTTPS access. These are not very memorable, so it would be nice to redirect `https://swan.physics.wsu.edu/discourse` to `https://swan.physics.wsu.edu:10443`. To do this, we simply add a `Redirect /discourse https://swan.physics.wsu.edu:10443/` statement to one of the Apache config files:\n", "\n", "```bash\n", "#/etc/apache2/sites-enabled/000-default.conf \n", "\n", " ...\n", " Redirect /heptapod https://swan.physics.wsu.edu:11443/\n", " Redirect /discourse https://swan.physics.wsu.edu:10443/ \n", "\n", "```\n", "\n", "Don't forget to restart the server:\n", "\n", "```bash\n", "sudo service apache2 restart\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**References:**\n", "\n", "* [Install Discourse in Under 30 Minutes](https://blog.discourse.org/2014/04/install-discourse-in-under-30-minutes/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CoCalc\n", "\n", "[CoCalc](cocalc.com) can also be [installed with\n", "docker](https://github.com/sagemathinc/cocalc-docker). I created the images with the\n", "following file:\n", "\n", "```bash\n", "#!/bin/bash\n", "# initialize_cocalc.sh\n", "docker create \\\n", " --name cocalc \\\n", " --restart always \\\n", " --hostname localhost \\\n", " --publish 127.0.0.1:9443:443 \\\n", " --publish 127.0.0.1:9022:22 \\\n", " --volume ~/srv/cocalc:/projects \\\n", " sagemathinc/cocalc\n", "```\n", "\n", "These listen on port 9443. Note: you must connect with https://localhost:9443, not with HTTP.\n", "\n", "Launch with:\n", "\n", "```bash\n", "$ ssh swandocker\n", "$ docker ps -a\n", "...\n", "c54a50cbac06 sagemathinc/cocalc ... cocalc\n", "... \n", "$ docker start cocalc\n", "```\n", "\n", "Now you can connect with appropriately forwarded ports. Make sure you have something like this in your `~/.ssh/config` file:\n", "\n", "```ini\n", "# ~/.ssh/config\n", "...\n", "Host swand\n", " Host swan.physics.wsu.edu\n", " User mforbes\n", " ForwardAgent yes\n", " # CoCalc https access\n", " LocalForward 9443 localhost:9443\n", "```\n", "\n", "Start this connection:\n", "\n", "```\n", "$ ssh swand\n", "```\n", "\n", "Now visit the server on your local computer with https://localhost:9443." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Toubleshooting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**See the logs:**\n", "\n", "```bash\n", "(Laptop)$ ssh swandocker\n", "(Swan)$ docker logs cocalc -f # Sometimes does not work... if not, try\n", "(Swan)$ docker exec -it cocalc tail -f /var/log/hub.log\n", "```\n", "\n", "\n", "**Make a user Admin:**\n", "\n", "```bash\n", "(Laptop) $ ssh swandocker\n", "(Swan)$ docker exec -it cocalc make-user-admin m.forbes@wsu.edu\n", "```\n", "### Issues\n", "\n", "* New project stuck on \"Loading...\" when running Rootless. Works when running Docker as root. See:\n", " * https://github.com/sagemathinc/cocalc-docker/issues/118\n", "* The Docker interface does not have GUI support for setting SSH keys, so you must\n", " manually create them in each project. (Use a Linux Terminal for example.)\n", " \n", " ```bash\n", " cd ~\n", " mkdir .ssh\n", " chmod 700 .ssh\n", " vi .ssh/authorized_keys\n", " ... paste in contents of ~/.ssh/id_cocalc.pub from host computer ...\n", " chmod 600 .ssh/authorized_keys\n", " ```\n", "\n", " In our `initialize_cocalc.sh` we forward the SSH port 22 on the container to port 9022\n", " on the host (but only listen locally for security), so you can connect in two ways:\n", " \n", " 1. Connect to the host with SSH, forwarding ports 9443 and 9022 to your local\n", " machine. Then you can connect to the web interface from your computer at\n", " https://localhost:9443 and you can SSH to the container by connecting to your local\n", " computer on port 9022. (Tested and works.)\n", " 2. Connect to the container by using SSH directly to connect first to the host, then\n", " to the container. (Not tested yet.)\n", "\n", "## [Nextcloud](https://nextcloud.com) (incomplete)\n", "Open source replacement for Google Cloud etc. There is a [docker image](https://github.com/nextcloud/docker).\n", "\n", "Incomplete because this needs MySQL etc. and I don't want to figure this out yet.\n", "\n", "```bash\n", "ssh swandocker\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Let's Encrypt and Certbot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Let's Encrypt](https://letsencrypt.org) uses [Certbot](https://certbot.eff.org) as a tool for enabling free site certification. This requires using a supported Ubuntu release.\n", "\n", "\n", "```bash\n", "sudo apt update\n", "sudo apt install software-properties-common\n", "sudo add-apt-repository universe\n", "#sudo add-apt-repository ppa:certbot/certbot\n", "sudo apt update\n", "sudo apt install certbot python3-certbot-apache\n", "sudo certbot --apache\n", "#sudo apt-add-repository -r ppa:certbot/certbot\n", "```\n", "\n", "The commented out commands were needed earlier ([pre Ubuntu 19.10](https://community.letsencrypt.org/t/certbot-install-fails-on-ubuntu-20-04/120924/2)) but should no longer be needed.\n", "Without removing this repo I ran into the [following error](https://stackoverflow.com/a/61853542/1088938)) during `apt update`:\n", "\n", "```\n", "E: The repository 'http://ppa.launchpad.net/certbot/certbot/ubuntu focal Release' does not have a Release file.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With BitBucket \"sunsetting\" their support for Mercurial, we needed to find a new option. Here we explore options for self-hosting." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Version Control Hosting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## [Kallithea](https://kallithea-scm.org) (incomplete)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Kallithea needs a database, so we make a directory for this in `/data/kalithea` and store it there.\n", "\n", "```bash\n", "sudo apt-get install npm\n", "su admin\n", "sudo apt-get install build-essential git python-pip python-virtualenv libffi-dev python-dev\n", "hg clone https://kallithea-scm.org/repos/kallithea\n", "cd kallithea\n", "conda create -n kallithea python=2\n", "conda activate kallithea\n", "pip install --upgrade .\n", "mkdir -p \"${CONDA_PREFIX}/etc/conda/activate.d\"\n", "echo \"export LC_ALL='C.UTF-8'\" >> \"${CONDA_PREFIX}/etc/conda/activate.d/env_vars.sh\"\n", "```\n", "\n", "```bash\n", "su admin\n", "conda activate kallithea\n", "mkdir /data/kallithea\n", "cd /data/kallithea\n", "kallithea-cli config-create my.ini\n", "kallithea-cli db-create -c my.ini #Created with root path `/data/kallithea`, user `mforbes` etc.\n", "kallithea-cli front-end-build\n", "gearbox serve -c my.ini\n", "```\n", "\n", "This runs Kallithea on port 5000 which could be accessed by users with ssh tunelling." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## [Heptapod](https://heptapod.net)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another alternative is Heptapod. This is a mercurial interface to a friendly fork of [GitLab CE](https://about.gitlab.com/install/?version=ce) intended to ultimately bring mercurial support to [GitLab](https://about.gitlab.com). This can be installed with Docker as discussed below.\n", "\n", "**As of 1 July 2020:**\n", "This is my primary alternative to Bitbucket hosted as discussed below. We will probably ultimately host this on an AWS instance.\n", "\n", "**As of 14 March 2020:**\n", "* Heptapod is [implemented under the hood with git](https://heptapod.net/pages/faq.html#faq).\n", "* [Heptapod 0.8](https://heptapod.net/heptapod-080-released.html) supports import of Bitbucket Pull Requests." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Podman\n", "\n", "An alternative to Docker is [Podman](https://podman.io). This behaves very similarly to Docker, but runs rootless natively." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "jupytext": { "formats": "ipynb,md:myst" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.8" }, "nikola": { "date": "2020-11-30 18:37:38 UTC-07:00", "description": "", "link": "", "slug": "docker", "tags": "", "title": "Docker", "type": "text" }, "toc": { "base_numbering": 1, "nav_menu": { "height": "246px", "width": "252px" }, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": { "height": "calc(100% - 180px)", "left": "10px", "top": "150px", "width": "234.796875px" }, "toc_section_display": "block", "toc_window_display": true }, "varInspector": { "cols": { "lenName": 16, "lenType": 16, "lenVar": 40 }, "kernels_config": { "python": { "delete_cmd_postfix": "", "delete_cmd_prefix": "del ", "library": "var_list.py", "varRefreshCmd": "print(var_dic_list())" }, "r": { "delete_cmd_postfix": ") ", "delete_cmd_prefix": "rm(", "library": "var_list.r", "varRefreshCmd": "cat(var_dic_list()) " } }, "types_to_exclude": [ "module", "function", "builtin_function_or_method", "instance", "_Feature" ], "window_display": false } }, "nbformat": 4, "nbformat_minor": 1 }