Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
Download

Draft Forbes Group Website (Build by Nikola). The official site is hosted at:

https://labs.wsu.edu/forbes

5910 views
License: GPL3
ubuntu2004
Tweet
Share
Share
Kernel: Python 3

Docker

Some notes about using Docker to host various applications like Heptapod, Discourse, CoCalc etc. on Linux and AWS.

Docker

Several packages (such as Heptapod and CoCalc) require a rather complete system, so are easiest to install using Docker containers. Here we discuss how to set these up. Where possible, we try to use Rootless mode, which prevents the need for providing docker with root access, but has limitations. Currently, Rootless works for Heptapod, but causes problems with CoCalc.

Docker Account

We generally work with a docker user, and keep all docker-specific files for this account:

sudo useradd -m docker
sudo usermod -aG sudo docker         # Enable sudo for docker.
sudo su docker
chsh -s /bin/bash
bash
curl -fsSL https://get.docker.com/rootless | sh

On my machine, I then add a swandocker alias to my ~/.ssh/config file, e.g.:

# ~/.ssh/config

Host *docker
  User docker
  ForwardAgent yes
  SetEnv LC_EDITOR=vi

Host swan*
  HostName swan.physics.wsu.edu

Host penguin*
  HostName penguin.physics.wsu.edu

Host *
  ForwardAgent yes
  SendEnv LC_HG_USERNAME
  SendEnv LC_GIT_USERNAME
  SendEnv LC_GIT_USEREMAIL
  AddressFamily inet
  # Force IPv4
  # https://www.electricmonk.nl/log/2014/09/24/
  #         ssh-port-forwarding-bind-cannot-assign-requested-address/

All of the following is done in this account:

ssh ssh-copy-id swandocker
ssh swandocker
...

Installing Docker (With Root Access)

ssh swandocker
sudo apt-get update
sudo apt-get upgrade   # Optional
sudo apt-get install uidmap dbus-user-session
sudo apt-get purge docker docker.io  # Remove old root-version of docker.
sudo apt-get autoremove --purge
sudo apt-get install docker docker.io nvidia-docker2
sudo systemctl start docker

sudo systemctl stop docker docker.sock

Installing Docker Rootless

Note: Be sure to completely purge any previous root-enabled version of Docker before proceeding.

ssh swandocker
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install uidmap dbus-user-session
sudo apt-get purge docker docker.io  # Remove old root-version of docker.
sudo apt-get autoremove --purge

To actually install rootless, run the following script, downloaded with curl:

curl -fsSL https://get.docker.com/rootless | sh

This will put everything in ~/bin

This allows the docker user to add processes to start services that will start at login.

sudo loginctl enable-linger docker
systemctl --user start docker

Add the appropriate environmental variables to ~docker/.bashrc:

...
# Docker install (rootless)
export PATH=/home/docker/bin:$PATH
export DOCKER_HOST=unix:///run/user/1017/docker.sock

Docker Cheatsheet

Here are some useful commands:

  • docker pull: Pulls an image.

  • docker create: Creates a container from an image.

  • docker start: Starts running a container.

  • docker stop: Stops ...

  • docker attach: Attach to a running container.

  • docker log: See logs for specified container.

  • docker ps -a: List all containers (both running and stopped).

  • docker images: List all images.

  • docker rm: Remove a container.

  • docker rmi: Remove an image.

  • docker inspect: Lots of information about a container.

  • docker exec -it <name> /bin/bash: Connect to the specified container and run bash (like ssh-ing into the VM).

  • docker info: Information about the install. You can use this to see if the overlay2 storage driver is used, for example.

These appear in documentation, but I do not use them:

  • docker run: This is equivalent to docker create + docker start + docker attach. This can only be executed once. After the container is created, one cannot use subsequent calls to run to change, for example, port assignments. It is probably most useful for short foreground processes in conjunction with the --rm option.

Issues: I originally had a bunch of errors because of interference with the previously installed docker version (not rootless). These went away once I did sudo apt-get purge docker docker.io.

  • Aborting because rootful Docker is running and accessible. Set FORCE_ROOTLESS_INSTALL=1 to ignore.

  • Failed to start docker.service: Unit docker.socket failed to load: No such file or directory.

So I stopped the root docker service (from a previous install) and removed this file:

sudo service docker stop
sudo systemctl disable docker
sudo apt-get remove docker 
#sudo rm /var/run/docker.sock /lib/systemd/system/docker.service

After resolving these issues, I was having the following issue when trying to run the server with systemctl:

$ systemctl --user start docker
$ docker run hello-world
docker: Cannot connect to the Docker daemon at unix:///tmp/docker-1017/docker.sock. Is the docker daemon running?.
See 'docker run --help'.

Limiting Resouces

It can be useful to limit the resources available to Docker containers. To do this, you must enable cgroup swapping. This can be done by adding the following to /etc/default/grub:

# /etc/default/grub
...
GRUB_CMDLINE_LINUX_DEFAULT="swapaccount=1"
...

then running sudo update-grub. Once this is done, you can limit the resources used by Docker containers with flags like:

--memory=4g --cpus=1

etc.

Docker Compose

Sometimes you must run several processes and have them interact. For example, heptapod generally requires a mail-server like sendmail, but the omnibus docker image does not contain sendmail. To combine two containers in a single environment, one typically uses docker compose... but this does not seem to work with rootless.

Heptapod

Heptapod is a service providing Mercurial access to GitLab CE. When running the public server, we host it here:

Here we describe how to run Heptapod in a Docker container. This is a service based on GitLab CE that provides a backend with issue tracking etc. for Mercurial. As above, I have created a docker user account on swan. First I login to this, then make some directories for the server data in /data2/docker/heptapod. Then I pull the docker image.

ssh docker@swan                         # Login to swan as docker
echo 'export GITLAB_HOME="${HOME}/srv/heptapod' >> ~/.bashrc
sudo mkdir -p /data2/docker             # Make the data directory for docker
sudo chown docker /data2/docker         # Change owner...
sudo chmod a-wrx,u+rwxs /data2/docker   # ...and give appropriate permissions
mkdir -p /data2/docker/heptapod         # Now create the heptapod directory.
sudo ln -s /data2/docker/heptapod /srv/ # Link it to /srv/heptapod...
ln -s /data2/docker ~docker/srv         # ...and to the docker home directory.

Now we pull the heptapod image and start a couple of containers:

  • heptapod-local: Only listens on local ports. To use this, users must login with ssh and forward ports appropriately so they can connect (see below).

  • heptapod-public: Listens on public ports. This exposes Heptapod to the world, which may be a security risk. We do this to allow "weak" collaborators access, or to enable transferring repositories from Bitbucket.

docker pull octobus/heptapod
docker create                                            \
       --name heptapod-local                             \
       --restart always                                  \
       --hostname localhost                              \
       --publish 127.0.0.1:11080:80                       \
       --publish 127.0.0.1:11443:443                      \
       --publish 127.0.0.1:11022:22                       \
       --volume ${GITLAB_HOME}/config:/etc/gitlab        \
       --volume ${GITLAB_HOME}/logs:/var/log/gitlab      \
       --volume ${GITLAB_HOME}/data:/var/opt/gitlab      \
       octobus/heptapod

docker create                                            \
       --name heptapod-public                            \
       --restart always                                  \
       --hostname swan.physics.wsu.edu                   \
       --publish 11080:80                                 \
       --publish 11443:443                                \
       --publish 11022:22                                 \
       --volume ${GITLAB_HOME}/config:/etc/gitlab        \
       --volume ${GITLAB_HOME}/logs:/var/log/gitlab      \
       --volume ${GITLAB_HOME}/data:/var/opt/gitlab      \
       octobus/heptapod

Now we can run whichever one we want:

docker start heptapod-local       # Use this in general.
#docker start heptapod-public      # Use this when needed.

Once started, I initialized a mercurial repository in the configuration directory so I can keep track of configuration changes:

cd ~/srv/heptapod
# For some reason, the following file is given rw permission only
# for the user in the docker image, so we can't back it up...
sudo chgrp docker config/heptapod.hgrc
sudo chmod g+rw config/heptapod.hgrc
hg init
cat > .hgignore <<EOF
syntax: glob

data/
logs/
EOF
hg add
hg com -m "Initial commit"

Debugging

Look at the current logs with the following:

docker exec -it heptapod-public gitlab-ctl tail

Heptapod Backup (incomplete)

The recommended way to transfer a Heptapod/GitLab server is backup/restore.

To do this:

  1. Login to your host and find the name of your heptapod container:

    ssh aws-hg
docker ps
heptapod_id=ecs-heptapod-11-heptapod-f0effff7e3cddbc38001

    or

    ssh swandocker
docker ps
heptapod_id=heptapod-public

  2. First make sure both instances are running the same version:

    docker exec -t ${heptapod_id} gitlab-rake gitlab:env:info

    If not, update and migrate until they match, converting the database as needed.

  3. Create the backup on the source:

    docker exec -t ${heptapod_id} gitlab-backup create
``

This will put a file on the image which we exported:

* `/var/opt/gitlab/backups/1593678341_2020_07_02_12.10.11_gitlab_backup.tar`
* `~/srv/heptapod/data/backups/1593678341_2020_07_02_12.10.11_gitlab_backup.tar`

Optionally, the backup program can upload this to a remote storage location.

  4. Backup the configuration files:

    tar -czvf heptapod-config-2020-12-13.tgz ~/srv/heptapod/config

  5. Copy everything over to new server:

    backup=1607073575_2020_12_04_13.3.5_gitlab_backup.tar
mkdir heptapod_backup
cd heptapod_backup
scp -r swandocker:srv/heptapod/config .

# Need to change owner of this file first on swan with sudo
scp -r swandocker:srv/heptapod/data/backup/${backup} .
cd ..
scp -r heptapod_backup/ aws-hg:
ssh aws-hg
heptapod_id=ecs-heptapod-11-heptapod-f0effff7e3cddbc38001
docker cp heptapod_backup ${heptapod_id}:heptapod_backup
docker exec -it bash ${heptapod_id}
cd heptapod_backup
cp -r /etc/gitlab /etc/gitlab-backup
cp -r /heptapod_backup/config/* /etc/gitlab/
cp /etc/gitlab-backup/gitlab.rb /etc/gitlab/
mv 1607073575_2020_12_04_13.3.5_gitlab_backup.tar /var/opt/gitlab/backups/
chown git.git /var/opt/gitlab/backups/1607073575_2020_12_04_13.3.5_gitlab_backup.tar
gitlab-ctl reconfigure
gitlab-ctl stop unicorn
gitlab-ctl stop puma
gitlab-ctl stop sidekiq
gitlab-ctl status
gitlab-backup restore BACKUP=1607073575_2020_12_04_13.3.5
gitlab-ctl reconfigure
gitlab-ctl restart
gitlab-rake gitlab:check SANITIZE=true

Another option is to backup the repositories, I use rclone to copy these to my Google Drive to a remote called gwsu_backups from my docker account on swan using the root_folder_id corresponding to a folder My Drive/backups/RClone/swan/repo_backup.

screen -S RCloneBackup
rclone sync -Pl repo_backups gwsu_backups:


$ ssh swandocker

docker@swan:~$ docker ps
CONTAINER ID        IMAGE ...         PORTS                                                                          NAMES
638b8e2d6535        octobus/heptapod  0.0.0.0:11022->22/tcp, 0.0.0.0:11080->80/tcp, 0.0.0.0:11443->443/tcp   heptapod-public
docker@swan:~$ export heptapod_id=heptapod-public
docker@swan:~$ docker exec -t ${heptapod_id} gitlab-rake gitlab:env:info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.4.6
Revision:	7ea571d2b76a
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.9
URL:		http://swan.physics.wsu.edu
HTTP Clone URL:	http://swan.physics.wsu.edu/some-group/some-project.git
SSH Clone URL:	ssh://[email protected]:11022/some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: bitbucket

GitLab Shell
Version:	13.7.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git
docker@swan:~$ docker exec -t ${heptapod_id} gitlab-backup create
docker@swan:~$ docker exec -t ${heptapod_id} gitlab-backup create
2020-12-04 08:17:52 +0000 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2020-12-04 08:17:57 +0000 -- done
2020-12-04 08:17:57 +0000 -- Dumping namespaces (Mercurial specific) ...
Calling tar czf /var/opt/gitlab/backups/namespaces.tar.gz -C /var/opt/gitlab/backups/namespaces .
output=, status=0
2020-12-04 08:17:59 +0000 -- done
2020-12-04 08:17:59 +0000 -- Dumping repositories ...
 * mforbes/pymmf (@hashed/6b/...) ... 
 * mforbes/pymmf (@hashed/6b/...) ... [DONE]
 * mforbes/pymmf (@hashed/6b/...) ... [DONE] Wiki
 * mforbes/kzsolve (@hashed/d4/...) ... 
...
 * mforbes/zdk_phys_final_eos_fast (@hashed/52/...) ... 
 * mforbes/zdk_phys_final_eos_fast (@hashed/52/...) ... [DONE]
 * mforbes/zdk_phys_final_eos_fast (@hashed/52/...) ... [SKIPPED] Wiki
2020-12-04 08:25:57 +0000 -- done
2020-12-04 08:25:57 +0000 -- Dumping uploads ... 
2020-12-04 08:25:58 +0000 -- done
2020-12-04 08:25:58 +0000 -- Dumping builds ... 
2020-12-04 08:25:58 +0000 -- done
2020-12-04 08:25:58 +0000 -- Dumping artifacts ... 
2020-12-04 08:25:58 +0000 -- done
2020-12-04 08:25:58 +0000 -- Dumping pages ... 
2020-12-04 08:25:58 +0000 -- done
2020-12-04 08:25:58 +0000 -- Dumping lfs objects ... 
2020-12-04 08:25:58 +0000 -- done
2020-12-04 08:25:58 +0000 -- Dumping container registry images ... 
2020-12-04 08:25:58 +0000 -- [DISABLED]
Creating backup archive: 1607070358_2020_12_04_13.4.6_gitlab_backup.tar ... done
Uploading backup archive to remote storage  ... skipped
Deleting tmp directories ... done
done
...
Deleting old backups ... skipping
Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
and are not included in this backup. You will need these files to restore a backup.
Please back them up manually.
Backup task is done.


$ ssh aws-hg
[ec2-user@ip-10-147-105-73 ~]$ docker ps
CONTAINER ID ... PORTS                                             NAMES
650311e25626 ... 0.0.0.0:80->80/tcp, 22/tcp, 0.0.0.0:443->443/tcp  ecs-heptapod-11-heptapod-f0effff7e3cddbc38001
...
[ec2-user@... ~]$ export heptapod_id="ecs-heptapod-11-heptapod-f0effff7e3cddbc38001"
[ec2-user@... ~]$ docker exec -it ${heptapod_id} gitlab-rake gitlab:env:info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.3.5
Revision:	11c5a0a148c9
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		http://650311e25626
HTTP Clone URL:	http://650311e25626/some-group/some-project.git
SSH Clone URL:	ssh://git@650311e25626/some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	13.6.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

HTTP Redirect

Note: SSL does not yet work with non-standard ports... so I am using HTTP only. I have randomly chosen ports 11080, 11443 and 11022 for HTTP, HTTPS, and SSH access. These are not very memorable, so it would be nice to redirect https://swan.physics.wsu.edu/heptapod to https://swan.physics.wsu.edu:11443. To do this, we simply add a Redirect /heptapod https://swan.physics.wsu.edu:11443/ statement to one of the Apache config files:

#/etc/apache2/sites-enabled/000-default.conf 
<Virtualhost *:80>
    ...
    Redirect /heptapod http://swan.physics.wsu.edu:11080/
    Redirect /discourse http://swan.physics.wsu.edu:10080/    
</VirtualHost>

#/etc/apache2/sites-enabled/000-default-le-ssl.conf 
<Virtualhost *:443>
    ...
    Redirect /heptapod https://swan.physics.wsu.edu:11443/
    Redirect /discourse https://swan.physics.wsu.edu:10443/    
</VirtualHost>

Don't forget to restart the server:

sudo service apache2 restart

Bitbucket Import

  1. Enable OAuth2 integration on Bitbucket. I used my public settings http://swan.physics.wsu.edu/heptapod/users/sign_in.

Note: I had issues resulting in redirect_uri issue because I was using my alias http://swan.physics.wsu.edu/heptapod/users/auth but in my configuration, I used the http://swan.physics.wsu.edu:9080/users/auth form. If you look at the URL sent, it includes the redirect_uri which must match.

  1. Edit the /etc/gitlab/gitlab.rb file on the server. Since we mapped /etc/gitlab to ~docker/srv/config, we can edit it there without connecting.

vi ~docker/srv/heptapod/config/gitlab.rb

#/etc/gitlab/gitlab.rb
gitlab_rails['omniauth_enabled'] = true
...
gitlab_rails['omniauth_providers'] = [
  {
    "name" => "bitbucket",
    "app_id" => "Rc...",
    "app_secret" => "hA...",
    "url" => "https://bitbucket.org/",
  }
]

  1. Start the public server, or reconfigure GitLab:

ssh docker@swan
docker start heptapod-public

or

docker exec -it heptapod-public gitlab-ctl reconfigure

  1. Register for an account on our Heptapod instance.

  2. Login.

  3. Import new project from Bitbucket Cloud.

References

Issues

  • Some imports are broken.

  • Cloning links are incorrect: http://swan.physics.wsu.edu/mforbes/mmfutils mmfutils_heptapod. Probably need to update the hostname to include the port and/or the /heptapode alias.

  • Cloning from http://swan.physics.wsu.edu:9080/mforbes/mmfutils mmfutils_heptapod does not work.

  • Cloning from ssh://[email protected]:9022/mforbes/mmfutils works on swan but not from outside.

  • Cloning from ssh://git@localhost:9022/mforbes/mmfutils works with SSH tunnel.

Password Changes

On swan, we do not have a mail server setup, so if you want to recover a password, then you need to follow this procedure:

ssh swandocker
docker exec -it heptapod-local /bin/bash
gitlab-rails console -e production
user = User.where(id: 2).first    # Play with the number to find the user you want
user.password = 'secret_pass'
user.password_confirmation = 'secret_pass'
user.admin = true  # If you want user to be admin.
user.save!

Upgrades/Migration

GitLab does not allow you to jump to much between versions, so you might need to specify an upgrade path. This means you should recreate the container with a sequence of images, then let the backend migration take place. Here is an example:

  1. First find out which tags are available:

$ wget -q https://registry.hub.docker.com/v1/repositories/octobus/heptapod/tags -O -  | sed -e 's/[][]//g' -e 's/"//g' -e 's/ //g' | tr '}' '\n'  | awk -F: '{print $3}'
latest
...
0.16.0               # <<<<<<<<<< This matched our aws instance
0.16.0-py2
0.16.0-py3
0.16.0rc1
0.16.0rc1-py2
0.16.0rc1-py3
0.16.1
0.16.1-py2
0.16.1-py3
0.16.2
0.16.2-py2
0.16.2-py3
0.17.0
0.17.0rc1
0.17.1
0.17.2
...

To see if the migration is done:

ssh swandocker
docker exec -it heptapod-local gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

Discourse

ssh docker@swan
mkdir repositories
cd repositories
git clone https://github.com/discourse/discourse_docker.git
cd discourse_docker

Edit the generated containers/app.yml file. I am trying to use Gmail with a custom alias [email protected] which I registered under my Gmail account settings.

  DISCOURSE_DEVELOPER_EMAILS: '[email protected]'
  DISCOURSE_SMTP_ADDRESS: in-v3.mailjet.com
  DISCOURSE_SMTP_USER_NAME: ******
  DISCOURSE_SMTP_PASSWORD: *********
  
## The Docker container is stateless; all data is stored in /shared
volumes:
  - volume:
      host: /home/docker/srv/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /home/docker/srv/discourse/shared/standalone/log/var-log
      guest: /var/log
      
...
  ## If you want to set the 'From' email address for your first registration, uncomment and change:
  ## After getting the first signup email, re-comment the line. It only needs to run once.
  - exec: rails r "SiteSetting.notification_email='[email protected]'"

Mailjet: I managed to

Notes:

  • I could not use [email protected] as the SMTP user here since I cannot login to gmail with this.

  • Gmail did not work: probably have to use an App password since two-factor authentication is enabled.

  • I had to use an absolute path for the host /home/docker: ~ did not work.

  • I also tried using Mailjet with - exec: rails r "SiteSetting.notification_email='[email protected]'" but this did not seem to activate either (I was expecting Mailjet to send me an activation email to make sure...)

After editing these, I was able to continue after making these directories:

rm -rf ~/srv/discourse
mkdir ~/srv/discourse

./launcher rebuild app

Not Working: Discourse is running, but not able to send emails.

HTTP Redirect

I have randomly chosen ports 10080 and 10433 for HTTP and HTTPS access. These are not very memorable, so it would be nice to redirect https://swan.physics.wsu.edu/discourse to https://swan.physics.wsu.edu:10443. To do this, we simply add a Redirect /discourse https://swan.physics.wsu.edu:10443/ statement to one of the Apache config files:

#/etc/apache2/sites-enabled/000-default.conf 
<Virtualhost *:443>
    ...
    Redirect /heptapod https://swan.physics.wsu.edu:11443/
    Redirect /discourse https://swan.physics.wsu.edu:10443/    
</VirtualHost>

Don't forget to restart the server:

sudo service apache2 restart

CoCalc

CoCalc can also be installed with docker. I created the images with the following file:

#!/bin/bash
# initialize_cocalc.sh
docker create                                            \
       --name cocalc                                     \
       --restart always                                  \
       --hostname localhost                              \
       --publish 127.0.0.1:9443:443                      \
       --publish 127.0.0.1:9022:22                       \
       --volume ~/srv/cocalc:/projects                   \
       sagemathinc/cocalc

These listen on port 9443. Note: you must connect with https://localhost:9443, not with HTTP.

Launch with:

$ ssh swandocker
$ docker ps -a
...
c54a50cbac06        sagemathinc/cocalc   ...  cocalc
... 
$ docker start cocalc

Now you can connect with appropriately forwarded ports. Make sure you have something like this in your ~/.ssh/config file:

# ~/.ssh/config
...
Host swand
  Host swan.physics.wsu.edu
  User mforbes
  ForwardAgent yes
  # CoCalc https access
  LocalForward 9443 localhost:9443

Start this connection:

$ ssh swand

Now visit the server on your local computer with https://localhost:9443.

Toubleshooting

See the logs:

(Laptop)$ ssh swandocker
(Swan)$ docker logs cocalc -f       # Sometimes does not work... if not, try
(Swan)$ docker exec -it cocalc tail -f /var/log/hub.log

Make a user Admin:

(Laptop) $ ssh swandocker
(Swan)$ docker exec -it cocalc make-user-admin m.forbes@wsu.edu

Issues

  • New project stuck on "Loading..." when running Rootless. Works when running Docker as root. See:

  • The Docker interface does not have GUI support for setting SSH keys, so you must manually create them in each project. (Use a Linux Terminal for example.)

    cd ~
mkdir .ssh
chmod 700 .ssh
vi .ssh/authorized_keys
... paste in contents of ~/.ssh/id_cocalc.pub from host computer ...
chmod 600 .ssh/authorized_keys

    In our initialize_cocalc.sh we forward the SSH port 22 on the container to port 9022 on the host (but only listen locally for security), so you can connect in two ways:

    1. Connect to the host with SSH, forwarding ports 9443 and 9022 to your local machine. Then you can connect to the web interface from your computer at https://localhost:9443 and you can SSH to the container by connecting to your local computer on port 9022. (Tested and works.)

    2. Connect to the container by using SSH directly to connect first to the host, then to the container. (Not tested yet.)

Nextcloud (incomplete)

Open source replacement for Google Cloud etc. There is a docker image.

Incomplete because this needs MySQL etc. and I don't want to figure this out yet.

ssh swandocker

Let's Encrypt and Certbot

Let's Encrypt uses Certbot as a tool for enabling free site certification. This requires using a supported Ubuntu release.

sudo apt update
sudo apt install software-properties-common
sudo add-apt-repository universe
#sudo add-apt-repository ppa:certbot/certbot
sudo apt update
sudo apt install certbot python3-certbot-apache
sudo certbot --apache
#sudo apt-add-repository -r ppa:certbot/certbot

The commented out commands were needed earlier (pre Ubuntu 19.10) but should no longer be needed. Without removing this repo I ran into the following error) during apt update:

E: The repository 'http://ppa.launchpad.net/certbot/certbot/ubuntu focal Release' does not have a Release file.

With BitBucket "sunsetting" their support for Mercurial, we needed to find a new option. Here we explore options for self-hosting.

Version Control Hosting

Kallithea (incomplete)

Kallithea needs a database, so we make a directory for this in /data/kalithea and store it there.

sudo apt-get install npm
su admin
sudo apt-get install build-essential git python-pip python-virtualenv libffi-dev python-dev
hg clone https://kallithea-scm.org/repos/kallithea
cd kallithea
conda create -n kallithea python=2
conda activate kallithea
pip install --upgrade .
mkdir -p "${CONDA_PREFIX}/etc/conda/activate.d"
echo "export LC_ALL='C.UTF-8'" >> "${CONDA_PREFIX}/etc/conda/activate.d/env_vars.sh"

su admin
conda activate kallithea
mkdir /data/kallithea
cd /data/kallithea
kallithea-cli config-create my.ini
kallithea-cli db-create -c my.ini  #Created with root path `/data/kallithea`, user `mforbes` etc.
kallithea-cli front-end-build
gearbox serve -c my.ini

This runs Kallithea on port 5000 which could be accessed by users with ssh tunelling.

Another alternative is Heptapod. This is a mercurial interface to a friendly fork of GitLab CE intended to ultimately bring mercurial support to GitLab. This can be installed with Docker as discussed below.

As of 1 July 2020: This is my primary alternative to Bitbucket hosted as discussed below. We will probably ultimately host this on an AWS instance.

As of 14 March 2020:

Podman

An alternative to Docker is Podman. This behaves very similarly to Docker, but runs rootless natively.