We describe various strategies for working with CoCalc including version control, collaboration, and using Dropbox.

Note: In some places, such as my aliases, I still use the acronym SMC which refers to Sage Mathcloud – the previous name for CoCalc.

TL;DR

1. Setup your local computer (once) as discussed below.

2. Create a CoCalc project, add network access etc., and add your ssh key, and create an alias on your local computer for convenience.

3. SSH to your CoCalc project and then do something like this:

ssh smcbec  # Or whatever you called your alias

cd ~        # Do this in the top level of your cocalc project.

cat >> ~/.hgrc <<EOF
[ui]
username = \$LC_HG_USERNAME
merge = emacs
paginate = never

[extensions]
# Builtin extensions:
rebase=
graphlog=
color=
record=
histedit=
shelve=
strip=
#extdiff =
#mq =
#purge =
#transplant =
#evolve =
#amend =

[color]
custom.rev = red
custom.author = blue
custom.date = green
custom.branches = red

[merge-tools]
emacs.args = -q --eval "(ediff-merge-with-ancestor \""$local"\" \""$other"\" \""$base"\" nil \""$output"\")"
EOF

cat >> ~/.gitconfig <<EOF
[push]
    default = simple
[alias]
    lg1 = log --graph --abbrev-commit --decorate --date=relative --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all
    lg2 = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n''          %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all
    lg = !"git lg1"]
EOF

cat >> ~/.bash_aliases <<EOF
# Add some customizations for mercurial etc.
. mmf_setup

# Specified here since .gitconfig will not expand the variables
git config --global user.name "\${LC_GIT_USERNAME}"
git config --global user.email "\${LC_GIT_USEREMAIL}"
EOF

cat >> ~/.hgignore <<EOF
syntax: glob

*.sage-history
*.sage-chat
*.sage-jupyter
EOF

cat >> ~/.inputrc <<EOF
"\M-[A":       history-search-backward
"\e[A":        history-search-backward
"\M-[B":       history-search-forward
"\e[B":        history-search-forward
EOF

cat >> ~/.mrconfig <<EOF
# myrepos (mr) Config File; -*-Shell-script-*-
# dest = ~/.mrconfig     # Keep this as the 2nd line for mmf_init_setup

include = cat "${MMFHG}/mrconfig"

[DEFAULT]
hg_out = hg out
EOF

pip install --user mmf_setup
anaconda2019

# Should use conda or mamba, but this needs a new
# environment, so we just use pip for now.
pip install --user mmf_setup mmfutils

# Create a work environment and associate a kernel
mamba env create mforbes/work
mkdir -p ~/.local/share/jupyter/kernels/
cd ~/.local/share/jupyter/kernels/
cp -r /ext/anaconda-2019.03/share/jupyter/kernels/python3 work-py
cat > ~/.local/share/jupyter/kernels/work-py/kernel.json <<EOF
# kernel.json
{
 "argv": [
  "/home/user/.conda/envs/work/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "work",
 "language": "python"
}
EOF

exit-anaconda

mkdir -p ~/repositories
git clone git://myrepos.branchable.com/ ~/repositories/myrepos
ln -s ~/repositories/myrepos/mr ~/.local/bin/

Setup

Knowledge: To follow these instructions you will need to understand how to work with the linux shell. If you are unfamilliar with the shell, please review the shell novice course. For a discussion of environmental variables see how to read and set environmental and shell variables.

Prior to creating a new project I do the following on my local computer:

1. Set the following environment variable in one of my startup file. CoCalc automatically sources ~/.bash_aliases if it exists (this is specifeid in ~/.bashrc) so I use it.:

   Copy

   # ~/.bash_aliases
   ...
   # This hack allows one to forward the environmental variables using
   # ssh since LC_* variables are permitted by default.
   export LC_HG_USERNAME="Michael McNeil Forbes <[email protected]>"
   export LC_GIT_USEREMAIL="[email protected]"
   export LC_GIT_USERNAME="Michael McNeil Forbes"
   

   Then in my ~/.hgrc file I include the following:

   Copy

   # ~/.hgrc
   [ui]
   username = $LC_HG_USERNAME
   

   This way, you specify your mercurial username in only one spot - the LC_HG_USERNAME environmental variable. (This is the DRY principle.)

   A similar configuration should be used if you want to use git but variable expansion will not work with git. Instead, we need to set the user and email in the .bash_aliases file with something like:

   Copy

   # ~/.bash_aliases
   git config --global user.name = "$LC_GIT_USERNAME"
   

git config --global user.name = "$LC_GIT_USEREMAIL"

*The reason for using a variable name staring with `LC_*` is that these variables are generally allowed by the `sshd` server so that they can be send when one uses ssh to connect to a project (see below).*

2. Find the `host` and `username` for your CoCalc project (look under the project **Settings** page under the `>_ SSH into your project...` section) and add these to my local ``~/.ssh/config`` file.  For example: CoCalc might say to connect to `e397631635174e21abaa7c59fa227655@compute5-us.sagemath.com`.  This would mean I should add the following to my `~/.ssh/config` file:

```ini
# ~/.ssh/config
Host smc*
  ForwardAgent yes
  SendEnv LC_HG_USERNAME
  SendEnv LC_GIT_USERNAME
  SendEnv LC_GIT_USEREMAIL

Host smcbec
  HostName compute5-us.sagemath.com
  User e397631635174e21abaa7c59fa227655

The SendEnv instruction will then apply to all smc* hosts and sends the LC_HG_USERNAME environmental variable. This allows us to refer to this in the ~/.hgrc file so that version control commands will log based on who issues them, which is useful because CoCalc does not provide user-level authentication (only project level). Thus, if a user sends this, then mercurial can use the appropriate username. (A similar setup with git should be possible). See issue #370 for more information.

1. Once the project has been created, I add the contents of my ~/.ssh/id_rsa.pub to CoCalc using the web interface for SSH Keys. This allows me to login to my projects. (On other systems, this would be the equivalent of adding it to ~/.ssh/authorized_keys.)

2. Add any resources for the project. (For example, network access simplifies installing stuff below, and using a fixed host will prevent the compute node from changing so that the alias setup in the next step will keep working. However, you must pay for these upgrades.)

3. Create a ~/.hgrc file like the following:

   Copy

   [ui]
   username = $LC_HG_USERNAME
   [extensions]
   
   #####################
   # Builtin extensions:
   rebase=
   graphlog=
   color=
   record=
   histedit=
   shelve=
   strip=
   
   [color]
   custom.rev = red
   custom.author = blue
   custom.date = green
   custom.branches = red
   

   This one enables some extensions I find useful and specifies the username using the $LC_HG_USERNAME environmental variable sent by ssh in the previous step.

4. Create a ~/.gitconfig file like the following:

   Copy

   [user]
       name =
       email =
   [push]
       default = simple
   [alias]
       lg1 = log --graph --abbrev-commit --decorate --date=relative --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all
       lg2 = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n''          %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all
       lg = !"git lg1"
   

   This one provide a useful git lg command and specifies the username using the $LC_GIT_USERNAME etc. environmental variable sent by ssh in the previous step.

5. Install the mmf_setup package (I do this also in the anaconda3 environment):

   Copy

   pip install mmf_setup
   anaconda3
   pip3 install mmf_setup
   exit-anaconda
   

   Note: this requires you to have enabled network access in step 2.

6. (optional) Enable this by adding the following lines to your ~/.bash_aliases file on :

   Copy

   cat >> ~/.bash_aliases <<EOF
   # Add some customizations for mercurial etc.
   . mmf_setup
   git config --global user.name "\${LC_GIT_USERNAME}"
   git config --global user.email "\${LC_GIT_USEREMAIL}"
   EOF
   

   This will set your $HGRCPATH path so that you can use some of the tools I provide in my mmf_setup package, for example, the hg ccommit command which runs nbstripout to remove output from Jupyter notebooks before committing them.

7. (optional) I find the following settings very useful for tab completion etc., so I also add the following ~/.inputrc file on CoCalc: (The default configuration has LC_ALL=C so I do not need anything else, but see the comment below.)

   Copy

   #~/.inputrc
   
   # This file is used by bash to define the key behaviours.  The current
   # version allows the up and down arrows to search for history items
   # with a common prefix.
   #
   # Note: For these to be properly intepreted, you need to make sure your locale
   # is properly set in your environment with something like:
   # export LC_ALL=C
   
   #
   # Arrow keys in keypad mode
   #"\M-OD":        backward-char
   #"\M-OC":        forward-char
   #"\M-OA":        previous-history
   #"\M-OB":        next-history
   #
   # Arrow keys in ANSI mode
   #
   #"\M-[D":        backward-char
   #"\M-[C":        forward-char
   "\M-[A":        history-search-backward
   "\M-[B":        history-search-forward
   #
   # Arrow keys in 8 bit keypad mode
   #
   #"\M-\C-OD":       backward-char
   #"\M-\C-OC":       forward-char
   #"\M-\C-OA":       previous-history
   #"\M-\C-OB":       next-history
   #
   # Arrow keys in 8 bit ANSI mode
   #
   #"\M-\C-[D":       backward-char
   #"\M-\C-[C":       forward-char
   #"\M-\C-[A":       previous-history
   #"\M-\C-[B":       next-history
   

8. (optional) Update and configure pip to install packages as a user:

   Copy

   pip install --upgrade pip
   hash -r   # Needed to use new pip before logging in again
   
   mkdir -p ~/.config/pip/
   cat >> ~/.config/pip/pip.conf <<EOF
   [install]
   user = true
   find-links = https://bitbucket.org/mforbes/mypi/
   EOF
   

   The configuration uses my personal index which allows me to point to various revisions of my software.

Custom Environments

TL;DR:

Create an appropriate environment.yml file:

# environment.yml
name: _my_environment
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3
  - matplotlib
  - scipy
  - sympy

  - ipykernel
  - ipython
  #- notebook
  #- numexpr

  - pytest

  # Profiling
  - line_profiler
  #- psutil
  - memory_profiler

  - pip:
    - mmf_setup
    - hg+ssh://hg@bitbucket.org/mforbes/mmfutils-fork@0.4.10

We would like to move towards a workflow with custom conda environments. The idea is described here:

• https://github.com/sagemathinc/cocalc/wiki/Programming#anaconda-sage

ssh smctov
anaconda2019
conda create -n work3 --clone base   # Clone the base environment

Notes:

• Apparently this will download a whole bunch of stuff even though we are asking for a clone. This is a known issue: conda/constructor#145, conda#7398.

Once you have a custom environment, you can locate it and make a custom Jupyter kernel for it. First locate the environment:

$ conda env list
# conda environments:
#
base                  *  /ext/anaconda5-py3
xeus                     /ext/anaconda5-py3/envs/xeus
_gpe                     /home/user/.conda/envs/_gpe

Now copy another specification, then edit the kernel.json file. Here is what I ended up with:

mkdir -p ~/.local/share/jupyter/kernels/
cd ~/.local/share/jupyter/kernels/
cp -r /ext/anaconda2020.02/share/jupyter/kernels/python3 work-py
vi ~/.local/share/jupyter/kernels/work-py/kernel.json

The name of the directory here work-py matches the name of the kernel on my machine where I use the ipykernel package. This allows me to use the same notebooks going back and forth without changing the kernel.

# kernel.json
{
 "argv": [
  "/home/user/.conda/envs/work/bin/python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "work",
 "language": "python"
}

MayaVi

MayaVi is a nice rendering engine for analyzing 3D data-structures, but poses some problems for use on CoCalc. Here we describe these and how to get it working.

1. Create an appropriate conda environment and associated kernel as described above. For example:

   1. Create an environment.yml file:

      Copy

      # environment.mayavi3.yml
      name: mayavi3
      channels:
        - defaults
        - conda-forge
      dependencies:
        - python=3
        - ipykernel
        - mayavi
        - xvfbwrapper
      
        # jupyter is only needed in the first environment to run 
        #    jupyter nbextension install --py mayavi --user
        # Once this is run from an environment with *both* jupyter
        # and mayavi, it is not needed in future environments.
        - jupyter
      

      We need jupyter here so we can install the appropriate CSS etc. to allow or rendering.

   2. Activate anaconda and create the mayavi3 environment:

      Copy

      anaconda2019
      conda env create -f environment.mayavi3.yml
      

   3. Create the appropriate kernel:

      Find the location of current kernels:

      Copy

      # This path is a kludge.  Check it!  The awk command strips spaces
      # https://unix.stackexchange.com/a/205854
      base_kernel_dir=$(jupyter --paths | grep ext | grep share | awk '{$1=$1;print}')
      echo "'$base_kernel_dir'"
      

      At the time of running, this is: /ext/anaconda-2019.03/share/jupyter

      Copy

      mkdir -p ~/.local/share/jupyter/kernels/
      cp -r "$base_kernel_dir"/kernels/python3 \
            ~/.local/share/jupyter/kernels/conda-env-mayavi3-py
      vi ~/.local/share/jupyter/kernels/conda-env-mayavi3-py/kernel.json
      

2. Activate the environment and install the javascript required to render the output:

   Copy

   anaconda2019
   conda activate mayavi3
   jupyter nbextension install --py mayavi --user
   jupyter nbextension enable mayavi --user --py
   

   This places the javascript etc. in ./.local/share/jupyter/nbextensions/mayavi and adds an entry in ~/.jupyter/nbconfig/notebook.json:

   Copy

   {
     "load_extensions": {
       "mayavi/x3d/x3dom": true
     }
   }
   

3. Start a new notebook with your kernel, run it in the classic notebook server ("switch to classic notebook..." under "File").

4. Start a virtual frame-buffer and then use MayaVi with something like the following in your notebook:

   Copy

    from xvfbwrapper import Xvfb
    with Xvfb() as xvfb:
        from mayavi import mlab
        mlab.init_notebook()
        s = mlab.test_plot3d()
        display(s)
   

   Alternatively, you can skip the context and do something like:

   Copy

   from xvfbwrapper import Xvfb
   vdisplay = Xvfb()
   vdisplay.start()
   from mayavi import mlab
   mlab.init_notebook()
   s = mlab.test_plot3d()
   display(s)
   vdisplay.stop()   # Calling this becomes a bit more onerous, but might not be critical
   

   See xvfbwrapper for more details.

System Software

One can definitely build system software from source, linking it into ~/.local/bin/ etc. I am not sure if there is a way of using apt-get or other package managers yet.

Synchronization

The automatic synchronization mechansim of CoCalc has some issues. The issue (#96) is that when you VCS updates the files, it can change the modification date in a way tha triggers the autosave system to revert to a previous version. The symptom is that you initially load the notebook that you want, but within seconds it reverts to an old (or even blank version).

Thus, it is somewhat dangerous to perform a VCS update on CoCalc: you risk losing local work. Note: none of the work is lost - you can navigate to the project page and look for the "Backups" button on the file browser. This will take you to read-only copies of your work which you can use to restore anything lost this way.

Presently, the only safe solution to update files from outside of the UI is to update them in a new directory.

MathJaX

MathJaX is rather slow, so in 2018 CoCalc has enabled KaTeX. Unfortunately, KaTeX is not as feature rich as MathJaX. If you need full MathJaX functionality, then you can revert to MathJaX in your account settings.

RClone for Google Drive etc.

The program rclone provides a command-line interface for several applications like Dropbox and Google Drive. Here are some notes about using it:

1. Make sure internet access is enabled for you project.

2. Install it by downloading the binary and installing it in ~/.local/bin/rclone. The rclone software is already installed on CoCalc.

3. Add a remote by running rclone config. Some notes:

   • For the "Scope" I chose "Full access all files". More secure would be "Access to files created by rclone only" but this will not work if you add files from another device.

   • If you want to link a specific folder, you can copy the "ID of the root folder" from the last part of the URL when you open the folder in Google Drive. It looks something like "1RAtwfvaJUk4vULw1z1t1qKWJCsJIRqCZ".

   • When asked for "Auto config" choose no - this is a "headless" configuration.

   I choose the name gd for the Google Drive remote. Following the provided link and link your account.

4. You can now explore with:

   Copy

   rclone lsd gd:   # Show remote directories
   rclone copy gd:PaperQuantumTurbulence .   # Copy a folder
   rclone check gd:PaperQuantumTurbulence PaperQuantumTurbulence 
   rclone sync gd:PaperQuantumTurbulence PaperQuantumTurbulence  # Like rsync -d
   

Notebook Extensions

One can use Jupyter notebook extensions (nbextensions) with only the Classic Notebook interface. As per issue 985, extensions will not be supported with the Modern Notebook interface, but their goal is to independently implement useful extensions, so file an issue if there is something you want. In this section, we discuss how to enable extensions with the Classic Interface.

1. Open your notebook.

2. At the bottom of the File menu choose File/Switch to classical notebook.... (As per issue 1537, this will not work with Firefox.)

3. At the bottom of the Edit menu choose Edit/nbextensions config. This will allow you to enable various extensions.

(Old Procedure: Not Needed Anymore)

1. Enable internet access.

2. Install the code:

   Copy

   pip install --user https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tarball/master  
   

   Note: this will also install a copy of jupyter which is not ideal and is not the one that is run by default, but it will allow you to configure things.

3. Symbolically link this to your user directory:

   Copy

   ln -s ~/.local/lib/python2.7/site-packages/jupyter_contrib_nbextensions/nbextensions .jupyter/
   

4. Install the extensions:

   Copy

   jupyter contrib nbextension install --user
   

   This step adds the configuration files.

5. Restart the server.

6. Reload your notebooks.

You should now see a new menu item: Edit/nbextensions config. From this you can enable various extensions. You will need to refresh/reload each notebook when you make changes.

Note: This is not a proper installation, so some features may be broken. The Table of Contents (2) feature works, however, which is one of my main uses.)

Raw File Access

If you want to directly access files such as HTML files without the CoCalc interface, you can. This is described here.

Configuration Files

Here are the ultimate contents of the configuration files I have on my computer and on CoCalc. These may get out of date. For up-to-date versions, please see the configurations/complete_systems/cocalc folder on my confgurations project.

Your Computer

~/.bashrc

~/.ssh/config

CoCalc Project

~/.bash_alias

~/.inputrc

~/.hgrc

~/.hgignore

~/.gitconfig

~/.mrconfig

Note: Until symlinks work again, I can't really used myrepos.

References

• CoCalc User Manual

• http://mostlyunixish.franzoni.eu/blog/2012/06/21/dropbox-and-mercurial-and-lived-happily-ever-after/

Old Information (Out of Data)

The following information is recorded for historical purposes. It no longer applies to CoCalc.

Dropbox no longer works!

Dropbox dropped linux support for all file systems except ext4 which is not an option for CoCalc.

You can use Dropbox on CoCalc:

dropbox start -i

The first time you do this, it will download some files and you will need to provide a username etc. Note, as pointed out in the link, by default, dropbox wants to share everything. I am not sure of the best strategy for this, but chose to create a separate Dropbox account for the particular project I am using, then just adding the appropriate files to that account.

Once nice think about Dropbox is that it works well with symbolic links, so you can just symlink any files you want into the ~/Dropbox folder and everything should work fine, but hold off for a moment - first exclude the appropriate folders, then make the symlinks.

If you want to run Dropbox everytime you login, then add the following to your ~/.bash_aliases on CoCalc:

cat >> ~/.bash_aliases <<EOF

# Start Dropbox
dropbox start
EOF

This did not link with my dropbox account. I had to manually kill the dropbox process, then run the following:

.dropbox-dist/dropboxd

Dropbox and Version Control

Although tempting, one should not use Dropbox to share VCS files since corruption might occur. Instead, you can use Dropbox to sync the working directory (for collaborators) but use selective sync to ignore the sensitive VCS files. Here is one way to do this:

cd ~/Dropbox
find -L . -name ".hg" -exec dropbox exclude add {} \;
find -L . -name ".git" -exec dropbox exclude add {} \;

Note: This will remove the .hg directories! (The behaviour of excluding a file with selective sync is to remove it from the local Dropbox.) Thus, my recommendation is the following:

1. First copy the files to the Dropbox folder using cp -r.

2. Then run the previous find commands to exclude the .hg directories.

3. Check this with dropbox exclude list.

4. Turn off dropbox dropbox stop

5. Now replace the folders with appropriate symlinks.

6. Finally, start dropbox again dropbox start.

Here is a summary for example. I am assuming that you have a version controlled project called ~/paper that you want mirrored in your Dropbox folder:

dropbox start
cp -r ~/paper ~/Dropbox/
pushd Dropbox; find -L. -name ".hg" -exec dropbox exclude add {} \; ; popd
dropbox exclude list
dropbox stop
rm -r ~/Dropbox/paper
ln -s ~/paper ~/Dropbox/paper
dropbox start

Remember to do the same on your local computer!

Dropbox Issues

Occasionally there will be issues with dropbox. One of the issues may be when the host for the project changes (this happens for example when you add or remove member hosting for a project). To deal with this you might have to unlike or relink the computer to Dropbox:

• Make sure your project has "Internet access" in Settings.

• dropbox stop on the CoCalc server.

• Sign into Dropbox with the account you have linked to the project. (Hint: if you can't remember the name, look in your main dropbox project where you should have shared this with the CoCalc project. It can be useful to use Incognito mode or a different browser to sign into an auxilliary account.)

• Go to Settings (top right), then the Security tab.

• In the Devices section, unlink the device.

• Restart dropbox with ~/.dropbox-dist/dropboxd. This should give you a link to paste into the browser you have signed into and it will relink.