{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "We describe various strategies for working with [CoCalc](https://cocalc.com) including version control, collaboration, and using Dropbox.\n", "\n", "\n", "*Note: In some places, such as my aliases, I still use the acronym SMC which refers to Sage Mathcloud – the previous name for CoCalc.*" ] }, { "cell_type": "markdown", "metadata": { "toc": true }, "source": [ "

Table of Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# TL;DR" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. Setup your local computer (once) as discussed below.\n", "2. Create a CoCalc project, add network access etc., and add your ssh key, and create an alias on your local computer for convenience.\n", "3. SSH to your CoCalc project and then do something like this:\n", "\n", "```bash\n", "ssh smcbec # Or whatever you called your alias\n", "\n", "cd ~ # Do this in the top level of your cocalc project.\n", "\n", "cat >> ~/.hgrc <> ~/.gitconfig <> ~/.bash_aliases <> ~/.hgignore <> ~/.inputrc <> ~/.mrconfig < ~/.local/share/jupyter/kernels/work-py/kernel.json <\"\n", " export LC_GIT_USEREMAIL=\"michael.forbes+github@gmail.com\"\n", " export LC_GIT_USERNAME=\"Michael McNeil Forbes\"\n", " ```\n", " \n", " Then in my `~/.hgrc` file I include the following:\n", " \n", " ```ini\n", " # ~/.hgrc\n", " [ui]\n", " username = $LC_HG_USERNAME\n", " ```\n", " \n", " This way, you specify your mercurial username in only one spot - the `LC_HG_USERNAME` environmental variable. (This is the [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) principle.)\n", " \n", " A similar configuration should be used if you want to use `git` but variable expansion will [not work with git](https://stackoverflow.com/a/11262153). Instead, we need to set the user and email in the `.bash_aliases` file with something like:\n", " \n", " ```bash\n", " # ~/.bash_aliases\n", " git config --global user.name = \"$LC_GIT_USERNAME\"\n", "git config --global user.name = \"$LC_GIT_USEREMAIL\" \n", " ```\n", " \n", " *The reason for using a variable name staring with `LC_*` is that these variables are generally allowed by the `sshd` server so that they can be send when one uses ssh to connect to a project (see below).*\n", " \n", "2. Find the `host` and `username` for your CoCalc project (look under the project **Settings** page under the `>_ SSH into your project...` section) and add these to my local ``~/.ssh/config`` file. For example: CoCalc might say to connect to `e397631635174e21abaa7c59fa227655@compute5-us.sagemath.com`. This would mean I should add the following to my `~/.ssh/config` file:\n", "\n", " ```ini\n", " # ~/.ssh/config\n", " Host smc*\n", " ForwardAgent yes\n", " SendEnv LC_HG_USERNAME\n", " SendEnv LC_GIT_USERNAME\n", " SendEnv LC_GIT_USEREMAIL\n", "\n", " Host smcbec\n", " HostName compute5-us.sagemath.com\n", " User e397631635174e21abaa7c59fa227655\n", " ```\n", " \n", " The `SendEnv` instruction will then apply to all `smc*` hosts and sends the `LC_HG_USERNAME` environmental variable. This allows us to refer to this in the `~/.hgrc` file so that version control commands will log based on who issues them, which is useful because CoCalc does not provide user-level authentication (only project level). Thus, if a user sends this, then mercurial can use the appropriate username. (A similar setup with git should be possible). See [issue #370](https://github.com/sagemathinc/smc/issues/370) for more information. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. Once the project has been created, I add the contents of my `~/.ssh/id_rsa.pub` to CoCalc using the web interface for SSH Keys. This allows me to login to my projects. *(On other systems, this would be the equivalent of adding it to `~/.ssh/authorized_keys`.)*\n", "2. Add any resources for the project. (For example, network access simplifies installing stuff below, and using a fixed host will prevent the compute node from changing so that the alias setup in the next step will keep working. However, you must pay for these upgrades.)\n", "3. Create a `~/.hgrc` file like the following:\n", " \n", " ```ini\n", " [ui]\n", " username = $LC_HG_USERNAME\n", " [extensions]\n", "\n", " #####################\n", " # Builtin extensions:\n", " rebase=\n", " graphlog=\n", " color=\n", " record=\n", " histedit=\n", " shelve=\n", " strip=\n", "\n", " [color]\n", " custom.rev = red\n", " custom.author = blue\n", " custom.date = green\n", " custom.branches = red\n", " ```\n", " \n", " This one enables some extensions I find useful and specifies the username using the `$LC_HG_USERNAME` environmental variable sent by ssh in the previous step.\n", "\n", "3. Create a `~/.gitconfig` file like the following:\n", " \n", " ```ini\n", " [user]\n", " name =\n", " email =\n", " [push]\n", " default = simple\n", " [alias]\n", " lg1 = log --graph --abbrev-commit --decorate --date=relative --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all\n", " lg2 = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n'' %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all\n", " lg = !\"git lg1\"\n", " ```\n", " \n", " This one provide a useful `git lg` command and specifies the username using the `$LC_GIT_USERNAME` etc. environmental variable sent by ssh in the previous step.\n", "\n", "4. Install the [`mmf_setup`] package (I do this also in the `anaconda3` environment):\n", "\n", " ```bash\n", " pip install mmf_setup\n", " anaconda3\n", " pip3 install mmf_setup\n", " exit-anaconda\n", " ```\n", "\n", " Note: this requires you to have enabled network access in step 2.\n", "\n", "5. (optional) Enable this by adding the following lines to your `~/.bash_aliases file` on :\n", " \n", " ```bash\n", " cat >> ~/.bash_aliases <> ~/.config/pip/pip.conf <Install it by downloading the binary and installing it in `~/.local/bin/rclone`. The `rclone` software is already installed on CoCalc.\n", "3. Add a remote by running `rclone config`. Some notes:\n", " * For the \"Scope\" I chose \"Full access all files\". More secure would be \"Access to files created by rclone only\" but this will not work if you add files from another device.\n", " * If you want to link a specific folder, you can copy the \"ID of the root folder\" from the last part of the URL when you open the folder in Google Drive. It looks something like \"1RAtwfvaJUk4vULw1z1t1qKWJCsJIRqCZ\".\n", " * When asked for \"Auto config\" choose no - this is a \"headless\" configuration.\n", " \n", " I choose the name `gd` for the Google Drive remote. Following the provided link and link your account.\n", "4. You can now explore with:\n", "\n", " ```\n", " rclone lsd gd: # Show remote directories\n", " rclone copy gd:PaperQuantumTurbulence . # Copy a folder\n", " rclone check gd:PaperQuantumTurbulence PaperQuantumTurbulence \n", " rclone sync gd:PaperQuantumTurbulence PaperQuantumTurbulence # Like rsync -d\n", " ```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Notebook Extensions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One can use Jupyter notebook extensions ([nbextensions](https://github.com/ipython-contrib/jupyter_contrib_nbextensions)) with only the Classic Notebook interface. As per [issue 985](https://github.com/sagemathinc/smc/issues/985), extensions will not be supported with the Modern Notebook interface, but their goal is to independently implement useful extensions, so file an issue if there is something you want. In this section, we discuss how to enable extensions with the Classic Interface.\n", "\n", "1. Open your notebook.\n", "2. At the bottom of the `File` menu choose `File/Switch to classical notebook...`. (As per [issue 1537](https://github.com/sagemathinc/cocalc/issues/1537), this will not work with Firefox.)\n", "3. At the bottom of the `Edit` menu choose `Edit/nbextensions config`. This will allow you to enable various extensions.\n", "\n", "\n", "## (Old Procedure: Not Needed Anymore)\n", "\n", "1. Enable internet access.\n", "2. Install the code:\n", "\n", " ```bash\n", " pip install --user https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tarball/master \n", " ```\n", " \n", " Note: this will also install a copy of `jupyter` which is not ideal and is not the one that is run by default, but it will allow you to configure things.\n", "3. Symbolically link this to your user directory:\n", "\n", " ```bash\n", " ln -s ~/.local/lib/python2.7/site-packages/jupyter_contrib_nbextensions/nbextensions .jupyter/\n", " ```\n", "4. Install the extensions:\n", "\n", " ```bash\n", " jupyter contrib nbextension install --user\n", " ```\n", "\n", " This step adds the configuration files.\n", "5. Restart the server.\n", "6. Reload your notebooks.\n", "\n", "You should now see a new menu item: `Edit/nbextensions config`. From this you can enable various extensions. You will need to refresh/reload each notebook when you make changes.\n", "\n", "*Note: This is not a proper installation, so some features may be broken. The Table of Contents (2) feature works, however, which is one of my main uses.)*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Raw File Access" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to directly access files such as HTML files without the CoCalc interface, you can. This is [described here](https://github.com/sagemathinc/cocalc/wiki/share#file-sharing-tips)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Configuration Files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here are the ultimate contents of the configuration files I have on my computer and on CoCalc. These may get out of date. For up-to-date versions, please see the [configurations/complete_systems/cocalc](https://bitbucket.org/mforbes/configurations/src/trunk/complete_systems/cocalc/) folder on my confgurations project." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Your Computer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.bashrc`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# ~/.bashrc\n", "\n", "export HG_USERNAME=\"Michael McNeil Forbes \"\n", "export GIT_USEREMAIL=\"michael.forbes+github@gmail.com\"\n", "export GIT_USERNAME=\"Michael McNeil Forbes\"\n", "export LC_HG_USERNAME=\"${HG_USERNAME}\"\n", "export LC_GIT_USERNAME=\"${GIT_USERNAME}\"\n", "export LC_GIT_USEREMAIL=\"${GIT_USEREMAIL}\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.ssh/config`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# ~/.ssh/config:\n", "Host smc*\n", " HostName ssh.cocalc.com\n", " ForwardAgent yes\n", " SendEnv LC_HG_USERNAME\n", " SendEnv LC_GIT_USERNAME\n", " SendEnv LC_GIT_USEREMAIL\n", "\n", "Host smcbec... # Some convenient name\n", " User 01a3... # Use the code listed on CoCalc\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CoCalc Project" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.bash_alias`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# Bash Alias File; -*-Shell-script-*-\n", "# dest = ~/.bash_alias # Keep this as the 2nd line for mmf_init_setup\n", "#\n", "# On CoCalc, this file is automatically sourced, so it is where you\n", "# should keep your customizations.\n", "#\n", "# LC_* variables:\n", "#\n", "# Since each user logs in with the same user-id (specific to the\n", "# project), I use the following mechanism to keep track of who is\n", "# logged in for the purposes of using version control like hg and git.\n", "#\n", "# You should define the following variables on your home machine and then tell\n", "# ssh to forward these:\n", "#\n", "# ~/.bashrc:\n", "#\n", "# export HG_USERNAME=\"Michael McNeil Forbes \"\n", "# export GIT_USEREMAIL=\"michael.forbes+github@gmail.com\"\n", "# export GIT_USERNAME=\"Michael McNeil Forbes\"\n", "# export LC_HG_USERNAME=\"${HG_USERNAME}\"\n", "# export LC_GIT_USERNAME=\"${GIT_USERNAME}\"\n", "# export LC_GIT_USEREMAIL=\"${GIT_USEREMAIL}\"\n", "#\n", "# ~/.ssh/config:\n", "#\n", "# Host smc*\n", "# HostName ssh.cocalc.com\n", "# ForwardAgent yes\n", "# SendEnv LC_HG_USERNAME\n", "# SendEnv LC_GIT_USERNAME\n", "# SendEnv LC_GIT_USEREMAIL\n", "# Host smc... # Some convenient name\n", "# User 01a3... # Use the code listed on CoCalc\n", "#\n", "# Then you can `ssh smc...` and your username will be forwarded.\n", "\n", "# This content inserted by mmf_setup\n", "# Add my mercurial commands like hg ccom for removing output from .ipynb files\n", ". mmf_setup\n", "\n", "\n", "# Specified here since .gitconfig will not expand the variables\n", "git config --global user.name \"\\${LC_GIT_USERNAME}\"\n", "git config --global user.email \"\\${LC_GIT_USEREMAIL}\"\n", "\n", "export INPUTRC=~/.inputrc\n", "export SCREENDIR=~/.screen\n", "\n", "# I structure my projects with a top level repositories directory\n", "# where I include custom repos. The following is installed by:\n", "#\n", "# mkdir ~/repositories\n", "# hg clone ssh://hg@bitbucket.org/mforbes/mmfhg ~/repositories/mmfhg\n", "export MMFHG=~/repositories/mmfhg\n", "export HGRCPATH=\"${HGRCPATH}\":\"${MMFHG}\"/hgrc\n", "\n", "export EDITOR=vi\n", "\n", "# Finding stuff\n", "function finda {\n", " find . \\( \\\n", "\t -name \".hg\" -o -name \".ipynb_checkpoints\" -o -name \"*.sage-*\" \\) -prune \\\n", "\t -o -type f -print0 | xargs -0 grep -H \"${*:1}\"\n", "}\n", "\n", "function findf {\n", " find . \\( \\\n", "\t -name \".hg\" -o -name \".ipynb_checkpoints\" -o -name \"*.sage-*\" \\) -prune \\\n", "\t -o -type f -name \"*.$1\" -print0 | xargs -0 grep -H \"${*:2}\"\n", "}\n", "\n", "# I used to use aliases, but they cannot easily be overrriden by\n", "# personalzed customizations.\n", "function findpy { findf py \"${*}\"; }\n", "function findipy { findf ipynb \"${*}\"; }\n", "function findjs { findf js \"${*}\"; }\n", "function findcss { findf css \"${*}\"; }\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.inputrc`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# Bash Input Init File; -*-Shell-script-*-\n", "# dest = ~/.inputrc # Keep this as the 2nd line for mmf_init_setup\n", "\n", "# This file is used by bash to define the key behaviours. The current\n", "# version allows the up and down arrows to search for history items\n", "# with a common prefix.\n", "#\n", "# Note: For these to be properly intepreted, you need to make sure your locale\n", "# is properly set in your environment with something like:\n", "# export LC_ALL=C\n", "\n", "\"\\M-[A\": history-search-backward\n", "\"\\M-[B\": history-search-forward\n", "\"\\e[A\": history-search-backward\n", "\"\\e[B\": history-search-forward\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.hgrc`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# Mercurial (hg) Init File; -*-Shell-script-*-\n", "# dest = ~/.hgrc # Keep this as the 2nd line for mmf_init_setup\n", "\n", "[ui]\n", "username = $LC_HG_USERNAME\n", "merge = emacs\n", "paginate = never\n", "\n", "[extensions]\n", "# Builtin extensions:\n", "rebase=\n", "graphlog=\n", "color=\n", "record=\n", "histedit=\n", "strip=\n", "#extdiff =\n", "#mq =\n", "#purge =\n", "#transplant =\n", "#evolve =\n", "#amend =\n", "\n", "[color]\n", "custom.rev = red\n", "custom.author = blue\n", "custom.date = green\n", "custom.branches = red\n", "\n", "[merge-tools]\n", "emacs.args = -q --eval \"(ediff-merge-with-ancestor \\\"\"$local\"\\\" \\\"\"$other\"\\\" \\\"\"$base\"\\\" nil \\\"\"$output\"\\\")\"\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.hgignore`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# Mercurial (hg) Init File; -*-Shell-script-*-\n", "# dest = ~/.hgignore # Keep this as the 2nd line for mmf_init_setup\n", "\n", "syntax: glob\n", "\n", "\\.ipynb_checkpoints\n", "*\\.sage-jupyter2\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.gitconfig`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# Git Config File; -*-Shell-script-*-\n", "# dest = ~/.gitconfig # Keep this as the 2nd line for mmf_init_setup\n", "\n", "[push]\n", " default = simple\n", "[alias]\n", " lg1 = log --graph --abbrev-commit --decorate --date=relative --format=format:'%C(bold blue)%h%C(reset) - %C(bold green)(%ar)%C(reset) %C(white)%s%C(reset) %C(dim white)- %an%C(reset)%C(bold yellow)%d%C(reset)' --all\n", " lg2 = log --graph --abbrev-commit --decorate --format=format:'%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n'' %C(white)%s%C(reset) %C(dim white)- %an%C(reset)' --all\n", " lg = !\"git lg1\"]\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### `~/.mrconfig`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note: Until [symlinks work again](https://myrepos.branchable.com/todo/Allow_symlinked_directories), I can't really used myrepos." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "# myrepos (mr) Config File; -*-Shell-script-*-\n", "# dest = ~/.mrconfig # Keep this as the 2nd line for mmf_init_setup\n", "#\n", "# Requires the myrepos perl script to be installed which you can do with the\n", "# following commands:\n", "#\n", "# mkdir -P ~/repositories\n", "# git clone git://myrepos.branchable.com/ ~/repositories/myrepos\n", "# pushd ~/repositories/myrepos; git checkout 52e2de0bdeb8b892c8b83fcad54543f874d4e5b8; popd\n", "# ln -s ~/repositories/myrepos/mr ~/.local/bin/\n", "#\n", "# Also requires the mmfhg package which can be enabled by installing\n", "# mmf_setup and running . mmf_setup from your .bash_aliases file.\n", "\n", "include = cat \"${MMFHG}/mrconfig\"\n", "\n", "[DEFAULT]\n", "hg_out = hg out\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# References\n", "\n", "* [CoCalc User Manual](https://doc.cocalc.com)\n", "* http://mostlyunixish.franzoni.eu/blog/2012/06/21/dropbox-and-mercurial-and-lived-happily-ever-after/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Old Information (Out of Data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following information is recorded for historical purposes. It no longer applies to CoCalc." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Dropbox [no longer works!](https://github.com/sagemathinc/cocalc/issues/11)\n", "\n", "Dropbox dropped linux support for all file systems except ext4 which is not an option for CoCalc.\n", "\n", "You can [use Dropbox on CoCalc](http://ask.sagemath.org/question/10713/is-there-a-way-to-store-and-sync-the-files-from-sage-cloud-on-local-hard-drive/):\n", "\n", "```bash\n", "dropbox start -i\n", "```\n", "\n", "The first time you do this, it will download some files and you will need to provide a username etc. Note, as pointed out in the link, by default, dropbox wants to share *everything*. I am not sure of the best strategy for this, but chose to create a separate Dropbox account for the particular project I am using, then just adding the appropriate files to that account.\n", "\n", "Once nice think about Dropbox is that it works well with symbolic links, so you can just symlink any files you want into the `~/Dropbox` folder and everything should work fine, but hold off for a moment - first exclude the appropriate folders, then make the symlinks.\n", "\n", "If you want to run Dropbox everytime you login, then add the following to your `~/.bash_aliases` on CoCalc:\n", "\n", "```bash\n", "cat >> ~/.bash_aliases <