{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", " \"git-annex\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# [git-annex](https://git-annex.branchable.com)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[git-annex] is a tool for managing large data files with [git](http://git-scm.com). The idea is to store the information *about* the file in a git repository that can be synchronized, but to store the actual data separately. The annex keeps track of where the file actually resides (which may be in a different repository, or on another compute) and allows you to control the file (renaming, moving, etc.) without having to have the actual file present.\n", "\n", "Here we explore [git-annex] as a mechanism for replacing *and interacting* with Dropbox, Google Drive, One Drive etc. with the following goals:\n", "\n", "1. Multiple users can share data.\n", "2. Data shared across many platforms: HPC clusters, laptops, desktops, Mac, Windows, Linux, [CoCalc], etc.\n", "3. Allow only a subset of data to be stored on any particular device (esp. laptops) if memory on that device is limited.\n", "4. Utilize cloud storage options including Google Cloud, Dropbox, Microsoft One Drive both as redundant backups, but also as a mechanism for sharing data with others who need to be able to use only once of these services.\n", "5. Automatic and manual sync options.\n", "\n", "[git-annex]: https://git-annex.branchable.com\n", "[CoCalc]: https://cocalc.com\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "My use-case is that I run a research group at WSU with ~10 collaborators. We need to share source code, experimental data, papers, plots, and simulation data on a regular basis. Some of the collaborators are used to using Dropbox or Google Drive, which work for syncing, but have issues (listed below). WSU provides 1TB of storage through Microsoft One Drive for all students and faculty, so this would be a natural storage too, but few use it yet. We run simulations on our local machines, office desktops, a local HPC cluster [Kamiak] and online using [CoCalc].\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "toc": true }, "source": [ "

Table of Contents

\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Overview of git-annex" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is my current understanding of [git-annex] and the [git-annex assistant].\n", "\n", "* Each \"annex\" is a git repository where the annex information is stored under the `.git` directory. Note: the annex information is separate from regular git information. Thus, if you do not initialize the git repository, `git status` will return `fatal: This operation must be run in a work tree`. You can use the annex as both a git-annex and a regular git repo if you choose. All of the annex information (log files, repository information, etc.) is stored in the `.git` directory.\n", "* For a GUI you can run `git-annex-webapp`. This seems to open the annex in the current folder or the last used annex. You can create other repositories from the GUI, but they will not appear in the Dashboard - only the repositories managed by the current \"annex\" (i.e. stored in the `.git` folder for that annex) appear in the Dashboard.\n", "* Configuration information is stored in `~/.config/git-annex` and `.git/config`. The former includes autostart information about which repositories should be synced.\n", "* Look at the various [workflows](https://git-annex.branchable.com/workflow/). This explains the difference between `sync` (which syncs meta-data but not the file contents) and `sync --content`.\n", "\n", "[git-annex]: https://git-annex.branchable.com\n", "[git-annex assistant]: https://git-annex.branchable.com/assistant/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Installation on major platforms is quite straightforward using the prebuilt binaries. Read the instructions carefully to see what you need to add to your path so you can access the ``git annex`` commands. (On my Mac OS X laptop, I added ``/Applications/git-annex.app/Contents/MacOS`` to my path using the [GNU Environment Modules](http://modules.sourceforge.net) mechanism, which I have installed for keeping track of various paths.)\n", "\n", "To build git-annex from source might be a challenge since it is written in Haskell. Fortunately, one can use remote computers without installing git-annex on them. In many cases, it can be installed using package managers." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [Conda](http://git-annex.branchable.com/install/conda/)\n", "\n", "This is now my preferred method on Linux servers:\n", "\n", "```bash\n", "conda install -c conda-forge git-annex\n", "conda install -c conda-forge tenacity pydrive\n", "pip install annexremote git-annex-remote-googledrive\n", "```\n", "\n", "*Note: this requires python 3 but also includes the client require for using Google Drive (see below).*\n", "\n", "Instead I install this in a conda environment using the following yml file:\n", "\n", "```bash\n", "# environment.app_gitannex.yml\n", "name: app_gitannex\n", "channels:\n", " - defaults\n", " - conda-forge\n", "dependencies:\n", " - tenacity\n", " - pydrive\n", " - git-annex\n", "\n", " - pip:\n", " - annexremote\n", " - git-annex-remote-googledrive\n", "```\n", "\n", "by running \n", "\n", "```bash\n", "conda env install --file environment.app_gitannex.yml\n", "```\n", "\n", "then linking this to a `bin/` directory on path:\n", "\n", "```bash\n", "ln -s \"$(conda env list | grep app_gitannex | awk '{print $2}')/bin/git-annex-remote-googledrive\" /usr/local/bin/\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Mac OS X\n", "\n", "Unfortunately, a conda git-annex binary is not maintained for OS X so I use instead the [pre-built binary for Os X](https://git-annex.branchable.com/install/OSX/).\n", "\n", "* [`git-annex-turtle`](https://github.com/andrewringler/git-annex-turtle) might be useful for integration with FileManager (I have not tried yet)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Ubuntu\n", "\n", "The standard Ubuntu install is old. I tried adding [Justin Geibel's repo](https://launchpad.net/~jtgeibel/+archive/ubuntu/ppa?field.series_filter=trusty) as discussed at the bottom of the [Ubuntu install page](https://git-annex.branchable.com/install/Ubuntu/)\n", "\n", "```bash\n", "sudo add-apt-repository ppa:jtgeibel/ppa\n", "sudo apt-get update\n", "sudo apt-get install git-annex\n", "```\n", "\n", "but this version does not have the webapp, assistant, etc. I ended up resorting to [using Conda](http://git-annex.branchable.com/install/conda/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Walk-through (git-annex only)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Following the walkthrough (slightly), I created a repository and initialized the annex:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initialized empty Git repository in /private/tmp/annex_eg/.git/\n", "init my laptop:annex_eg ok\n", "(recording state in git...)\n" ] } ], "source": [ "%%bash\n", "mkdir -p /tmp/annex_eg\n", "cd /tmp/annex_eg\n", "git init\n", "git annex init \"my laptop:annex_eg\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now add some \"big\" files:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Writing /tmp/annex_eg/data.dat\n" ] } ], "source": [ "%%file /tmp/annex_eg/data.dat\n", "Tonnes of data!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And now add them to the annex." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "add data.dat ok\n", "(recording state in git...)\n", "[master (root-commit) c815f59] Added data.dat\n", " 1 file changed, 1 insertion(+)\n", " create mode 120000 data.dat\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg\n", "git annex add data.dat\n", "git commit -a -m \"Added data.dat\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is simply a symbolic link to the actuall data file which presently lives here in this repository, so we can view it." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "lrwxr-xr-x 1 mforbes wheel 188 Aug 18 23:22 \u001b[35m/tmp/annex_eg/data.dat\u001b[m\u001b[m -> .git/annex/objects/WM/JK/SHA256E-s15--fa0ec06b25ff9438f9d594f07685954a57e05a90f5081ad51d85bcf551ab594b.dat/SHA256E-s15--fa0ec06b25ff9438f9d594f07685954a57e05a90f5081ad51d85bcf551ab594b.dat\n", "Tonnes of data!" ] } ], "source": [ "!ls -la /tmp/annex_eg/data.dat\n", "!cat /tmp/annex_eg/data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### [Modifying Files](https://git-annex.branchable.com/tips/unlocked_files/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We cannot modify it though: git-annex is very cautious about potential data loss!" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "bash: line 1: /tmp/annex_eg/data.dat: Permission denied\n" ] } ], "source": [ "%%bash\n", "echo \"More!\" >> /tmp/annex_eg/data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to modify it, you need to first unlock it. This replaces the symlink with the actual file and allows you to edit it." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "unlock data.dat (copying...) ok\n", "-rw-r--r-- 1 mforbes wheel 15B May 19 23:36 data.dat\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg/\n", "git annex unlock data.dat\n", "ls -lah data.dat" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [], "source": [ "%%bash\n", "echo \"Revised!\" >> /tmp/annex_eg/data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to discard the data, you could just lock the file again. Of course, safety checks are in place to make sure you do not loose data:" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "lock data.dat " ] }, { "name": "stderr", "output_type": "stream", "text": [ "git-annex: Locking this file would discard any changes you have made to it. Use 'git annex add' to stage your changes. (Or, use --force to override)\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg/\n", "git annex lock data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can force this:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "lock data.dat ok\n", "(recording state in git...)\n" ] } ], "source": [ "%%bash \n", "cd /tmp/annex_eg/\n", "git annex lock --force data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to save the changed data, you need to add and commit that data:" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "unlock data.dat (copying...) ok\n", "add data.dat ok\n", "(recording state in git...)\n", "[master 7bf9548] Modified data.dat\n", " 1 file changed, 1 insertion(+), 1 deletion(-)\n", "lrwxr-xr-x 1 mforbes wheel 188 May 19 23:36 data.dat -> .git/annex/objects/m9/8k/SHA256E-s24--4b6a60412e8ef0f79851a1c922880e1311f9ded6e3acdd8ca9b9241cb8ef1f82.dat/SHA256E-s24--4b6a60412e8ef0f79851a1c922880e1311f9ded6e3acdd8ca9b9241cb8ef1f82.dat\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg/\n", "git annex unlock data.dat\n", "echo \"Revised!\" >> /tmp/annex_eg/data.dat\n", "git annex add data.dat\n", "git commit -a -m \"Modified data.dat\"\n", "ls -la data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note that git-annex keeps both versions of the file:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "commit c815f594dda628d1437be8c7e38a69251acc23db\n", "Author: Michael McNeil Forbes \n", "Date: Sat Aug 18 23:22:51 2018 -0700\n", "\n", " Added data.dat\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg/\n", "git log data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Updating to the specified revision will link to the appropriate version (if available):" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Tonnes of data!Revised!\n", "Tonnes of data!" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Previous HEAD position was bee37bf... Added data.dat\n", "Switched to branch 'master'\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg/\n", "cat data.dat\n", "git checkout -q master^\n", "cat data.dat\n", "git checkout master" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Remotes: Directory" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's add a remote. We will start with a simple remote directory in another folder. This is called a [special remote](https://git-annex.branchable.com/special_remotes/). Note that this is *not* a git repository - it is simply a directory that will hold the data." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "initremote data_dir ok\n", "(recording state in git...)\n" ] } ], "source": [ "%%bash\n", "mkdir -p /tmp/annex_eg_data_dir\n", "cd /tmp/annex_eg\n", "git annex initremote data_dir \\\n", " type=directory directory=/tmp/annex_eg_data_dir encryption=none" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can move data to that repository:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "move data.dat (to data_dir...) \n", "\r", "100% 15 B 34 KiB/s 0s\r", " \r", "ok\n", "(recording state in git...)\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg\n", "git annex move data.dat --to data_dir" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now the data is not here since it was moved rather than copied." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cat: /tmp/annex_eg/data.dat: No such file or directory\r\n" ] } ], "source": [ "!cat /tmp/annex_eg/data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "... it is in the data directory:" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "here\n", "|data_dir\n", "||web\n", "|||bittorrent\n", "||||\n", "_X__ data.dat\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg\n", "git annex list " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "*(Technical note: only the current version of the data was moved. The previous revision is still in `/tmp/annex_eg`.) If you want to move all versions, follow the [instructions here](https://git-annex.branchable.com/forum/Reduce_disk_space_by_dropping_older_versions_in_local_git_annex/):*\n", "\n", "```bash\n", "git annex move --unused --to data_dir\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we need it, we can ``git annex get`` it:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "get data.dat (from data_dir...) \n", "\r", "100% 15 B 52 KiB/s 0s\r", " \r", "(checksum...) ok\n", "(recording state in git...)\n", "Tonnes of data!" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg\n", "git annex get data.dat\n", "cat data.dat" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now it is in both places." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "here\n", "|data_dir\n", "||web\n", "|||bittorrent\n", "||||\n", "XX__ data.dat\n" ] } ], "source": [ "%%bash\n", "cd /tmp/annex_eg\n", "git annex list " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cheat-Sheet" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* `get annex get `: Get a file or directory\n", "* `git annex list`: Shows which files are where." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Remotes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The utility of git-annex comes from storing data on remotes. These are managed within your repository but can point to many different places, such as folders, Google Drive, etc. which do not run git.\n", "\n", "There are a couple of points to note:\n", "\n", "* Each remote has a name that is stored in the git repository. You must make sure that your remotes have useful names so that you and your collaborators can find them. For example, the default documentation uses names such as \"google\" which are fine for testing, but should be replaced by something more specific later. I use things like the following:\n", "\n", " * `google-wsu`: My generic git annex repo in `My Drive/git-annex/` at WSU. I would use a prefix such as `mmfbb.gpe_data` to organize the data within a folder that points to where I host the master repo: https://bitbucket.org/mforbes/gpe_data/.\n", " * `google-team-wsu`: Typically I share data through a google Team Drive. This be in something like `Team Drives/Project 1/git-annex/` hosted at WSU. To specify this folder, I would use the `remote_id` option.\n", " * `google-uw`: A similar repository at UW." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Rclone" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we try setting up some other remotes that are enabled with [rclone](https://rclone.org/). First one needs to [install rclone](https://rclone.org/install/):\n", "\n", "**Mac OS X**:\n", "\n", "```bash\n", "cd ~/zips && curl -O https://downloads.rclone.org/rclone-current-osx-amd64.zip\n", "unzip -a rclone-current-osx-amd64.zip && cd rclone-*-osx-amd64\n", "mv rclone /usr/local/bin/\n", "#mv rclone.1 /usr/local/man/man1/\n", "rm -rf ~/zips/rclone-*-osx-amd64\n", "```\n", "\n", "Now one should install Daniel Dent's [git-annex-remote-rclone](https://github.com/DanielDent/git-annex-remote-rclone) project:\n", "\n", "```bash\n", "cd /usr/local/bin && curl -O https://raw.githubusercontent.com/DanielDent/git-annex-remote-rclone/master/git-annex-remote-rclone \n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Microsoft One Drive" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For my WSU buisness account I need to grant permission for RClone to access this. I could not find a way to do it without admin access, so I have filed a ticket with ITS." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Google Drive" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Currently, to use Google Drive you must use the [git-annex-remote-googledrive](https://github.com/Lykos153/git-annex-remote-googledrive) project. This requires Python 3 to run, but installed nicely from source. Follow the instructions to authenticate with your drive and you should be good to go.\n", "\n", "I install this in a conda environment using the following yml file:\n", "\n", "```bash\n", "# environment.app_gitannex.yml\n", "name: app_gitannex\n", "channels:\n", " - defaults\n", " - conda-forge\n", "dependencies:\n", " - tenacity\n", " - pydrive\n", "\n", " - pip:\n", " - annexremote\n", " - git-annex-remote-googledrive\n", "```\n", "\n", "by running \n", "\n", "```bash\n", "conda env install --file environment.app_gitannex.yml\n", "```\n", "\n", "then linking this to a `bin/` directory on path:\n", "\n", "```bash\n", "ln -s \"$(conda env list | grep app_gitannex | awk '{print $2}')/bin/git-annex-remote-googledrive\" /usr/local/bin/\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Initialize Remote\n", "\n", "Once you have this environment installed, you can create an initialize a remote.\n", "\n", "```bash\n", "git-annex-remote-googledrive setup\n", "```\n", "\n", "This will provide a URL to follow, which you must use to connect to the specific account where the data will be stored. This will store an authentication token on a file `token.json`. Now add the remote using the token:\n", " \n", "```bash\n", "conda activate app_gitannex\n", "git annex initremote \\\n", " google-wsu # Name you will use to refer to remote \\\n", " prefix=git-annex/mmfbb.gpe_data # Path in your Google Drive \\\n", " type=external externaltype=googledrive \\\n", " chunk=50MiBencryption=shared mac=HMACSHA512\n", "```\n", " \n", "Note, there are several ways of specifying where the data will be stored on the Google Drive:\n", " \n", "* `prefix=`: This allows you to specify a folder name, which will be located off of your `My Drive`. Simple, but not so versatile.\n", "* `root_id=`: When you connect to the drive online, the last part of the URL is this number. This allows you to connect with folders anywhere, such as in a Team Drive or one that is not included in your `My Drive`.\n", "\n", "### Team Drives (not working)\n", "\n", "Team Drives support is not yet working: See [this issue](https://github.com/Lykos153/git-annex-remote-googledrive/issues/28).\n", " \n", "```bash\n", "git annex initremote google-team-wsu \\\n", " root_id= \\\n", " type=external \\\n", " externaltype=googledrive \\\n", " chunk=50MiB encryption=shared mac=HMACSHA512\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Cleanup" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To remove these repositories is a bit tricky because git-annex sets its permissions to be very restrictive." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "find: /tmp/annex_eg: No such file or directory\n", "find: /tmp/annex_eg_data_dir: No such file or directory\n" ] } ], "source": [ "%%bash\n", "find /tmp/annex_eg -exec chmod u+rw {} \\;\n", "find /tmp/annex_eg_data_dir -exec chmod u+rw {} \\;\n", "rm -rf /tmp/annex_eg /tmp/annex_eg_data_dir" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# git-annex assistant" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Without the assistant, one must explicitly synchronize between two repositories. This allows one explicit control of what is on any particular machine, but does not satisfy my desire for automatic synchronization." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Sync between two folders" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First we create the two annex folders." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%bash\n", "PREF=\"/tmp/\"\n", "cd \"$PREF\"\n", "rm -rf annex1 annex2\n", "\n", "cd \"$PREF\"\n", "mkdir annex1\n", "cd annex1\n", "git init\n", "git annex init \"Annex 1\"\n", "\n", "cd \"$PREF\"\n", "mkdir annex2\n", "cd annex2\n", "git init\n", "git annex init \"Annex 2\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we add each annex as a remote." ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Initialized empty Git repository in /Users/mforbes/tmp/git/annex1/.git/\n", "init Annex 1 ok\n", "(recording state in git...)\n", "Initialized empty Git repository in /Users/mforbes/tmp/git/annex2/.git/\n", "init Annex 2 ok\n", "(recording state in git...)\n" ] } ], "source": [ "%%bash\n", "PREF=\"/tmp/\"\n", "# Add annex2 as a remote to annex1\n", "cd \"$PREF\"/annex1\n", "git remote add annex2 ../annex2\n", "\n", "# Add annex1 as a remote to annex2\n", "cd \"$PREF\"/annex2\n", "git remote add annex1 ../annex1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now start the git-annex assistant." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here I am following the 10-minute [intro walk-through](https://git-annex.branchable.com/assistant/).\n", "\n", "1. Install git-annex on a remote server (I did this on my office desktop `swan`):\n", "\n", " ```bash\n", " ssh swan\n", " sudo apt-get install git-annex\n", " ```\n", "2. Run the git-annex locally:\n", "\n", " ```bash\n", " git-annex-webapp\n", " ```\n", " \n", "3. Add a remote server.\n", "\n", " *I had issues connecting with my SSH key. Password worked fine.**\n", "\n", " \n", "\n", " \n", " \n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " # Issues and Gotchas" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## [Conflicts](https://git-annex.branchable.com/automatic_conflict_resolution/)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Git annex automatically merges files. If there is a conflict, then the conflicted files will appear with a name like `file.AAA` and `file.BBB`. You must then decide what to do with this. This mechanism is chosen so that git-annex never enters a conflicted merge state." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## SSH" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When I tried to configure a repository on a remote server with an \"existing SSH key\" the connection failed. Here is the log (enabled by going to `Preferences` in the webapp and enabling debug information in the log):\n", "\n", "* The first problem is that I tried to use an SSH alias I had setup (`swan`) which was translated to `swan.home` for some reason. The reason is [explained here](https://git-annex.branchable.com/todo/assistant_cannot_set_up_remote_repo_via_an_ssh_alias_or_an_ip_address/). The solution is to use the fully qualified host-name and manually correct after.\n", "* The second problem was that the assistant attempts to run the following checks on the remote host:\n", "\n", " ```bash\n", " cd && git config --list\n", " ```\n", "\n", " Thus, the repo must exist on the remote\n", "[2018-05-19 13:38:55.564412] process done ExitFailure 2\n", "[2018-05-19 13:43:39.138347] read: ssh-keygen [\"-F\",\"swan.physics.wsu.edu\"]\n", "[2018-05-19 13:43:39.163278] process done ExitSuccess\n", "[2018-05-19 13:43:39.163418] chat: ssh [\"-oNumberOfPasswordPrompts=0\",\"-oStrictHostKeyChecking=yes\",\"-n\",\"-p\",\"22\",\"mforbes@swan.physics.wsu.edu\",\"sh -c 'echo '\\\"'\\\"'git-annex-probe loggedin'\\\"'\\\"';if which git-annex-shell; then echo '\\\"'\\\"'git-annex-probe git-annex-shell'\\\"'\\\"'; fi;if which git; then echo '\\\"'\\\"'git-annex-probe git'\\\"'\\\"'; fi;if which rsync; then echo '\\\"'\\\"'git-annex-probe rsync'\\\"'\\\"'; fi;if which ~/.ssh/git-annex-shell; then echo '\\\"'\\\"'git-annex-probe ~/.ssh/git-annex-shell'\\\"'\\\"'; fi;if which ~/.ssh/git-annex-wrapper; then echo '\\\"'\\\"'git-annex-probe ~/.ssh/git-annex-wrapper'\\\"'\\\"'; fi;cd '\\\"'\\\"'annex'\\\"'\\\"' && git config --list'\"]\n", "[2018-05-19 13:43:40.021729] process done ExitFailure 2\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```bash\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "ssh -oNumberOfPasswordPrompts=0 \\\n", " -oStrictHostKeyChecking=yes\n", " -n\n", " -p 22\n", " mforbes@swan.physics.wsu.edu\n", " \n", " \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Questions" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "1. How does git-annex handle conflicts? *Probably diverging branches?*\n", "2. Can I symlink a directory for distribution with git-annex? (For example: suppose I want to add sync an existing folder. Can I symlink it into a git-annex folder and have it go over automatically?)\n", "3. How do I removed files? The correct way is [discussed here](http://git-annex.branchable.com/forum/How_To_Permanently_Delete_a_File__63__). Basically, one should use `git annex drop $file`, however, this will fail if there are not copies safely stored somewhere else. I am not sure how to quickly remove all files..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Cheat Sheet" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Basic add, commit, and sync:\n", "\n", " ```bash\n", " git annex add data.dat\n", " git commit -a -m \"Added data.dat\"\n", " ```\n", "* [`git annex unannex`](https://git-annex.branchable.com/git-annex-unannex/): Undo a previous addition of files.\n", "\n", "* `get annex get `: Get a file or directory from some remote so it exists on the compute where the command is run.\n", "* `git annex drop `: Drops the file or directory from the local computer. Only allowed if enough copies exist elsewhere.\n", "* `git annex copy --to `\n", "* `git annex move --to `: Move or copy the specified file or directory to the specified remote. This is manual syncing.\n", "\n", "* `git annex repair`: If things get corrupted, try this." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Inspecting a Repo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This section contains useful commands for inspecting a repository to understand its status and configuration.\n", "\n", "* Start with `git-annex-webapp`: This will open your browser, showing a list of remotes and their syncing status.\n", "\n", "![GitAnnexStatus.png]()\n", "\n", "* `git remote`: List names of remotes which are git repositories names. (Add `-v` if you want more detail.) This does not list special remotes.\n", "* `git annex info`: Information about the git repository, including a list of special remotes and summary information about disk usage.\n", "* `git annex info --fast *`: Gives information about the files and folders in the current directory such as their total size, and current size on disk.\n", "* `git annex list`: Shows which files are where.\n", "\n", "* `du -L | sort -n`, `du -Lh | sort -n`: Shows the disk usage of files in the annex (see [this discussion](http://git-annex.branchable.com/forum/__34__du__34___equivalent_on_an_annex__63__/)]\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Workflows" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### New Collaborator" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To share data with a new collaborator do the following.\n", "\n", "Prerequisites:\n", "\n", "0. Have a repository, annex, and data setup, with both the repository and the remotes accessible to the collaborator. (I.e. host the repo on bitbucket or github, and the data in a shared Google Team Drive.)\n", "1. Have the collaborator install git-annex and the appropriate drivers.\n", "\n", "Here is the workflow for the collaborator to connect with and share the data:\n", "\n", "1. Clone the repository (not the data) and go into the cloned repo.\n", "\n", " ```bash\n", " git clone git@bitbucket.org:mforbes/gpe_data.git\n", " cd gpe_data\n", " ```\n", "\n", "2. Connect to the remote. Here we use the example of Google Drive, which requires several steps.\n", "\n", " 1. Authenticate/connect to the google drive:\n", " \n", " ```bash\n", " git-annex-remote-googledrive setup\n", " ```\n", " \n", " This will provide a URL to follow, which you must use to connect to the specific account where the data will be stored. This will store an authentication token on a file `token.json`.\n", " 2. Add the remote using the token:\n", " \n", " ```bash\n", " git annex initremote google-wsu \\\n", " prefix=git-annex/mmfbb.gpe_data \\\n", " type=external externaltype=googledrive \\\n", " chunk=50MiB encryption=shared mac=HMACSHA512\n", " \n", " git annex initremote google-team-wsu \\\n", " root_id= \\\n", " type=external \\\n", " externaltype=googledrive \\\n", " chunk=50MiB encryption=shared mac=HMACSHA512\n", " ```\n", " \n", " Note, there are several ways of specifying where the data will be stored on the Google Drive:\n", " \n", " * `prefix=`: This allows you to specify a folder name, which will be located off of your `My Drive`. Simple, but not so versatile.\n", " * `root_id=`: When you connect to the drive online, the last part of the URL is this number. This allows you to connect with folders anywhere, such as in a Team Drive or one that is not included in your `My Drive`.\n", " \n", " **Team Drives Not Working**: See [this issue](https://github.com/Lykos153/git-annex-remote-googledrive/issues/28).\n", " \n", "\n", "```bash\n", "git clone git@bitbucket.org:mforbes/gpe_data.git\n", "git-annex-remote-googledrive setup\n", "\n", "git annex initremote google type=external externaltype=googledrive prefix=git-annex chunk=50MiB encryption=shared mac=HMACSHA512\n", "```\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Rename a Remote" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It turns out that renaming a special remote is a [bit tricky](https://git-annex.branchable.com/forum/How_to_rename_a_remote__63__/). There is a [todo request](https://git-annex.branchable.com/todo/renameremote/) but it has not yet been implemented. Here is how to do this:\n", "\n", "* **Regular git remotes:** If the remote is another git repository, then you can just rename it using:\n", "\n", " ```bash\n", " git remote rename \n", " ```\n", " \n", "* **Special remotes:** Special remotes such as Google Drive, Folders, etc. (which are not git repositories) need some extra work as [described here](https://git-annex.branchable.com/forum/How_to_rename_a_remote__63__/). The easiest way, however, is to run the webapp:\n", "\n", " ```bash\n", " git-annex-webapp\n", " ```\n", "\n", " When you rename a remote in the webapp, it will change everything appropriately.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Performance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Git and git-annex can become slow if you have lots of files. Here are some tips to help:\n", "\n", "* [Repositories with large number of files\n", "](https://git-annex.branchable.com/tips/Repositories_with_large_number_of_files/):\n", " * Use a version 4 index:\n", " \n", " ```bash\n", " git update-index --index-version 4\n", " ```\n", " * If `git count-objects` returns a large number (>25000), run the garbage collector:\n", " \n", " ```bash\n", " git repack -ad\n", " git gc\n", " git prune\n", " ```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# References" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## git-annex" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* [Marking a remote dead](http://git-annex.branchable.com/tips/what_to_do_when_you_lose_a_repository/): When you lose access to a remote, you may lose data, but you can mark that remote dead.\n", "* [Related Software](https://git-annex.branchable.com/related_software/): Has some references to potentially useful tools.\n", "* [Preferred Content](https://git-annex.branchable.com/preferred_content/): These are rules for specifying what gets automatically synced." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Alternatives\n", "\n", "Here is my current use-case. I run a research group at WSU with ~10 collaborators. We need to share source code, experimental data, papers, plots, and simulation data on a regular basis. Some of the collaborators are used to using Dropbox or Google Drive, which work for syncing, but have issues (listed below). WSU provides 1TB of storage through Microsoft One Drive for all students and faculty, so this would be a natural storage too, but few use it yet. We run simulations on our local machines, office desktops, a local HPC cluster [Kamiak] and online using [CoCalc].\n", "\n", "## Dropbox\n", "\n", "Currently our preferred tool for sharing. The main issue is lack of space relative to Google Drive and One Drive, as well as issues with selective sync:\n", "\n", "**Pros:**\n", "- Automatic syncing with conflict resolution.\n", "- Fast syncing.\n", "- Available Linux client (works on CoCalc).\n", "- Support symlinks. (Allows you to have your actual directory elsewhere on your computer rather than have to keep it in the drive.)\n", "- Selective syncing by folder.\n", "\n", "**Cons:**\n", "- Poor selective syncing:\n", " - Cannot exclude based on filename.\n", " - Excluding a directory from syncing deletes it locally so you have to go through a convoluted procedure of copying it (you can't move it or you wont be able to exclude the non-existent directory), excluding it, and then restoring the contents.\n", "- Potential issues with syncing `.hg` or `.git` directories (these should be typically excluded).\n", "- Limited space by default (~3GB).\n", "\n", "## Google Drive (Google Backup and Sync)\n", "\n", "Because it offers more space, we use this for syncing data with experimentalists, but it also has a range of issues. Some of these may be resolved using [Insync] (commercial one-time-fee). This may be a more attractive solution once G-Suite for Education is setup at WSU.\n", "\n", "**Pros:**\n", "- Automatic syncing with conflict resolution.\n", "- Reasonably fast syncing (slower than Dropbox).\n", "- Selective syncing by folder.\n", "- Larger default space (~15GB)\n", "\n", "**Cons:**\n", "- No Linux client (cannot use on CoCalc).\n", "- No symlink support.\n", "- Poor selective syncing:\n", " - Cannot exclude based on filename.\n", " - Excluding a directory from syncing deletes it locally so you have to go through a convoluted procedure of copying it (you can't move it or you wont be able to exclude the non-existent directory), excluding it, and then restoring the contents.\n", "- Potential issues with syncing `.hg` or `.git` directories (these should be typically excluded).\n", "\n", "### References\n", "\n", "* [Insync](https://www.insynchq.com/) is a commercial option for Google Drive that apparently alliviates many of the problems including Linux, symlink support, and ignore lists.\n", "* [Temporary fix for using symlinks with google drive (gdrive) on mac](https://gist.github.com/rtorr/2493270)\n", "* [Symbolic link between files WITHIN google drive](https://productforums.google.com/forum/#!topic/drive/mt-imQeeDxs)\n", "\n", "## Microsoft One Drive\n", "\n", "The main reason to consider this is that WSU offers all faculty and students a 1TB share. I have much less experience with this.\n", "\n", "**Pros:**\n", "- Automatic syncing (with conflict resolution?).\n", "- (Selective syncing by folder?)\n", "- Larger default space (~1TB for us)\n", "\n", "**Cons:**\n", "- No Linux client (cannot use on CoCalc).\n", "- No symlink support.\n", "- Poor selective syncing:\n", " - Cannot exclude based on filename.\n", " - Excluding a directory from syncing deletes it locally so you have to go through a convoluted procedure of copying it (you can't move it or you wont be able to exclude the non-existent directory), excluding it, and then restoring the contents.\n", "- Potential issues with syncing `.hg` or `.git` directories (these should be typically excluded).\n", "\n", "### References\n", "\n", "* [Symlinks in One Drive](https://techcommunity.microsoft.com/t5/OneDrive-for-Business/Include-files-in-OneDrive-sync-without-copying-them/td-p/74119)\n", "\n", "\n", "## git-annex\n", "\n", "[git-annex] works by having the large file actually be a symlink to the file, which is hidden (or which may not be present). When you ``git annex get`` the file, it will be transferred from the closest repository that has a copy so you can use it locally. When you are finished, you can remove the actual file to save disk space, or move it offsite.\n", "\n", "A really nice feature is that you can store the files in many place, even if they do not have git. For example, you can store the files on a remote server where you have ssh/rsync access, or in Amazon S3 storage, etc. My hope is that I can use [git-annex] to interact with the other three services. By itself, [git-annex] does not provide automatic syncing - everything must be done manually - but the [git-annex assistant] fills this role.\n", "\n", "[git-annex]: https://git-annex.branchable.com\n", "[git-annex assistant]: https://git-annex.branchable.com/assistant/\n", "[CoCalc]: https://cocalc.com\n", "[Kamiak]: https://hpc.wsu.edu/what-is-kamiak/\n", "[Insync]: https://www.insynchq.com/" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.8" }, "nikola": { "date": "2018-05-19 09:37:38 UTC-07:00", "description": "", "link": "", "slug": "git-annex", "tags": "", "title": "git-annex", "type": "text" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": true, "toc_position": { "height": "calc(100% - 180px)", "left": "10px", "top": "150px", "width": "270.390625px" }, "toc_section_display": "block", "toc_window_display": true } }, "nbformat": 4, "nbformat_minor": 2 }