Contexts

Context is the most important thing when you deal with complex environment. You should learn figuring out where you are wherever you are and bring the context to others.

Your Python code in Jupyter Notebooks hosted on Colab is running on the following nested environment

Colab

Docker Machine

Shell

Jupyter

python code

!shell command

python code

Colab gives you access to ⟼ Docker runtime environment that runs ⟼ some shell which starts ⟼ Jupyter web-server where ⟻ Colab connects to and runs ⟼ cells with python code or shell command.

You even can access to the python variable from other cells executed previously

You knew it if you use Jupyter before. But what does it mean?

That means python namespace lives forever until you restart the kernel of Jupyter Notebook

Inside Jupyter the lifetime of shell environment (name it session) is shorter and spans single line, not even single cell.

Exclamation point ! starts new instance of the shell and teleports you from Cell to Shell until the line ends

Jupyter

!shell command

!shell command

The scope of namespaces is not important when you're working in the shell interactively within single session, but it does matter when you're making your own Docker Image, especially with environment driven control flow.

Connect Colab to Local Jupyter

To mount your Google Drive into Jupyter hosted on Colab run cell below (not locally) and follow Colab intructions

Mount means you attach remote disk and inject its content somewhere on current filesystem. Without mounting you will have to rebuild disk content of Jupyter instace everytime you restart Colab.

However, you have an option to switch Colab runtimes from Google host to your own local host (e.g. your laptop) and work on your home disk. You will need Anaconda. Open your terminal and do the following.

Install module for web sockets:

pip install --upgrade 'jupyter_http_over_ws>=0.0.7' && jupyter serverextension enable --py jupyter_http_over_ws

Run your Jupyter locally allowing Colab access to it:

jupyter-lab --NotebookApp.allow_origin='https://colab.research.google.com' --port=8888 --NotebookApp.port_retries=0

Copy from your terminal and paste Jupyter backend URL into Colab:

http://localhost:8888/?token=e01060f000d010b0800060f0c0a080600050f0e080d06050

In this case your Jupyter Notebooks still are hosted on Google Drive, but Jupyter web-server and it's environment are hosted on your machine

Colab

Local Machine

Shell

Jupyter

python code

!shell command

python code

Away

You can be free if you can explain or guess what the following command does

(set | cut -d= -f1) | grep -v -E "'$(env | cut -d= -f1 | tr '\n' '|')'"

First Hour at the Terminal

Who?

What is my login name?

Who else is there?

Typically all users has own directory at /home, but not on Colab where you're alone

When I was here last time?

What is my machine name?

Where?

Where am I on the filesystem? Print work directory

What is on the filesystem? List long format with all (including hidden) files

There are two default directories: current . (omitted in ls)

and parent ..

Here parent directry is root / directory

What is directory structure?

Who is the owner of the files?

Add new user. Change the owner Recursively (in depth of the directory)

Remove the user. Check the owner

There is no more user me, but files still hold his UID (user ID) and GID (group ID): 1000. Change back the owner

Space

Disk

How much sample_data uses the disk in sum?

How much space is free on the disk the /content is mounted to?

Info about disk devices

and its' mount points

Memory

Free memory on CPU

Free memory on GPU

CPU Cores

Number of logical cores

CPU info

Namespaces

As you might see echo command outputs text string to the terminal.

To understand interrelations between parent and child shells, let's create shell-script inside.sh with the following content

echo "inside: $namespace PID:$$"
namespace=internal
echo "inside: $namespace PID:$$"

You can do it with changing the context of commands output.

Operator >

redirects shell-command output from the terminal to the head of the file

Operator >>

redirects shell-command output from the terminal to the tail of the file

You haven't seen any output of echo that has been input for inside.sh

Sub-shell and Super-shell

Subsstitutions

Shell strings (quoted stuff) possess ability to be constant or variable. Compare

Permissions

Although inside.sh extension .sh means executing by shell, you can't execute text file ordinary way until you make it executable

Change permission mode of inside.sh to executable for the owner (user)

When you wonder what command in the script is outputting to terminal, you can enable debug mode set -x on the head of the script