Draft Forbes Group Website (Build by Nikola). The official site is hosted at:

Public/ForbesGroupWebsite / draft / mac-os-x / index.ipynb

⁵⁹¹⁰ views
License: GPL3
ubuntu2004

Kernel: Python 3 (ipykernel)

Here are some notes about how I configure Mac OS X.

MacPorts

MacPorts brings to Mac OS X many programs available on Linux. The programs are compiled from source, which means it can take a while to install everything and that they need the Xcode command line tools installed:

These are installed in /opt/local/ and to use them, you should add /opt/local/bin to your initialization scripts. (I do this with various configuration files in my configurations project: see for example my ~/.environment_site file.)

I do not like running these commands as root, so once I install MacPorts, the first thing I do is make everything owned by an unprivledged user (me).

sudo find /opt/local -exec chown admin {} +

This has a couple of complications (see rsync below) but works well for the most part. If you do this, be sure to run port without sudo which will be counter to the usual instructions found on the internet. If there is a permissions issue, then check where and deal with it on a case-by-case basis.

Alternatives

Homebrew provides a popular alternative. From what I can gather from various discussions, MacPorts is more stable, better designed, and more reliable. Some key points are:
- MacPorts installs in /opt/local. In contrast, Homebrew puts everything in /usr/local while also changing the permissions of everything in there to the single user running. This might make sense with a single user – and indeed is similar to what I do with permissions for /opt/local, but apparently makes it much more difficult for multi-user machines. It also presents a security hazard and is bad practice. For me, this is an issue, because I expect everything in /usr/local to be customize by me – stuff I install from source for example.
- MacPorts installs redundant but isolated libraries, so does not break when Apple updates things. In contrast, Homebrew uses system libraries. Makes it smaller, but less robust.

Discussions

Useful Commands

See Common Tasks for more details.

Information:

port outdated
port installed inactive
port installed requested
port echo leaves
port dependents <portname>
port contents <portname>

Marking ports as requested (so they don't appear as leaves)
```
port setrequested ...
```
Marking ports as requested (so they don't appear as leaves)

Updating and Cleaning

port selfupdate
port upgrade outdated
port uninstall inactive
port uninstall leaves         # Removes one level of leaves
port uninstall rleaves        # Removes all leaves (unrequested ports)

Here is what I typically install:

port install aspell aspell-dict-en 
port install bash-completion
port install ffmpeg +nonfree ImageMagick pngcrush pdf2svg
port install exiftools
port install coreutils fswatch gawk wget tree shellcheck ncdu
port install bzr   # Needs python 2.7!!!  Maybe skip for now. 
port install cvs git myrepos
port install openssl
port install gsl
port install cmake gmake gperf swig-python
port install symlinks
port install rmtrash
port install munin
port install lua-luarocks    # For Lmod: see below
port install jq              # JSON parser - used for migrating to hg.src.ht
port install multimarkdown
port install py-altgraph graphviz   # For visualizing dependencies
port install sshuttle  # Tool for simple ssh VPN.
port install smartmontools  # Tools for monitoring harddive performance etc.
port install rust cargo # Rust compiler for qiskit-terra
# port install pkgconfig # I needed this for pycairo when developing manim... use conda!
# port install python36  # Broken on Mac OS X under arm.  No longer supported
port install npm9 
port install python37 python38 python39 python310 python311
port install pandoc

Or, all of them:

port install aspell aspell-dict-en \
            bash-completion \
            ffmpeg +nonfree ImageMagick pngcrush pdf2svg \
            exiftools\
            coreutils fswatch gawk wget tree shellcheck ncdu \
            cvs myrepos \
            openssl \
            gsl \
            cmake gmake gperf swig-python \
            symlinks \
            rmtrash \
            lua-luarocks \
		    jq \
            multimarkdown \
            py-altgraph graphviz \
            sshuttle \
            rust cargo \
            npm9 \
            python37 python38 python39 python310 python311 \
            pandoc

# These need some attention.
port install bzr git munin smartmontools

The rsync package needs access to /Library/LaunchDaemons.

sudo chmod a+w /Library/LaunchDaemons
port install rsync
sudo chmod a-w /Library/LaunchDaemons

Cleaning

To clean your installation you might want to do the following:

port clean --all all        # Can be very slow...
port uninstall inactive
port uninstall rleaves

According to this answer, you might also be able to delete the following:

rm -rf /opt/local/var/macports/software

After installation, my typical setup consumes about 1.7GB of disk space – most of which is due to ffmpeg:

$ du -sh /opt/local/
1.7G	/opt/local/

Dependencies

In [ ]:

!curl -O https://raw.githubusercontent.com/Synss/macports_deptree/master/port_deptree.py
!pip install --user altgraph
!/opt/local/bin/python27 port_deptree.py python27 | dot -Tpdf | open -fa Preview
#!python port_deptree.py py-altgraph | dot -Tpdf | open -fa Preview
#!python port_deptree.py ffmpeg +nonfree | dot -Tpdf | open -fa Preview

Custom Portfiles

If you need to create your own Portfile or slightly modify an existing one you can quite easily by creating your own Local Portfile Repositories. Here is how I did this using a local repository in ~/src/ports to edit the sshuttle Portfile to depend on Python 3.9 (so MacPorts only brings in one version of python).

Add file:///Users/mforbes/src/ports to /opt/local/etc/macports/sources.conf.

# /opt/local/etc/macports/sources.conf
...
file:///Users/mforbes/src/ports
rsync://rsync.macports.org/release/tarballs/ports.tar [default]

Create the Portfile:

mkdir -p /Users/mforbes/src/ports/net/sshuttle
curl https://raw.githubusercontent.com/macports/macports-ports/master/net/sshuttle/Portfile > /Users/mforbes/src/ports/net/sshuttle/Portfile

Edit the Portfile:

#/Users/mforbes/src/ports/net/sshuttle/Portfile
...
python.default_version 39
...

Update the index:
```
cd /Users/mforbes/src/ports
portindex
```

Update the port and clean:

port uninstall sshuttle   # Get rid of old version
port search sshuttle
port install sshuttle
port uninstall inactive
port uninstall leaves
port uninstall rleaves

References

Shell (bash)

Sometimes one might like to use another shell instead of /bin/bash. For example, one might like to use the version /opt/local/bin/bash provided by MacPorts so that one can use the bash-completion package. To do this, we first install the shell, then enable it, finally we change the user shell:

port install bash-completion
echo "/opt/local/bin/bash" | sudo tee -a /etc/shells
chsh -s /opt/local/bin/bash
mkdir -p ~/.local/share/bash-completion/completions

The for completions, I do things like:

poetry completions bash > ~/.local/share/bash-completion/completions/poetry.bash

Finally, in my ~/.bashrc file, I have:

#~/.bashrc
...
if [ -f /opt/local/etc/profile.d/bash_completion.sh ]; then
  . /opt/local/etc/profile.d/bash_completion.sh
fi

Startup Scripts

To see exactly what happens when you run bash, you can do the following:

bash -xlic exit 2>@1 # Login + Interactive shell
bash -xlc exit 2>@1  # Login but no interactive
bash -xc exit 2>@1   # Neither login nor interactive
sh -xc exit 2>@1     # Run when you connect via ssh to run a command

To see the nature of your shell, you can run echo $0 $-. The first $0 will show you the shell being run, while the latter $- shows the current set of options. Interactive shells will have i in $-, while login shells will have - starting $0

For example, when you SSH to your computer you see:

$ echo $0 $-
-bash himBHs
$ ssh mforbes@localhost               # Actually log in.
$$ echo $0 $-
-bash himBHs
$$ exit
$ ssh mforbes@localhost 'echo $0 $-'  # Executing a command - no login
bash hBc

The following sequence can be deduced for Mac OS X (see INVOCATION in man bash):

Interactive Login Shells – i.e. whenever you start a new Terminal window, or when you SSH in and start an interactive session, or when you call bash -li:

/etc/profile
First of ~/.bash_profile, ~/.bash_login, ~/.profile. (Inhibit with bash --noprofile.)
On exit: ~/.bash_logout

Interactive non-Login Shells – i.e. when you run bash after opening a terminal:

~/.bashrc

Non-Interactive, Non-Login Shells - i.e. when you run a command with bash -c <cmd> or run a command with SSH:

$BASH_ENV: I.e. nothing in general, but you can trigger loading of a file with something like:
```
BASH_ENV='~/.bashrc' bash -c ...
```

XCode

XCode is huge (~10GB), so I remove it (just drag XCode.app to the trash) and just keep the command line tools installed:

xcode-select --install

After agreeing to the licence, you should see

$ xcode-select -p
/Library/Developer/CommandLineTools

Note: after you do this you will likely see the following warning from MacPorts:

* `Warning: xcodebuild exists but failed to execute`
* `Warning: All compilers are either blacklisted or unavailable; defaulting to first fallback option`

Conda

In install Conda with the following in mind:

I use Miniconda as opposed to the full Anaconda: see Anaconda or Miniconda? distribution. If I need Anaconda, then I install it in a special environment.
I install Conda as the conda user so that I can't accidentally muck up the environments. This simulates what happens on HPC compute clusters etc. where we share conda environments.
Note: I originally tried to use the admin account for this role, but this caused problems because administrators get added to the admin group, and then have some permissions in the conda environment that confuse conda into thinking that it can write to the package directory, which it can't. Conda relies on this to switch to local installs.

ARM issues: Some python libraries are not yet available for the ARM platform. In these cases, we need to create an environment with the osx-64 subdir.

CONDA_SUBDIR=osx-64 conda ...
conda config --env --set subdir osx-64
CONDA_SUBDIR=osx-arm64 conda ...
conda config --env --set subdir osx-arm64

CONDA_SUBDIR=osx-64 conda create -n tst64 python=3.9
conda activate tst64
conda config --env --set subdir osx-64

Environments are specified with environment.yml files and Picky for Conda should be used to lock these.
Custom work should be done in a virtual environment that sits on one of these base conda environments.

Here is the complete setup process:

Create a conda user account. (Probably easiest just to use the GUI.)

I was going to enable passwordless su access, but it seems easier just to enable an ssh alias:

#~/.ssh/config
...
Host conda_local
  User conda

Host admin_local
  User admin

Host *_local
  Hostname localhost
  ForwardAgent yes

Now copy the keys:

ssh-copy-id conda_local

Create the base environments:

sudo mkdir -p /data/apps/conda    # Remove an old installation if needed
sudo chown conda /data/apps/conda
ssh conda
mkdir -p zips
cd ~/zips
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O
bash Miniconda3-latest-MacOSX-x86_64.sh -bfp /data/apps/conda
/data/apps/conda/bin/conda init
. ~/.bash_profile

# Update the base environment with anaconda-client, then from my anaconda page:
# https://anaconda.org/mforbes/base
/data/apps/conda/bin/conda install anaconda-client -n base
/data/apps/conda/bin/conda update -n base mforbes/base
echo 'eval "$(mmf_setup -v -H)"' >> ~/.bash_profile

# Added some bare python environments.  I use these for testing with Nox.
for py in 3.6 3.7 3.8 3.9; do
  /data/apps/conda/bin/conda create -y -c defaults --override-channels -n "py${py}" python=${py}
done

Create some specialized environments. Ideally these should be able to be created from my anaconda channel:

ssh conda
conda env update mforbes/jupyter
conda env create mforbes/work

If you are updating them, it is useful to have them locally:

ssh conda
mkdir -p repositories
cd repositories
hg clone ssh://git@hg.iscimath.org:11022/mforbes/configurations
conda env create

Networking

To see your MAC address:

ifconfig en0 ether

If you need to change it, then you can spoof it with

sudo ifconfig en0 ether xx:xx:xx:xx:xx:xx

Note: make sure that en0 is the correct device. You might want to check the ouput of ifconfig first to see that this is indeed the active device.

VPN

To connect to various services at WSU, I downloaded their Global Protect client. This is needed for some reason. In particular, without using the VPN, the following does not work:

(swan) $ nc -l 12345   # Start a netcat server listing on port 12345 on swan

then on my Mac:

(Rentdem) $ nc swan.physics.wsu.edu 12345   # Start a netcat client

Without VPN, I can only send about 4 messages in either direction, then the connection hangs. Not sure why.

Wireshark

Wireshark is a full-featured debugging tool for networks. I install it and add it to the PATH so I can use it in a terminal.

SSH

To allow SSH acces to your mac (i.e. from one account to another) the Admin should go to System Preferences/Sharing and enable Remote login.

To tell KeyChain to remember your SSH keys do the following:

/usr/bin/ssh-add -K [path/to/private SSH key]

Note: It is important to make sure you use the Apple version of ssh-add here and not something installed by port, conda, etc.

SSHFS

You can use SSH to mount remote file systems locally with sshfs. On Mac OS X, do this by installing FUSE and the associated SSHFS package from that site. Once it is installed, you can mount a filesystem as follows:

mkdir -p ~/Volumes/swan
sshfs -o follow_symlinks swan:/ ~/Volumes/swan

To unmount:

umount ~/Volumes/swan

SSH Tunnels (SOCKS/VPN)

sshuttle -r user@host 0.0.0.0/0 -vv

This will forward all traffic through the specified host. See https://superuser.com/a/757974. sshuttle can be installed with pip or macports.

BBCP

Related to SCP is bbcp which can be downloaded for OS X here. This is a fast tool for transferring files. I just copy it to /usr/local/bin/bbcp.

LaTeX

I install the MacTeX which puts most things in /usr/local/texlive and also includes BibDesk, Skim etc. When you upgrade from one year to the next (say 2017 to 2018), the new installation will be placed in /usr/local/texlive/2018 leaving the old version in /usr/local/texlive/2017. These can be big (~5.4GB) so you might want to move the old version off your hard-drive (but keep it for a bit in case things go wrong.)

Once you upgrade, you may need to make a few changes to keep things working, setting the path for the LaTeX programs. These should be set to

/Library/TeX/texbin

but might need to be updated. Check in the preferences of the following packages if things do not work as expected:

BibDesk: (Seems okay with symlink above.)
TeX Live: (Needed path refreshed as it seetermsn to usoe thMe absolute path, but can refresh this for you if you confirm.)
LaTeXiT: (Seems okay with symlinks.)

Safari

Extensions

Ad Blocking: There are several extensions that block ads.
- AdBlock: I have been using this and am reasonably happy with it. The blocking of ads in YouTube is especially valuable.
- AdBlock Plus: Another alternative that is almost the same (small UI differences).
- AdGuard for Safari: Was recommended in an uBlock issue about Safari. Does not block YouTube ads, but I have not tried the full AdGuard (just the browser extension).
For a comparison of the first two, see AdBlock vs AdBlock Plus.
StopTheMadness: Some websites disable useful features such as copy and paste. This extension claims to restore this functionality.
I used a different approach - temporarily disable JavaScript. This can be done under the Safari Develop/Disable JavaScript menu item, but I also enabled a shortcut ⌘J to Disable JavaScript..

Emacs

There are several options for using Emacs on Mac OS X:

GNU Emacs for Mac OS X: I use this. It is a fairly generic version of emacs with traditional key bindings. The only annoyance is that one must use Esc for the Meta key. (M-x is achieved by "pressing and releasing Esc, then pressing x.)
Aquamacs: This more tightly integrates with the OS X interface, but is significantly slower.

Configuration

I install the following packages using the Options/Manage Emacs Packages menu option. Note, there are three main package archives. I use these with the following preference: 1) gnu, 2) marmalade, 3) melpa (see this discussion for more details.

Editing Features

autopair: Automagically pair braces and quotes like TextMate.
org: Outline-based notes management and organizer.

Modes:

auctex-latexmk: Add LatexMk support to AUCTeX.
haskell-mode: A Haskell editing mode.
lua-mode: A major-mode for editing Lua scripts.
markdown-mode+: Extra functions for markdown-mode.
markdown-preview-mode: Markdown realtime preview minor mode.
markdown-toc: A simple TOC generator for markdown file.
yaml-mode: Major mode for editing YAML files

Elisp Debugging (init files)

benchmark-init: Benchmarks Emacs require and load calls.
bug-hunter: Hunt down errors by bisecting elisp files.
use-package: A configuration macro for simplifying your .emacs.

Git: (I don't really use these, but they were recommended at some point.)

git-commit-mode: Major mode for editing git commit messages [github].
git-rebase-mode: Major mode for editing git rebase files [github].
magit: A Git porcelain inside Emacs.

Python: Modes for working with python.

conda: Work with your conda environments
python-mode: Python major mode. This has some major problems when using Tramp for remote editing, so I do not use it any more.
elpy: Emacs Python Development Environment

Web/Blog:

sass-mode: Major mode for editing Sass files.
scss-mode: Major mode for editing SCSS files.
json-mode: Major mode for editing JSON files.
jinja2-mode: A major mode for jinja2.
tidy: Interface to the HTML Tidy program.

I load these in my .emacs file with the following code

;; Make sure required packages are installed
(setq package-list
      '(use-package
	autopair
	org
	auctex-latexmk
	haskell-mode
	lua-mode
	markdown-mode+
	markdown-preview-mode
	markdown-toc
	yaml-mode
	benchmark-init
	bug-hunter
	use-package
	;git-commit-mode
	;git-rebase-mode
	;magit
	conda
	;python-mode
	elpy
	sass-mode
	scss-mode
	json-mode
	jinja2-mode
	tidy))

;; list the repositories containing them
(setq package-archives
      '(
	("gnu" . "http://elpa.gnu.org/packages/")
	("marmalade" . "http://marmalade-repo.org/packages/")
	("melpa" . "http://melpa.milkbox.net/packages/")
	;("elpa" . "http://tromey.com/elpa/")
	))

;; activate all the packages (in particular autoloads)
(package-initialize)

;l fetch the list of packages available 
(unless package-archive-contents
  (package-refresh-contents))

;; install the missing packages
(dolist (package package-list)
  (unless (package-installed-p package)
    (package-install package)))

BBEdit and .plist Files

BBEdit is another editor with one killer feature - the ability to decode and edit Apple binary property list (.plist) files. One can edit these with XCode (if you install it all), convert it with plutil -convert xml1 config.plist, or use defaults write com.apple...., but it is much easier just to open it with BBEdit.

iPhone/iPad

Backups

If you make backups of your iPhone or iPad with iTunes, you can locate these with

iTunes/Preferences/Devices/Device Backups and right-click to Reveal in Finder.

These files, however, are not organized and obfuscated. To interpret this data you seem to need to use a third party application. Most of these have a demo mode that will allow you to view the files, but put severe limitations on how much data you can actually extract without paying.

iPhoneBackupViewer: View photos, but must save them one at a time.

Useful Applications

GPG

This is useful for encryption and signing files. I use the vi plugin which allows you to edit encrypted files.

https://gpgtools.org
https://www.vim.org/scripts/script.php?script_id=3645
Simply move the gnugpg.vim script into ~.vim/plugin:
```
mkdir -P ~/.vim/plugin
```

Preference Panes

Choosy: A "browser" that redirects websites to specific browsers. I am just trying this, but hope to use it to open CoCalc websites exclusively in Chrome for example even though I generally prefer Safari.
MultiBrowser: A similar "bowser" to Choosy that allows you to select which browser to open a link with. Make this your default browser, then you can choose whenever you open a URL. I have been using this for a while and it works quite nicely. Choosy might be better in that it allows you to specify rules, but MultiBrowser is very simple and works well.

Vagrant

Vagrant provides a platform for running code in isolated virtual environments. If you need more fine-grained control and provisioning, [Docker] is another option, but I use Vagrant for developing websites for example.

Docker

I installed the Docker App for OS X. It seems like one can install this without root access (opening it as an unprivileged user says it will "create symlinks in ~/.docker/bin" instead of /usr/local/bin which requires root access.

After installing as admin, I ran Docker.app and then went to the Preferences and changed the location of images to /data/apps/Docker. This fails, and apparently one needs to create a symlink:

ssh admin
mkdir /data/Users/admin/Library/Containers/com.docker.docker
cd ~admin
ln -s /data/Users/admin _data
ln -s ~/_data/Library/Containers/com.docker.docker ~admin/Library/Containers

I had to do this as a user. Force-quitting all docker processes using Utility Manager, then moving via Finder worked.

(Docker has an option for excluding from Time Machine backups, but I use SuperDuper!)

CoCalc

One can run CoCalc locally using the [CoCalc Docker image][]. This takes about 25GB.

docker run --name=cocalc -d -v ~/cocalc:/projects -p 443:443 sagemathinc/cocalc-aarch64

This should run CoCalc locally so you can access it via

https://localhost

Note: you might run into certificate issues. You can allow HTTPS for localhost in chrome by going to

chrome://flags/

and selecting Allow invalid certificates for resources loaded from localhost. [CoCalc Docker image]: https://github.com/sagemathinc/cocalc-docker#cocalc-docker-image

AWS-CLI

I don't really use Docker on my Mac, but install it so I can provision images on AWS. To do this I did the following:

Make a directory for installing the aws-cli without sudo:

sudo mkdir /usr/local/aws-cli/
sudo chown mforbes /usr/local/aws-cli/

Download and install the aws-cli as a local user ("Install for Me Only").

$ ln -s /usr/local/aws-cli/aws /usr/local/aws-cli/aws_completer /usr/local/bin/
$ aws --version
aws-cli/2.1.10 Python/3.7.4 Darwin/18.7.0 exe/x86_64 prompt/off

Install the Docker App for OS X. Note: I though I could use port install docker docker-machine and save on some disk space, but this needs Vagrant, so no clear savings there. There are some relevant discussions in this direction though:
- Building containers without Docker (Alex Ellis's Blog Jan 2020).

Disk Space:

Docker uses lots of space. Some can be reclaimed with:

docker ps -a
docker stop <container id>
docker rm <container id>
docker images
docker rmi <images>
docker system prune
docker run --privileged --pid=host docker/desktop-reclaim-space

https://docs.docker.com/docker-for-mac/space/

Environment Modules (Lmod)

To maintain some coherence with HPC environments, we install some version of the Environment Modules package. Here we use the Lmod variant as this is used on our local cluster.

port select --set luarocks lua53-luarocks
luarocks install luaposix
luarocks install luafilesystem
LUAROCKS_PREFIX=/opt/local/share/luarocks
export LUA_PATH="$LUAROCKS_PREFIX/share/lua/5.3/?.lua;$LUAROCKS_PREFIX/share/lua/5.3/?/init.lua;;"
export LUA_CPATH="$LUAROCKS_PREFIX/lib/lua/5.3/?.so;;"


APP="Lmod"
VER="8.7"
NAME="${APP}-${VER}"
FILE="${NAME}".tar.bz2
cd ~/zips/
wget https://sourceforge.net/projects/lmod/files/"${FILE}"
mkdir -p ~/src
cd ~/src/
tar -jxvf ~/zips/"${FILE}"
cd "${NAME}"
./configure --prefix=/data/apps/
make install

To use these, I add the following to my ~/.environment_site file:

# Site specific bash environment init file; -*-Shell-script-*-
# dest = ~/.environment_site #### Keep this as the 2nd line for mmf_init_setup

# This file is loaded by .environment and should be used to set the
# environment for site-specific customizations

...

test -f "/data/apps/lmod/lmod/init/profile" \
  && . "/data/apps/lmod/lmod/init/profile"

...
# Load useful modules.
module use ~/.modules
module load use.own cuda git-annex node mongodb fftw # anaconda

For details about what these do, see my modules configurations folder.

Graphics

MacSVG

Program for producing SVG graphics.

GIMP

Opensource replacement for Photoshop.

CMake

Needed by some applications (I needed it to build VisIt plugins. I simply added this line to my environment_site file:

prepend_path PATH "/Applications/CMake.app/Contents/bin"

Julia

Some tools - in particular the diffeq library - use Julia, but it is not so easy to install via Conda. I install the native application. After installing I had to link it:

ln -s /Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia ~/.local/bin/
conda activate jupyter
export JUPYTER="$(type -p jupyter)"
julia

Then, using julia:

using Pkg
Pkg.add("IJulia")
Pkg.build("IJulia")

Note: If this fails, you might need to manually clone the registry:

git clone https://github.com/JuliaRegistries/General.git ~/.julia/registries/General

Then, in python, I needed to:

conda activate work
pip install diffeqpy
python -c "import diffeq;diffeq.install()"

This installed everything needed into ~/.julia.

References

R and Julia Kernels not available in Jupyter notebook.

Sound: Black Hole

I used to use SoundFlower to route audio on my Mac. This is no longer supported, but they recommend Loopback. Another option is BlackHole as discussed in this article. JACK is another option (this is used by Audacity – a great open-source tool for editing audio.)

LICEcap

Simple program for capturing activity on your screen and turning it into an animated GIF. Useful for demonstrating how to use an interface.

Sim Datlonism

Color-blind simulator. Useful for checking graphs, posters, etc.

Microsoft Office

I have the problem of running with fairly limited disk space, so installing the complete Office suite was prohibitive. For a long time I stuck with Microsoft Word for Mac 2011, which I got from the department. It appears now that individual apps are available through the App Store:

Synergy

I have Synergy which allows me to control different computers on my network with one keyboard and mouse.

AVG

I install AVG for virus protection. I don't enable the continuous monitoring, but scan files before I install them.

NPM

I install NodeJS from the downloads page and extract the macOS Binary in /data/apps/node, then add /data/apps/node/bin/ to my path.

Update npm with:

npm install -g npm@latest
npm audit fix
npm install -g jsonlint

Monterey

Here is how I setup my new M1 Macbook Pro. This is a university-owned computer (purchased with my NSF grant) so I keep my personal account separate from my work account.

Partitioned HD with one Container 'Macintosh HD' and a Volume 'Data'. The idea is to allow quick backup of the OS to previous hard-drives (256GB partitions), putting easily installable data on the separate partition. Note: Using separate partitions is a mistake as it precludes fixing things if you run out of space.
Install OS from scratch (only use migration assistant later for the user account. Recommended several places.)
Create admin account: (Disallow Apple ID to reset this password.)
Create a firm link for sudo vi /etc/synthetic.conf: add data /Volumes/Data.
Create additional accounts:
- wmforbes (work)
- mforbes (personal - to be copied from old computer)
- conda (for conda installations)
  - Hide from login: sudo defaults write /Library/Preferences/com.apple.loginwindow HiddenUsersList -array-add conda
  - Use Finder -> Get Info to give write permission to /usr/local/bin and /data/apps where we will make symlinks.
- Users should setup SSH access to admin and conda if needed:
```
# ~/.ssh/config
...
Host admin
  Hostname localhost
Host *admin
  User admin
Host conda
  Hostname localhost
  User conda
...
Host *
  UseKeychain yes
  AddKeysToAgent yes
```
```
ssh-keygen  # If needed
ssh-copy-id admin
ssh-copy-id conda
```
Change some preferences in System Settings:
- Trackpad speed, scrolling direction, keyboard repeat rate, Fn key, Caps-Lock->Control).
- Sharing and enable Remote login.
- Security & Privacy/Firewall: Trying Block all incoming connections. Local SSH still works, but this will probably break AirDrop etc. (incoming is broken.)
Change shell to bash chsh -s /bin/bash.

Create the following files:

# ~/.inputrc

"\M-[A":        history-search-backward
"\M-[B":        history-search-forward
"\e[A":        history-search-backward
"\e[B":        history-search-forward

# ~/.bash_aliases
export BASH_SILENCE_DEPRECATION_WARNING=1   # Don't warn about zsh
# User specific aliases and functions
export INPUTRC=~/.inputrc

Install Intego virus protection VirusBarrier from App store.
Install GNU Emacs for Mac OS X.

Install Markdown and YAML mode (so I can edit this file!)

Install Macports:

sudo find /opt/local -exec chown admin {} +
port selfupdate
port install ...

Change shell again:

echo "/opt/local/bin/bash" | sudo tee -a /etc/shells
chsh -s /opt/local/bin/bash

Add path for all users of bash:

sudo chmod a+w /etc/bashrc
vi /etc/bashrc
# Insert the following at the top
# export PATH="${PATH}:/opt/local/bin"
# It is dangerous to put it first though... eg git fails with pip.
sudo chmod a-w /etc/bashrc

Install Lmod:

port select --set luarocks lua53-luarocks
luarocks install luaposix
luarocks install luafilesystem
LUAROCKS_PREFIX=/opt/local/share/luarocks
export LUA_PATH="$LUAROCKS_PREFIX/share/lua/5.3/?.lua;$LUAROCKS_PREFIX/share/lua/5.3/?/init.lua;;"
export LUA_CPATH="$LUAROCKS_PREFIX/lib/lua/5.3/?.so;;"


APP="Lmod"
VER="8.7"
NAME="${APP}-${VER}"
FILE="${NAME}".tar.bz2
cd ~/zips/
wget https://sourceforge.net/projects/lmod/files/"${FILE}"
mkdir -p ~/src
cd ~/src/
tar -jxvf ~/zips/"${FILE}"
cd "${NAME}"
./configure --prefix=/data/apps/
make install

Install TexLive with MacTeX. To reduce disk space, I first linked the install directory to /data/apps/texlive:

mkdir /data/apps/texlive
sudo ln -s /data/apps/texlive /use/local/texlive

Reboot.

Install Conda as conda: On a shared system where many people might need to administer conda environments, we would have a separate account, but for a single-user computer we can use admin.

sudo mkdir /data/apps
sudo chown admin /data/apps
mkdir /data/apps/conda_arm64
mkdir /data/apps/conda_x86_64
ln -s conda_arm64 /data/apps/conda
sudo chown conda /data/apps/conda*
sudo chgrp conda /data/apps/conda*
ssh conda
cd ~/zips
bash Miniconda3-latest-MacOSX-arm64.sh -ubp /data/apps/conda
/data/apps/conda/bin/conda init bash zsh
# Start new shell
conda update -y conda
conda install -n base -y anaconda-client
conda update -n base --all -y

conda env update -n base mforbes/base.minimal   # Fails first time?
conda env update -n base mforbes/base.minimal   # Works next time?
conda update -n base --all -y

mkdir /data/apps/pipx
export PIPX_HOME=/data/apps/pipx
export PIPX_BIN_DIR=/usr/local/bin

echo "export PIPX_HOME=${PIPX_HOME}" >> ~/.bash_profile
echo "export PIPX_BIN_DIR=${PIPX_BIN_DIR}" >> ~/.bash_profile

for app in pdm poetry yapf black nox \
        nbdime jupytext nbstripout \
        poetry2conda conda-lock condax rst-to-myst twine \
        sphobjinv mercurial mmf_setup snakeviz \
        grip; do
    pipx install ${app}
done
pipx inject nox nox-poetry poetry 
pipx inject pdm pdm-shell
pipx inject mercurial hg-git hg-evolve
pipx install git+https://github.com/cookiecutter/cookiecutter.git@2.0.2#cookiecutter

# https://github.com/mariusvniekerk/condax/issues/16
#echo "link_destination: /usr/local/bin/" > ~/.condaxrc
# Currently this fails because it pulls in py=11.0.
    # https://github.com/yamaton/condax
    # https://github.com/mariusvniekerk/condax/issues/3 
#condax install anaconda-project

conda env create mforbes/hg
conda update -n hg --all

conda env create mforbes/jupyter
conda update -n jupyter --all
# Issues with itkwidgets, but pip seems to work

conda create -n anaconda-project anaconda-project
ln -s /data/apps/conda_arm64/envs/anaconda-project/bin/anaconda-project /usr/local/bin/

# conda env create mforbes/work
# No good option for rclone or filprofiler, also must force python<3.10
# and fix mmfutils.  Also, don't use macport git
conda env create -f ~/zips/environment.work.yml
conda update -n work --all

conda env create mforbes/hg
conda update -n hg --all
sudo ln -s /data/apps/conda/envs/hg/bin/hg /usr/local/bin/

One could use conda to provide environments for basic pythons, but becuase of the need for libssl which is provided by MacPorts, we use it instead. If you want to use Conda, do this:

# Note: only python 3.8 through 3.10 are supported on ARMs
# https://stackoverflow.com/a/70219965
for py in 3.6 3.7; do
  conda create -y -n py${py}
  conda activate py${py}
  conda config --env --set subdir osx-64
  conda install -y python=${py}
  conda update --all
  conda deactivate
done
  
for py in 3.8 3.9 3.10; do
  conda create -y -n py${py} python=${py}
  conda update -n py${py} --all
done

As admin, link these:

for py in 3.6 3.7 3.8 3.9 3.10; do
  sudo ln -s /data/apps/conda/envs/py{$py}/bin/python3 /usr/local/bin/python${py}
done

Install Apps:
- Chrome
  - Extensions:
- Firefox
- LastPass
  - Also for Firefox, Chrome, and Safari. (Needs to be done on each account.)
- Twilio Authy. Use the desktop version which needs Rosetta (the App store version does not work well on OS X - too klunky.)
- GPG
  - https://www.vim.org/scripts/script.php?script_id=3645
    Move the gnugpg.vim script into ~.vim/plugin for all users.
    for u in admin mforbes wmforbes; do _home="/Users/${u}" sudo mkdir -p "${_home}/.vim/plugins" sudo cp ~admin/Downloads/vim-gnupg-2.7.1/plugin/gnupg.vim "${_home}/.vim/plugins/" find "${_home}/.vim" -exec sudo chown ${u} {} + done
- Global Protect VPN
- Zoom
- Microsoft Teams
- App Store:
  - Intego VirusBarrier (done earlier)
  - GoodNotes 5
  - Notability
  - KeyNote
  - Pages
  - GoPro Player.
  - OneNote.

Google Drive for Desktop
Teams
Cmake
Skim
SuperDuper!
Skype
Wacom One Drivers
- One of the components was not properly installed, and after chatting with Tech Support, I fixed it by opening Applications/Wacom Tablet/.Tablet and dragging WacomTouchDriver to System Preferences/Security & Privacy/Privacy/Input Monitoring/. (I needed Command+Shift+. to show the hidden folder).
LICEcap: For making screen captures as animated gifs.
Mumble: Communication (we run a server on Swan).

Issues

Apple only allows applications from trusted developers to run. This causes a problem with MacPort installed programs after some virus scanners are run. To get around this, you can add /opt/local to your virus scanners list of Trusted files. Retroactively you can run something like this as admin:
```
find /opt/local/ -exec xattr -d com.apple.quarantine {} +
```
For example, port install python38 will give a problem after running Intego's VirusBarrier Scanner app.
See here and here for some details.
Some version of python require earlier versions of libssl leading to the warning:
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
and subsequent errors. (Especially during testing with [Nox][].)
We get the libraries like libssl from MacPorts, so we use this to provide the various versions of python rather than conda.

Catalina

I did a fresh install of Mojave then upgraded to Catalina. Here is what I did afterwards:

System Preferences:

Apple ID -> App Store: Disable Password for Free Downloads.
Dock -> "Automatically hide and show Dock"
Disable Siri.
Keyboard -> Keyboard -> Key Repeat = Fast, Delay = Short, Use F1, ... as function, Modifier Keys -> Caps Locks = Control.
keys.
- Text -> Disable smart quotes.
- Dictation On.
Trackpad -> Scroll & Zoom: Disable scroll-direction "Natural"

Accounts:

Make conda and admin accounts. I have the following in my ~/.ssh/config file:

# ~/.ssh/config

Host admin
  HostName localhost

Host conda
  HostName localhost

Host *admin
  User admin
  ForwardAgent yes

Host *conda
  User conda
  ForwardAgent yes

Host *
  UseKeychain yes
  AddKeysToAgent yes

This allows me to login with ssh admin.

XCode:

xcode-select --install

MacPorts:

Download MacPorts.

sudo find /opt/local -exec chown admin {} +
port selfupdate
port install ... # See above.

Applications:

AVG:
LastPass: Download from the webpage, not the App Store.

Firm Links:

Catalina and later do not permit users to make files in root like /data. However, one can create a "firm link". I make a separate Data volume then create the following file as admin: sudo vi /etc/synthetic.conf:

# /etc/synthetic.conf
data	/Volumes/Data

then reboot.

https://stackoverflow.com/questions/58396821/what-is-the-proper-way-to-create-a-root-sym-link-in-catalina

Conda:

I keep the /data tree in ~admin, so first we create the conda folder:

su admin
mkdir ~admin/data/apps/conda
sudo chown conda ~admin/data/apps/conda

Now we install Conda etc.

su conda
ln -s ~admin/data/apps/conda conda
bash Miniconda3-latest-MacOSX-x86_64.sh -bu -p ~/conda
~conda/conda/bin/conda init bash zsh
. ~/.zshrc
conda update -y conda
conda install -n base -y anaconda-client
conda env update mforbes/base
conda update -n base --all -y
conda deactivate

conda env create mforbes/hg
conda activate hg
conda update -n hg --all

conda env create mforbes/jupyter
conda update -n jupyter --all

conda env create mforbes/work
conda update -n work --all

for py in 3.6 3.7 3.8 3.9 3.10; do
  conda create -y -n py${py} python=${py}
  conda update -n py${py} --all
done

conda clean --all -y

I downloaded Miniconda and installed it in the ~admin/data/conda.

Table of Contents

MacPorts

Alternatives

Discussions

Useful Commands

Cleaning

Dependencies

Custom Portfiles

References

Shell (bash)

Startup Scripts

XCode

Conda

Networking

VPN

Wireshark

SSH

SSHFS

SSH Tunnels (SOCKS/VPN)

BBCP

LaTeX

Safari

Extensions

Emacs

Configuration

BBEdit and .plist Files

iPhone/iPad

Backups

Useful Applications

GPG

Preference Panes

Vagrant

Docker

CoCalc

AWS-CLI

Disk Space:

Environment Modules (Lmod)

Graphics

MacSVG

GIMP

CMake

Julia

References

Sound: Black Hole

LICEcap

Sim Datlonism

Microsoft Office

Synergy

AVG

NPM

Monterey

Issues

Catalina