RSH 19: Working in shells and terminals

Video

Collaborative notes taken during the session

Questions/comments

• Intro to shell (and HPC): https://youtube.com/playlist?list=PLZLVmS9rf3nN_tMPgqoUQac9bTjZw8JYc

• Will you also show some tricks for macOS users?

  • Anything particular that you would like to know more about or see? (I think we use Linux and Windows ...)
    • No, because I came to learn :-)
      • Hopefully we will show useful things but maybe a watcher who uses macOS can also comment?
    • Mac user here: what works for Linux is basically also going to work on Mac, or at least I still have not found something that wouldn't work on both :) [however x11 is not preinstalled, so you need Xquartz]
      • Yes, so far I have not really discovered anything that only worked on Mac. Ah, okay, I get X11 via Xcode.
        • I personally don't want to install xcode because it's an elephant :)
          • I realised! I will have a look for Xquartz. Thank you!

• does X11 forwarding require any setup/installation on your side?

  • seems like no, nothing special required
  • Make sure x11-Forwarding box is ticked (it is by default) in Advanced SSH settings

• Can you switch on / off $CDPATH? Or do I have to change it each time.

  • Richard will answer later (Radovan: I am not using this yet)

• Many shells exist

  • Bash, Fish, zsh, csh, tcsh, ...
  • xonsh https://xon.sh/
  • elvish https://elv.sh/
    • hmm! new to me

• Could you maybe comment on what are the advantages/disadvantages of the various shells? So far, I have only used bash. macOS offers now to switch to zsh.

  • great question, we will come back to this
    • Thank you.

• When I type in bash echo $CDPATH I get only an empty line. Did Richard setup this $CDPATH?

  • yes I think he set it in .bashrc (?) where one can set variables and aliases
  • You can try it out by defining CDPATH to one of your folder.

• Many shells: is there a fast way to switch between them (e.g. I am on bash and then want to go to fish)? I guess by just typing the shell name?

  • yes. I use fish but sometimes need bash and then I type "bash" and enter and I am in bash. And vice versa. Alternative: if this is a script, you indicate with a shebang on top which shell it should use. Example: #!/usr/bin/env bash

• Let's also talk of sharing our shell config across systems

  • yes!
  • RD: I have one master .bashrc file one of my computers. All edits go there, it gets copied to other computers. So master config is always clear
    • Is this really so important? I never bothered. Maybe I am not advanced or not using the full shell potential.
      • I am switching between 3 computers and really appreciate that when I make an improvement on computer 1, I don't have to reinvent it on computers 2 and 3. I don't use any fancy solution but share my dotfiles via GitHub: https://github.com/bast/config
        • Okay, the master is in a private repo on github?
          • It's public. But I keep the more sensitive settings (ssh config) in another (private) repo.

• Do you use non-bash shells also on HPC clusters? I have seen that other shells than bash can give issues...

  • on clusters I recommend to use bash. I have tried zsh but some tools (module, lmod) assume bash and can break with "non-standard" shells.
  • I see lmod has a fish profile. But if your admins make it usable is another question.
    • hmm! I need to try this again then. I gave up after zsh didn't work but never tried fish - thanks!

Anne

• Terminal for Windows users: MobaXterm
  • https://mobaxterm.mobatek.net/

Richard

• CDPATH

  • just like PATH is used for running programs, CDPATH is used for cd
  • I set CDPATH=~/git/, so that I can cd project-name for any project in git directly

• ! aliases

  • We often need to re-run a command with the same argument. For example, you often mkdir dir-name and then cd dir-name. We want to type less?
  • !$ means last argument of last command line. ! commands expand to recent history. $ means "last argument"
  • !! means entire previous line
  • There is much more I won't go into
  • (don't say) Line designator
    • !-n refers to n:th previous line
    • !string is recent line beginning with string, or !?string? for previous line containing string
  • (don't say) Word designator: : + symbol, sometimes : optional
    • 0: zeroth argument (program name)
    • ^: first argument
    • $: last argument
    • x, x-y, x-: range of word numbers.
  • (don't say) Modifiers
    • h: remove trailing filename
    • t: remove head, leave only last pathname
  • Important options
    • shopt -s histreedit
    • shopt -s histverify - don't run, verify before running
  • bash manual page section: HISTORY EXPANSION

• per-directory bash history

  • I have a per-directory shell history, which gets saved in _bashhist in each directory (once the shell terminates).
  • You can see the commands in hackmd, you can also find ideas in stack overflow.
  • I have aliases histsave, histread, histoff as well
  • Ignore this in git: ~/.gitignore, git config core.excludesfile=~/.gitignore.
  • Warning: I wrote this myself, there are possibly bugs I haven't resolved yet.

histsave() { history -a ; }
histread() { history -n ; history -n ; } # Needs it twice somehow...
histoff()  { HISTFILE=/dev/null ; }
cd_hook() {
    # This function handles a per-directory history file
    history -a
    command cd "$@" ;
    if [ -z "$PS1" -o -z "$HISTFILE" -o "$HISTFILE" = "/dev/null" ] ; then
        # Don't do histfile magic if not in interactive mode, HISTFILE
        # is unset or set to /dev/null
        return
    fi
    #if echo "$PWD" | grep "^$HOME" >> /dev/null && test -w . -a -O . -a -G .;
    #if test -w . -a -O . -a -G .;
    if test -w . ;
    then
        # Within home directory
        HISTFILE=./_bashhist
    else
        HISTFILE=~/.bash_history
    fi
}

alias cd=cd_hook
cd_hook $PWD
if [ "$PS1" -a x$_loaded_hist_once = x ] ; then
    history -r
    _loaded_hist_once=True
fi

• Git aliases as bash aliases
  • We saw this when I tailed my _bashhist file, I used a lot of different aliases
  • This way I can run ci instead of git ci that is aliased to git commit -v:

if [ -n "$PS1" ] ; then
    for gitalias in $(git config --name-only --get-regex ^alias.) ; do
        #echo $gitalias
        gitalias="${gitalias#*.}"
        if ! type "$gitalias" &> /dev/null ; then
            eval alias "${gitalias}='git ${gitalias}'"
        fi
    done ; unset gitalias
fi

Roberto

I'll use the terminal in this binder instance

• why fish

• direnv

• nix-shell

• The prompt of my fish shell is configured using starship

• Where does nix store things per-user? I see /nix, so this is like a top-level OS add-on?

  • all dependencies are hashed and the paths are the hashes. This means that it can be stored top-level and if two people install the same dependency, it does not have to be twice on the hard-drive. So it's great for clusters (I wish clusters would use this ...)

• Can you have multiple environments?

  • yes (but I don't remember how but Roberto knows and will fill in ...)
    • Yes, you can have as many shells as you need/want. I usually have one shell.nix inside every project folder and this gives me a different environment for every project.

• What are the differences between Nix and Guix?

  • I think it's a similar idea implemented in two versions. Both are really nice. (but I have only tried the former)
    • The language is different and there might be some differences in what packages are accepted in the "canonical" repo for licensing reasons. I have never used Guix myself.

• can direnv load virtualenvs/conda environments?

  • yes, any valid bash can be executed by .direnv
  • would you do it by source venv/bin/activate in the .direnv? can you tell it to unload when leaving the directory?
  • OK, I see it is done with these functions that are smart about it

Radovan

• what I like about fish

  • aliases are functions
  • autosuggestions and colors
  • history is per-directory and context-aware
  • simpler syntax (at the cost of not being POSIX-compilant)
  • on a supercomputer/cluster I still advise to use bash (many tools assume bash and break otherwise)

• jumping between folders (cd -, pushd, popd)

• for loops: renaming many files at once

• let's use some command that I don't use every day

• tldr

  • There's also a Rust implementation of it! It's called tealdeer

• reusing past commands (reverse-i-search with CTRL+R)

  • I use fzf with Bash

• editing past commands (how to move to start or end of a line: CTRL+A or CTRL+E)

• That for loop was MATLAB syntax! :D <3

  • oh! TIL!

• Never tried but I heard good things about GNU Stow to keep track of dot files

  • thanks for sharing this!