When have you experienced some difficult to install software? How did you solve it?
A question came up whether Singularity images can be made executable and my answer there after a quick test was wrong and one of the watchers has sent a clarification which is amazing:
Got puzzled by that executable singularity container remark someone did.
Tried it out here and I'm amazed that it actually works:
# cat Singularity
BootStrap: docker
From: alpine:edge
%runscript
/bin/echo "$@"
and after:
# singularity build singularity.img Singularity
...
# chmod +x singularity.img
# ./singularity.img Hello world
Hello world
Then something else I found while googling:
# cat Singularity
BootStrap: docker
From: alpine:edge
%post
apk --no-cache add vim python3
%runscript
/usr/bin/$SINGULARITY_NAME "$@"
# l
lrwxrwxrwx 1 root root 15 Nov 19 22:05 python3 -> singularity.img
-rw-r--r-- 1 root root 120 Nov 19 22:05 Singularity
-rwxr-xr-x 1 root root 22192128 Nov 19 22:05 singularity.img
lrwxrwxrwx 1 root root 15 Nov 19 22:02 vim -> singularity.img
# ./python3 --version
Python 3.8.6
# ./vim --version | head -n1
VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Nov 19 2020 05:07:05)
So $SINGULARITY_NAME becomes the name of the symlink (or the container).
This is blew my mind, basically drop-in binary replacements.
Is it so difficult to write good installation instructions?
Ten simple rules for writing Dockerfiles for reproducible data science: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008316
Source code is to a compiled code, as container recipe file is to the container image
Not really a question, but I like .dmg files on MacOSX. I unpack them and drag the app into any folder. Done. Or they come just with a little installation program, done.
Newbie question, but: which container technology should I use and why? Is there a handy comparison chart somewhere?
To what extent is Docker containers / Kubernetes used in university HPC environments?
What is the advantage of these containers? Is this software not available via conda, for example? Create a virtual env and install the modules.
Is running graphical user interfaces a good use of Docker? In general it seems like there's quite a lot of trickery to make it work reliably? More useful for running services, no?
Containers often bundle many things without consideration for licensing. Are there any recommendations to handle the explosion of different licenses you have to consider once you are bundling a large fraction of an operating system in a container? I guess similar question for other "virtualization solutions" and images (VMWare, VirtualBox)
Anne, thank you for the encouragement! Thanks for telling us your first baby steps. It looks complicated. Does one have to know a lot of Linux?
On the topic of layers, what are the current recommendations to handle them? I recently read about a new solution to avoid pages of instructions terminating with \ or && but I don't understand this.
Any experiences with using Ansible for the recipe part instead of Dockerfiles? From what I understand it can produce Docker images, while also having the ability to deploy via SSH and other options.
In the example on screen, won't line 8 apt-get update change very often?
From what I understand, the recommendation is to not have apt update by itself because it will be cached? If we want it to update on each build it should be RUN apt update && apt install ... && apt clean to force the update.
Any advice for automatic cleanup after unsuccessful Docker builds or long unused images? Docker images fill up the hard drive really fast.
docker image prune and docker container(s?) prune--rm option that was mentionedBut the docker file does not strictly define the result. Those package repos added presumably change over time.
Do you not mess up your system with such misbehaving Docker installations? AFAI understand, it would install all over the place things into the system, or not just in a controlled virt env
I arrived late so maybe you already talked about this. But many sites disallow pure docker (but supports singularity or shifter). But it's typically not that much of added hassle since singularity can import directly from docker(hub)
Often the hurdle I see in using Docker container is the fact that the size of the images scales quite poorly. For example, a simple opencv based image resizing application (4 lines of code) will end up in a ~1.2 GB of image size. The image size grows bigger if you wish to have common Linux distributions as base image and then some libraries on top of it. So, if someone is working on developing and distributing a software library, Docker may not be a good choice for distributing it as it may not make for a good dev environment for potential contributors. However, it can be a good option to consider if one is developing a script/black box kind of software that the end user will only execute. Would like to hear your thoughts on this.
Are there any open-source registries as alternative to docker-hub? Like if I have sensitive data in images and don't want to host them in a public registry.
It's really easy to run your own registry as a Docker container, to be independent on DockerHub etc. :)
Anyone paying for hosting your large files can stop doing it (for free)
From a reproducibility point of view, since the underlying operating system is always changing, how to you handle this? How do you version such image?
FROM) a specific image tag, so it won't change too much. Of course, what if repositories or package versions change?
alpine vs ubuntu vs centos as base images. alpine is popular for being small but uses a different C runtime (musl libc). Any experience with incompatibilities while changing or using different bases?
You can use traditional software build systems (spack as an example) and put the result in a docker image
You mentioned miniconda within Docker containers. Any recommendations for using/activating environments in this setting? Can be a little cumbersome since conda activate requires a shell...
conda init, but that makes permanent changes and require login shell. I recommend to people source $conda_path/bin/activate, even though that seems less recommended.
conda in the environment you are sourcing. Then I think you don't have to go through the base environment first (can someone correct?)conda config --set auto_activate_base false : this command disables automatic conda 'base' environment activation upon launching a terminalconda init only affects interactive sessions like docker run -it ubuntu bash, but if there's an entrypoint it will not source .bashrc and hence conda things won't be available.conda run, that will solve a lot of problems!). It is indeed a difficult issue, I wish conda wasn't so focused on interactive work. I need to investigate more.can you chmod +x singularity.img && ./singularity.img ?
Where do you get help when you are stuck with Docker or Singularity? Is there something like stackoverflow for them?
Drawbacks/problems with containers
Decision for next week: debugging. Please send us stuff that doesn't work!