Launch collaborative Jupyter notebooks on the ML Cloud

If two people want to collaborate using a single Jupyter notebook - for example, if a PhD student and their advisor want to work on the same document - this quick tutorial describes how to use Jupyterlab to collaborate on a workbook running on a compute node.

First, the host's setup actions are described, then the collaborator's setup actions. Finally, the remote connection is described for both users. This is set up to describe Galvani.

Both host and collaborator need to have ML Cloud access.

Host Tutorial

Host prerequisites:

For this to work, you first need jupyterlab and a specific version of jupyter-server installed on your server Python environment. This tutorial assumes you are using conda and already have a conda environment.

  • If you do not have a conda environment, run the following commands starting on the login node:
    • srun --pty bash # will open a command prompt on a compute node.
    • conda create -p $WORK/conda_envs/jupyter-collab # creates a new conda environment
    • conda activate $WORK/conda_envs/jupyter-collab # activate the conda environment
    • conda install -c conda_forge jupyter # Install Jupyter generally
    • python3 -m pip install "jupyter_server>=2.0.0" # install the required version of the jupyter-server.

You will likely need to install other packages as well for your specific application.

  • If you do have a conda environment, then, starting on the login node,
    • srun --pty bash # will open a command prompt on a compute node.
    • Activate the conda environment and run python3 -m pip install "jupyter_server>=2.0.0" which will install the necessary version of the Jupyterlab server that supports collaboration.

Host actions:

  • When logged in on a login node, run srun --pty bash

  • Now, run hostname. It will return something like galvani-cn105.sdn which you should note down.

  • Now, run jupyter lab --collaborative. The --collaborative string enables collaborating on the same document. You will see the standard startup sequence ending with some lines like

Or copy and paste one of these URLs:

http://localhost:8888/lab?token=76ebcaaf860c9426a1a34d373ad11ecf165371074dc69030

http://127.0.0.1:8888/lab?token=76ebcaaf860c9426a1a34d373ad11ecf165371074dc69030

  • The host now needs to note down three things:

    • The compute node currently being used for computation (the result of the hostname command, but only the string before the .)
    • The compute node port, which is the number after "localhost"
    • The Jupyter server access URL, which is the entire line including localhost

  • In the example above,

    • The compute node is galvani-cn105
    • The compute node port is 8888
    • The Jupyter service access URL is http://localhost:8888/lab?token=76ebcaaf860c9426a1a34d373ad11ecf165371074dc69030

Keep this terminal window open as long as you're working on the Jupyter notebook.

Give these three things to your collaborator. They can then start on the Collaborator Tutorial below.

Now, open a new terminal window and proceed to the Remote Jupyter Connection tutorial below.

Collaborator Tutorial

Once the collaborator has been given the compute node, compute node port, and Jupyter service access URL, they need to do the following:

  • Log into the same cluster as the host.

  • On the login node, run srun --nodelist=$NODE --pty bash where $NODE is the compute node given above.

Keep this terminal window open as long as you're working on the Jupyter notebook.

Remote Jupyter Connection

For both the host and the collaborator, this section describes the protocol for connecting to a Jupyter notebook running on a compute node.

  • Obtain this session's compute node, compute node port, and Jupyter service access URL. They will change each time you go through the above steps
  • Open a new terminal window.
  • On your computer, run the following command, and note down the number as the bounce port, which we will call the $B_PORT.
    • echo $(( 8800 + RANDOM%100 ))
  • On your computer, run the following command, replacing $YOURLOGIN, $NODE, $B_PORT, and $COMPUTE_PORT with your login name, the compute node, the bounce port and the compute node port, respectively:
    • ssh -AtL $B_PORT:localhost:$B_PORT $YOURLOGIN@LOGIN_NODE_IP "ssh -AtL $B_PORT:localhost:$COMPUTE_PORT $YOURLOGIN@$NODE bash"

(Yes, we know that's a long command with many things to replace.) This long command will enable you to open a browser window to the running Jupyterlab server.

With that terminal window open (plus the one from the Host/Collaborator Tutorial steps above; you should have two terminal windows open now), you should be able to open the Jupyter service access URL in your browser and be able to collaborate! Have fun!

References