how to run Jupyter script on Slurm HPC

Question

[Bioinfonext]

Hello,

I need to run a Jupyter script, could you please suggest how can run it on HPC server? Do I need to load any module? Many thanks

Answer 1

[i.sudbery]

This very much depends on how your HPC is set up and what format the script is in.

Mostly likely you will need a python installation that has the jupyter package installed in it.

If the script is an ipynb file, once you have jupyter installed, running the script is easy. The command to run the script is is, strangely, nbconvert.

jupyter nbconvert --to notebook --execute mynotebook.ipynb

You would wrap that in what ever submission script you would normally use with your HPC system. For example:

[USER@login1]$ cat submission.sh
#!/bin/bash
#SBATCH --nodes=1
srun jupyter nbconvert --to notebook --execute mynotebook.ipynb  

[USER@login1]$ sbatch submission.sh

(disclaimer: I am not a regular SLURM user)

So the hard bit is getting jupyter installed on your HPC system. Almost all HPCs have systems for installing python packages without the need for root permissions, but the route varies from system to system (this is independent of the scheduller - SLURM in this case).

Our system uses conda to manage package installation. You would first need to create an environment with jupyter installed so you would want to do something like:

module load conda
conda create -n jupyter-env python=3 jupyter

You would also need to install any packages that the notebook required as well. Then you would need to load the env in batch submission script, so your script might look like:

[USER@login1]$ cat submission.sh
#!/bin/bash
#SBATCH --nodes=1
module load conda
source activate jupyter-env
srun jupyter nbconvert --to notebook --execute mynotebook.ipynb

Other systems might rely on pip and/or virtualenv. In pip you might try

[USER@login1]$ pip install jupyter --user

[Bioinfonext]

Hi @i.sudbery

Thanks for your detail response. I was able to install Jupyter on the server but while running script it shows the error? I have attached the script, could it be possible to convert it as python script? Thank you so much for your time and help.

#Run it in jupyter lab or notebook


import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

%matplotlib inline 


qza = 'alpha-rarefaction_20000.qzv' # path to rarefaction file from qiime2
a = !unzip $qza
digest = a[1].split('/')[0].replace('  inflating: ','')
inf = digest + '/data/observed_otus.csv'
otus = pd.read_csv(inf,index_col=0, sep=',')
!rm -r $digest 

cols = [col for col in otus.columns if 'iter' not in col]
mean,data = otus[cols],pd.DataFrame(columns=cols)
depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
otus = otus.drop(cols,axis=1)
otus.columns = depths
for depth in depths:
    mean['ASV'] = otus[depth].mean(axis=1)
    mean['depth']= depth.split('-')[-1]
    data = pd.concat([data,mean])

# here provide colors for each item that will be plotted
pal={'Leaf':'brown','Root':'green','Soil':'blue'} 
fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
sns.set(style='ticks',rc={"lines.linewidth":0.7,"axes.linewidth":0.5})
# use your column name to plot here instead of 'BodySite'

sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Compartment',sort=False,err_style='bars',\
             dashes=True,style='BodySite',ci=67)
ax.set_xlabel('Sequencing depth',fontsize=8)
ax.set_ylabel('Observed ASVs',fontsize=8)
ax.tick_params(axis='x', labelrotation=90)
ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
plt.savefig('rarefaction_ASVs.png', bbox_inches='tight')

Many thanks

[i.sudbery]

This should work fine as a script with the excpetion of the lines

%matplotlib inline

which can be removed

and

!rm -r $digest

This shell command should be replaced with the equivalent python command. Try a google search for "python delete directory".

[Bioinfonext]

Hi @i.sudbery,

thanks, but this line also shows the error and not sure what command to use to unzip the file in python?

a = !unzip $qza

Many thanks,

[i.sudbery]

There is not python equivalent for unzipping a file as far as I'm aware (I might be wrong). You will need to run a shell command from python.

[Bioinfonext]

Hi,

Now jupyter installed on the server and I am using below jupyter code to plot the rarefaction plot but I am still getting some error. could you please suggest how I can get rid of it?

#!/bin/bash
#SBATCH --job-name=plot    # Job name
#SBATCH --nodes=3                  # Number of nodes           
#SBATCH --time=2000:00:00              # Time limit hrs:min:sec
#SBATCH --partition=k2-lowpri
#SBATCH --mem=150G


module load apps/jupyter/4.6.3
module load libs/pandas/0.24.2/gcc-4.8.5+python-2.7.8+setuptools-39.2.0+numpy-1.16.2
module load apps/python3/3.7.4/gcc-4.8.5
module load apps/qiime2/2019.4.0/bin
module load libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3



#Run it in jupyter lab or notebook


import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd

%matplotlib inline 

qza = 'alpha-rarefaction-20000.qzv' # path to rarefaction file from qiime2
a = !unzip $qza
digest = a[1].split('/')[0].replace('  inflating: ','')
inf = digest + '/data/observed_otus.csv'
otus = pd.read_csv(inf,index_col=0, sep=',')
!rm -r $digest 

cols = [col for col in otus.columns if 'iter' not in col]
mean,data = otus[cols],pd.DataFrame(columns=cols)
depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
otus = otus.drop(cols,axis=1)
otus.columns = depths
for depth in depths:
    mean['ASV'] = otus[depth].mean(axis=1)
    mean['depth']= depth.split('-')[-1]
    data = pd.concat([data,mean])
# here provide colors for each item that will be plotted
pal={'Irrigated':'brown','Rainfed':'red'} 
fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
sns.set(style='ticks',rc={"lines.linewidth":0.7,"axes.linewidth":0.5})
# use your column name to plot here instead of 'BodySite'
sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Season',sort=False,err_style='bars',\
             dashes=True,style='Season',ci=67)
ax.set_xlabel('Sequencing depth',fontsize=8)
ax.set_ylabel('Observed ASVs',fontsize=8)
ax.tick_params(axis='x', labelrotation=90)
ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
plt.savefig('Infected_observed_ASVs.png', bbox_inches='tight')

Error;
import: unable to open X server `' @ error/import.c/ImportImageCommand/344.
import: unable to open X server `' @ error/import.c/ImportImageCommand/344.
import: unable to open X server `' @ error/import.c/ImportImageCommand/344.
/var/spool/slurmd.spool/job1452532/slurm_script: line 21: fg: no job control
/var/spool/slurmd.spool/job1452532/slurm_script: line 23: qza: command not found
/var/spool/slurmd.spool/job1452532/slurm_script: line 24: a: command not found
/var/spool/slurmd.spool/job1452532/slurm_script: line 25: syntax error near unexpected token `('
/var/spool/slurmd.spool/job1452532/slurm_script: line 25: `digest = a[1].split('/')[0].replace('  inflating: ','')'

metadata file ;
#SampleID       Villages        Region  Treatment       Batch   Season
PN0086D.1.S1    Kholabari       Mymensingh      T1      batch1  Irrigated
PN0086D.2.S2    Bogajan Mymensingh      T1      batch1  Irrigated
PN0086D.3.S3    Bonkua  Mymensingh      T1      batch1  Irrigated

many thanks

[Bioinfonext]

Now jupyter installed on server but still getting en error/

$ jupyter nbconvert --to notebook --execute mynotebook.ipynb
[NbConvertApp] Converting notebook mynotebook.ipynb to notebook
Traceback (most recent call last):
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbformat/reader.py", line 14, in parse_json
    nb_dict = json.loads(s, **kwargs)
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/python3/3.7.4/gcc-4.8.5/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/python3/3.7.4/gcc-4.8.5/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/python3/3.7.4/gcc-4.8.5/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/apps/alces/jupyter/4.6.3/bin/jupyter-nbconvert", line 10, in <module>
    sys.exit(main())
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/jupyter_core/application.py", line 270, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/traitlets/config/application.py", line 664, in launch_instance
    app.start()
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 340, in start
    self.convert_notebooks()
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 510, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 481, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 410, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 179, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 197, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbformat/__init__.py", line 143, in read
    return reads(buf, as_version, **kwargs)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbformat/__init__.py", line 73, in reads
    nb = reader.reads(s, **kwargs)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbformat/reader.py", line 58, in reads
    nb_dict = parse_json(s, **kwargs)
  File "/opt/apps/alces/jupyter/4.6.3/lib/python3.7/site-packages/nbformat/reader.py", line 17, in parse_json
    raise NotJSONError(("Notebook does not appear to be JSON: %r" % s)[:77] + "...")
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: '#Run it in jupyter lab or notebook\n\n\...

Answer 2

[edu.blancas]

I'm working on an open-source framework that allows you to execute notebooks in SLURM.

Here's the SLURM tutorial.

If you need help, feel free to ping us in our community.