sim01 - User’s guide

sim01 is an Ubuntu Server 22.04 machine hosted at CIBIT. It was designed to aid processing of large datasets or to perform intensive computational tasks. If you have any question to which you do not find an answer here, please refer to the #sim01 channel at Discord.

What works and what doesn’t in sim01

For making a decision on using sim01, we recommend that you check the Installed software section. If you see your software on the list, then there is a chance that you will profit from using sim01 for your analysis.

For example, intensive tasks in FSL or MATLAB may benefit from the use of sim01, while Brainvoyager (or other highly optimized and interface-based tools) may not.

Get in touch on Discord channel #sim01 to create an account for yourself and get started.

How to access sim01

The primary way to access sim01 is through SSH. You need to be connected to CIBIT’s network (phisically or using a VPN).

Open a terminal (or PowerShell in Windows) and type

ssh -X <username>@192.168.0.68

replacing <username> with your username on sim01.

Then, type the password and press ENTER (no feedback is given while writing the password). The option -X allows for graphical interfaces to show up if needed. Great, you are now connected to sim01!

Closing the connection

To exit sim01, just type logout and press Enter, and you will return to the local machine terminal.

File structure

There are two storage types in sim01:

  1. /DATAPOOL - Redundant storage based on hard-disk drives. The user home folders are located here (/DATAPOOL/home/<username>), as well as the folders for the several ongoing projects. This should be your primary storage option.
  2. /SCRATCH - Storage based on NVMe drives. This volume provides fast I/O speeds and is intended for processing data - if your script heavily relies on I/O (reads and writes many bytes per second) please use /SCRATCH/users/<your_username>.

Datasets should be stored in each project’s pool in /DATAPOOL/<projectName, which will be created for you after account setup. Please use your home folder solemly for scripts, logs, and other files that are not part of the dataset.

Software

Software should be installed in /SCRATCH/software/. Changing files in this folder requires being a member of the softwareusers usergroup.

A list of all software is provided below. Users should update this list as they install new software. Mind that some packages require the user to edit their own environment variables (defined in the file ~/.profile). To edit this file, type the following command:

nano ~/.profile

Here, you can edit or add new environment variables. Feel free to use another text editor, here we chose nano for simplicity. Mind that logout-login is needed for changes to take effect.

MATLAB toolboxes should be placed in /SCRATCH/software/toolboxes. Please list them here.

Scroll right on the table to see a brief guide on how to configure each software for your user.

Name Version Path Install/update date Description Guide
docker 20.10.12 apt install 16/06/2020   Docker images/containers will be installed in /DATAPOOL/docker-data
fmriPrep 20.2.7 docker image 2023 fmriprep LTS version in docker environment –
fmriPrep 23.0.2 docker image 2023 fmriprep lastest version in docker environment –
mriqc 0.16.1 docker image 16/06/2021 MR Quality assessment toolbox –
bids-validator 1.6.2 /usr/bin/bids-validator 2022 Validator for BIDS directories –
MATLAB 2019b /SCRATCH/software/MATLAB 18/11/2019   –
FreeSurfer stable v6.0.0 /SCRATCH/software/freesurfer 19/11/2019   Add to .profile # FreeSurfer Setup export FREESURFER_HOME=/SCRATCH/software/freesurfer source $FREESURFER_HOME/SetUpFreeSurfer.sh
FreeSurfer stable v7.3.2 /SCRATCH/software/freesurfer7 27/02/2023   Add to .profile # FreeSurfer Setup export FREESURFER_HOME=/SCRATCH/software/freesurfer7 source $FREESURFER_HOME/SetUpFreeSurfer.sh
Connectome Workbench 1.4.1 /SCRATCH/software/workbench 19/11/2019 HCP pipelines requirement  
tcsh 6.20.00 apt install 20/02/2020 Requirement of freesurfer –
FSL 6.0.4 /SCRATCH/software/fsl 2022   Add to .profile # FSL Setup FSLDIR=/SCRATCH/software/fsl PATH=${FSLDIR}/bin:${PATH} export FSLDIR PATH . ${FSLDIR}/etc/fslconf/fsl.sh
FSL 5.0.11 /SCRATCH/software/fsl-5.0.11 20/11/2019   FSLDIR=/SCRATCH/software/fsl-5.0.11
ANTs 2.3.4 Dorymyrmex /SCRATCH/software/ANTs-v2.3.4 28/09/2020 Advanced Normalization Tools - ANTs computes high-dimensional mappings to capture the statistics of brain structure and function. Add to .profile: # ANTS Setup export ANTSPATH=/SCRATCH/software/ANTs-v2.3.4/bin export PATH=${ANTSPATH}:$PATH export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=18
FSL FIX 1.06.15 /SCRATCH/software/fix 14/12/2020 https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FIX  
gradunwarp 1.2.0 /SCRATCH/software/gradunwarp 14/12/2020 https://github.com/Washington-University/gradunwarp/releases Add to .profile: # gradunwarp Setup: export PATH=”/SCRATCH/software/gradunwarp/bin:$PATH”
msm 3.0FSL /SCRATCH/software/msm 14/12/2020 https://github.com/ecr05/MSM_HOCR  
Convert3D - /SCRATCH/software/Convert3D 2021 Convert3D is a tool for converting between different file formats. http://www.itksnap.org/c3d  
AFNI 21.0.21 ‘Titus’ /SCRATCH/software/AFNI 2021    
Automatic Registration Toolbox 2.0 /SCRATCH/software/AutoRegToolbox 22/04/2020 Automatic Registration toolbox (https://www.nitrc.org/projects/art) Follow documentation to add folder to user path
Atom 1.41.0 apt install 18/11/2019   –
inxi 2.3.56-00 apt install 17/12/2019 System info –
s-tui 0.8.3 pip 20/11/2019 Frequency and temperature monitor run with sudo s-tui
cmake 3.10.2 apt install 28/09/2020 Compiler –
MRtrix3 3.0.1 /SCRATCH/software/mrtrix3 28/09/2020 Advanced tools for the analysis of diffusion MRI data. Add to .profile: # MRtrix3 Setup: export PATH=”/SCRATCH/software/mrtrix3/bin:$PATH”
MATLAB Runtime v97 /SCRATCH/software/MATLAB_Runtime 14/12/2020    

MATLAB Toolboxes

Name Version Path Install/update date Description Guide
SPM 12 v7771 /SCRATCH/software/toolboxes/spm12 19/11/2019   Add to matlab path
CONN Toolbox 19c /SCRATCH/software/toolboxes/conn19c 22/04/2020 Connectivity toolbox Add to path in matlab to install (without subfolders)
CONN Toolbox 20b /SCRATCH/software/toolboxes/conn20b 13/04/2021 Connectivity toolbox Add to path in matlab to install (without subfolders)
CONN Toolbox 21a /SCRATCH/software/toolboxes/conn21a 20/01/2022 Connectivity toolbox Add to path in matlab to install (without subfolders)
Computational Anatomy Toolbox for SPM (CAT) Version 1739 CAT12.7 2020-11-27 /SCRATCH/software/toolboxes/spm12/toolbox/cat12 21/12/2020 SPM toolbox (http://www.neuro.uni-jena.de/cat/index.html) See instructions for SPM.
TAPAS PhysIO R2021a-v8.0.1 /SCRATCH/software/toolboxes/spm12/toolbox/PhysIO 06/04/2022 https://github.com/translationalneuromodeling/tapas/tree/master/PhysIO Automatically shows on SPM batch.
MVGC 1.0 /SCRATCH/software/toolboxes/MVGC1-1.0 - - -
EEGlab 2019 /SCRATCH/software/toolboxes/eeglab2019_1 - - -
JSONlab 2.0 Magnus Prime /SCRATCH/software/toolboxes/jsonlab - - Add to matlab path
Neuroelf v1.1 /SCRATCH/software/toolboxes/neuroelf-matlab - - Add to matlab path
NifTI Tools - /SCRATCH/software/toolboxes/nifti-tools - - -
ShadedErrorBar - /SCRATCH/software/toolboxes/shadedErrorBar - - Add to matlab path

Copy files from your local machine to sim01

Data needs to be placed on sim01 in order to be processed by it. Here, you will find instructions on how to copy files from your local machine to sim01 depending on your operating system.

From Ubuntu

The easiest way is to connect to sim01 using the File Explorer. Press +Other Locations, then in the Connect to Server panel type sftp://192.168.0.68/. After successful login with your credentials you will have access to the sim01 folders.

From macOS or Windows

FileZilla is a good option both on macOS and Windows. Configure the connection with the following:

  • Protocol: SFTP
  • Host: 192.168.0.68
  • Logon type: Normal
  • User: <username>
  • Password: <userpassword>

Thread management

Given the concurrent use of this machine by many users, thread managment is strongly recommended, if not mandatory. Here, we provide guidelines for some of the most common apps.

MATLAB

It is possible to control the maximum number of threads used by MATLAB. By default, MATLAB uses all available threads (twice the number of CPU cores). Following this link, just add this line to your code to restrict to N threads:

maxNumCompThreads(N);

ANTs

ANTs will use all available threads by default (extensively). We control the number of threads by setting the environment variable ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS to a reasonable number. Add the following line to your ~/.profile file to limit to 18 threads, for instance:

export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=18

OpenMP

To limit the number of cores that an OpenMP job can bogart you can set the environment variable OMP_NUM_THREADS to a suitable number (reference):

export OMP_NUM_THREADS=12

Basic/useful bash commands

  1. Monitor sim01 processes, CPU and RAM usage

     htop

  2. Check disk space

     df -h

  3. Check disk utilization

     iostat -x 1

  4. Detach SSH session using screen - This allows the code to continue running even if the SSH connections fails.

    Start screen task1 exporting a log text file named task1-logfile.txt:

     screen -L -Logfile task1-logfile.txt -S task1

    To detach from screen press Ctrl+A and then Ctrl+D.

    To re-attach to the screen task1 type

     screen -r task1

    To close the screen, re-attach and then press Ctrl-A and then kand then y.

    To list all screens type:

     screen -ls

Creating an ssh key

You can ease the login process by creating an ssh key. This method of authentication will remove the need for typing a password everytime you login in the server.

In the local terminal, let’s first create a new ssh key:

ssh-keygen -t rsa -b 2048

This will prompt for a key path+name (you can leave it blank for the default name, or change it to whatever you want, for instance /home/<username>/.ssh/id_sim01 ). You will also need to define a password for unlocking this ssh key.

Then, copy the key to sim01 by running

ssh-copy-id -i /home/<username>/.ssh/<keyname> <username>@192.168.0.68

Now, login into sim01 using

ssh <username>@192.168.0.68

If the username is the same localy and in sim01, you can ommit it

ssh 192.168.0.68

You will be prompted for the ssh key password once, and then you can save it. Done!

Configure VNC (for graphical interfaces)

SSH is the only requirement for accessing the server, and it also provides graphical interface via Xserver. If you explicitely require a graphical environment (you use an application that is strongly interface-dependent) then continue reading for a step-by-step configuration.

Step-by-step

  1. Login through ssh into sim01.
  2. Run the command vncserver. This will initiate the server for the current user.
    1. Type in a password specific for the VNC connection.
    2. When prompted for view-only password, answer no.
  3. Kill the instance that we just opened with vncserver -kill :1
  4. Create a backup of the startup file mv ~/.vnc/xstartup ~/.vnc/xstartup.orig
  5. Start a new file nano ~/.vnc/xstartup and add the following lines: 
     #!/bin/bash
 xrdb $HOME/.Xresources
 startxfce4 &
  6. Make this file executable chmod +x ~/.vnc/xstartup
  7. Restart the server vncserver -localhost
  8. Kill the instance that we just opened with vncserver -kill :1
  9. Now we need to create a service to keep this vnc server running, even after a restart. For this, type sudo nano /etc/systemd/system/vncserver<INITIALS>@.service replacing <INITIALS> with your name initials (for instance JD).

  10. Copy this text to the file, replacing the fields <username> when asked:

    [Unit]
Description=Start TightVNC server at startup
After=syslog.target network.target

[Service]
Type=forking
User=<username>
Group=<username>
WorkingDirectory=/DATAPOOL/home/<username>

PIDFile=/DATAPOOL/home/<username>/.vnc/%H:%i.pid
ExecStartPre=-/usr/bin/vncserver -kill :%i > /dev/null 2>&1
ExecStart=/usr/bin/vncserver :%i -depth 24 -geometry 1920x1080
ExecStop=/usr/bin/vncserver -kill :%i

[Install]
WantedBy=multi-user.target
  11. After selecting a port for you user (see Registered users and ports), let’s say Y, run the following sequence of commands to start the service: 
    sudo systemctl daemon-reload
sudo systemctl enable vncserver<INITIALS>@<Y>.service
sudo systemctl start vncserver<INITIALS>@<Y>
sudo systemctl status vncserver<INITIALS>@<Y>

Registered users and ports

username YY field vnc port
alexandresayal 4 5904
rutebettencourt 7 5907
carolinatravassos 9 5909
rmartins 15 5915

Windows

The recommendation is to use MobaXterm to open the VNC and SSH tunnel both at the same time.

Create a VNC connection with SSH on in the advanced features.

Linux

Remmina is a great client for handling both VNC and SSH. Use the following settings:

  • Basic
    • Server: 127.0.0.1:509X (where X comes from the table below)
    • User password:
    • Color depth: 24 bpp
    • Quality: Best
  • SSH Tunnel
    • Click “Enable SSH tunnel”
    • Select custom and insert 192.168.6.192
    • Username:
    • Password:

macOS

Type this command in the terminal to open the SSH tunnel:

ssh -L 59000:localhost:59XX -C -N -l <username> 192.168.*.*

where 59YY must be replaced by the user-specific window configured, according to the table below (Registered users and ports), and <username must be replaced by your username.

Then, open the app Screen Sharing (or Screens), type localhost:59000 to access the graphical interface.

Hardware specs

  • 2x Intel(R) Xeon(R) GOLD 5220 @ 2.20GHz (total of 36 cores, 72 threads)
  • 384GB RAM 2666 MHz
  • NVIDIA RTX 2080 Ti
  • Boot SSD 60GB
  • SCRATCH - 1TB NVMe in RAID0
  • DATAPOOL - 36TB in RAIDZ2

Administration

Access this file for instructions (requires GitHub authentication).

Copyright © 2024 CIBIT, University of Coimbra, Portugal.

Page last modified: Jan 5 2023 at 09:00 PM.

Edit this page on GitHub.