Skip to content

EmSART

Jump to bottom
Utz Ermel edited this page Dec 6, 2019 · 2 revisions

EmSART is a command line program that reconstructs tomograms from a recorded tilt-series, based on the alignment information saved in a marker-file.
All parameters that can be adjusted are saved in a config-file, which is handed to the program when executing it.
Using this parameters one can for example choose to reconstruct the tomogram with the SART algorithm or with weighted backprojection (WBP).

Table of Contents

Installation

System Requirements

  • nVidia CUDA 8.0 toolkit
  • Multiple GPUs with a total memory size of the volume to reconstruct (16 or 32 bits per voxel) + some memory for the projection images. (We use four GeForce Titan X with 12 GByte each).
  • Enough RAM on host: There is no hard limit, but the more memory is available, the bigger the reconstruction can be.
  • OpenMPI 1.10.2 installed for multi-GPU support.

Algorithm

The details of the implemented algorithm can be found in our JSB-paper from 2014.
Super-sampling SART with ordered subsets
One can choose whether the program reconstructs with the SART algorithm or with weighted back projection (WBP), by setting the respective parameters in the config-file.

WBP = false The reconstruction will be done using the SART algorithm.
WBP = true The reconstruction will be done using the WBP algorithm. In this case, one can choose the filter that is used.
WBPFilter Ramp/Contrast2/Contrast10/Contrast30
Have a look at the description of the config-file for all details.

Parameters

Input to the Program

General Requirements

File Format Information Output from
tilt-series .st stack with motion corrected images of the tilt-series ImageStackAlignator
marker-file .em file with alignment information of the tilt-series Clicker
CTF-file .em optionally: file with CTF information for the whole tilt-series CTFDetector
config-file .cfg file to set all parameters that are used by the program Clicker / by hand

Command Line Input

-n MPIs Set a number for MPIs. This describes how many MPI-processes are spawned by the program. The number has to be related to the number of CUDA devices specified in the config-file.
-u config-file. Give the path to the config-file. The config-file contains all further parameters that are necessary for the execution of the program.

Config-File Input

Detailed descriptions for all parameters in the config-file can be found on the respective page.

Output from the Program

Reconstructed tomogram in .em format.

Usage

The program can be directly executed from the command line to reconstruct a single tomogram, or it can be executed using a bash-script (.sh-file) to reconstruct multiple tomograms.

Command Line Execution for single Reconstruction - single-GPU

If one wants to use only one GPU, the program can be executed directly without using MPI: 

./EmSART -u config-file.cfg

Command Line Execution for single Reconstruction - multiple-GPUs

If one wants to use more than one GPU, the program needs to be executed with openMPI: 

mpiexec -n MPIs EmSART -u config-file.cfg
The GPU device IDs to be used should be setup correctly in the config-file.

Bash-Script Execution for multiple Reconstructions

In order to reconstruct multiple tomograms one has to generate a bash-script first. This can be done using a text editor or a script. An example script-file could look like this: 

#!/bin/sh

# Tomogram 1
mpiexec -n MPIs EmSART -u config-file1.cfg

# Tomogram 2
mpiexec -n MPIs EmSART -u config-file2.cfg

# Tomogram 3
mpiexec -n MPIs EmSART -u config-file3.cfg

After this file was generated (saved as a .sh file), one has to make it executable with the following command: 

chmod -x bash-script.sh

The file can then be used to run multiple reconstructions on a GPU-machine. (Remember to open a screen when reconstructing multiple tomograms!) 

./bash-script.sh

Multi-GPU execution on different hosts

If a high number of tomograms needs to be reconstructed and especially if they are reconstructed with low/medium binning, it might be helpful to use more than one GPU-machine.
In preparation for such an execution, one needs to set up passwordless login for ssh and create a host-file that lists the different GPU-machines and that specifies, how many MPI processes (slots) are spawned on the different machines. The host-file can be created using a text editor and should look like the following example: 

10.10.12.144 slots=4
10.10.12.146 slots=4

The program can than be executed using the following command on one of the machines intended to bu used: 

mpiexec -n MPIs --hostfile link_to_hostfile EmSART -u config-file.cfg

In such a setup, the number that is set for MPIs may not exceed the sum of the slots specified in the host-file.
Remark for multi-host execution: In order to use multiple GPUs, the volume is split in slices along the Z-axis. The forward projection is then the sum of all partial projections, where the partial images are all transferred to the main process with ID 0. Then the difference image for SART is only computed with process 0 and then spread to all other processes with openMPI. Especially in the case of CTF corrected reconstructions, where the image for back-projection is computed many times for all defocus steps and if the network connection is slow, it might be faster to only reconstruct on one single host.

Example

High Contrast Tomogram - Judging the Quality / Picking the Particles

We routinely start by reconstructing tomograms with the following parameters:

  • medium (or high) binning
    • (assuming that the tilt-series is 8k)
    • VoxelSize = 4 for a 2k reconstruction
    • (in rare cases it might be useful to use a 1k reconstruction)
  • high contrast (but low resolution)
    • SIRTCount = 10
  • no CTF-correction
    • CtfMode = false
  • Supersampling
    • OverSampling = 1
Using this reconstruction we judge the quality of the tomogram / the tilt-series. If the quality of the tilt-series is proper we use this tomogram to pick the particles and we use CTFDetector to estimate the CTF. 
CudaDeviceID = 0 1 2 3
Lambda = 1
Iterations = 1
ProjectionFile = /path/to/file
OutVolumeFile = /path/to/file
MarkerFile = /path/to/file
CtfFile =                               # No CTF file is required
RecDimesions =  1856 1856 400           # Depends on sample
VolumeShift = 0 0 0                     # Depends on sample
VoxelSize = 4
AddTiltAngle = 0
AddTiltXAngle = 0
UseFixPsiAngle = false
PsiAngle = 0
PhiAngle = 0                            # As determined by Clicker
OverSampling = 1
CtfMode = false                         # CTF is not corrected
CTFBetaFac = 0 0 0 0
Cs = 2.7
Voltage = 300
IgnoreZShiftForCTF = false
CTFSliceThickness = 50
SkipFilter = true
fourFilterLP = 0
fourFilterLPS = 0
fourFilterHP = 0
fourFilterHPS = 0
SIRTCount = 10
CorrectBadPixels = true
BadPixelValue = 10
Crop = 50 50 50 50
CropDim = 10 10 10 10
DimLength = 50 50
CutLength = 10 10
FP16Volume = false
WriteVolumeAsFP16 = false
ProjectionScaleFactor = 1
ProjectionNormalization = std
WBP = false
MagAnisotropy = 1 0                    # Should be adjusted for the microscope-specific values

High Resolution Tomogram - Extracting the Particles

In order to use the particles for sub-tomogram-averaging we reconstruct a second tomogram with high resolution and CTF-correction.

  • medium (or high) binning
    • VoxelSize = 4 for a 2k reconstruction
    • (in rare cases it might be useful to use a 1k reconstruction)
  • high resolution (but low contrast)
    • SIRTCount = 1
  • CTF-correction
    • CTFMode = true
  • Supersampling
    • OverSampling = 4
CudaDeviceID = 0 1 2 3
Lambda = 1
Iterations = 1
ProjectionFile = /path/to/file
OutVolumeFile = /path/to/file
MarkerFile = /path/to/file
CtfFile = /path/to/file
RecDimesions =  1856 1856 400           # Depends on sample
VolumeShift = 0 0 0                     # Depends on sample
VoxelSize = 4
AddTiltAngle = 0
AddTiltXAngle = 0
UseFixPsiAngle = false
PsiAngle = 0
PhiAngle = 0                            # As determined by Clicker
OverSampling = 4
CtfMode = true
CTFBetaFac = 250 0 0.001 0              # Depends on sample
Cs = 2.7
Voltage = 300
IgnoreZShiftForCTF = false
CTFSliceThickness = 50
SkipFilter = true
fourFilterLP = 0
fourFilterLPS = 0
fourFilterHP = 0
fourFilterHPS = 0
SIRTCount = 1
CorrectBadPixels = true
BadPixelValue = 10
Crop = 50 50 50 50
CropDim = 10 10 10 10
DimLength = 50 50
CutLength = 10 10
FP16Volume = false
WriteVolumeAsFP16 = false
ProjectionScaleFactor = 1
ProjectionNormalization = std
WBP = false
MagAnisotropy = 1 0                    # Should be adjusted for the microscope-specific values

Comments

We no longer reconstruct in 4k, since 4k reconstructions consume a huge amount of disk space. If 4k (or even 8k) particles are needed, one can use the local reconstruction EmSartSubvolumes.

Category:Software

Home

Clone this wiki locally