Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions docs/source/howto/how_to_run/at_diamond.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,17 @@ This will allow HTTomo to be run on the local machine like so:
Parallel
~~~~~~~~

A parallel run of HTTomo at Diamond would usually be done on a compute cluster.
However, there are cases where a parallel run on a local machine on cropped data
is also useful, so that has also been described below.
Parallel execution of HTTomo at Diamond is typically performed on a compute cluster.
The HTTomo launcher is integrated with the SLURM workload manager via REST APIs and is
configured to submit jobs to the :code:`wilson` compute cluster.

TODO (:code:`httomo_mpi_local`?)
To run HTTomo, either log in to a Wilson compute node directly or load the HTTomo environment on a workstation
with :code:`module load httomo`.

Once the environment is loaded, HTTomo jobs can be submitted and executed in parallel as described below.

.. code-block:: console

$ httomo_mpi IN_FILE YAML_CONFIG OUT_DIR
`
Also look for help with :code:`httomo_mpi --help`.
23 changes: 14 additions & 9 deletions docs/source/howto/run_httomo.rst
Original file line number Diff line number Diff line change
Expand Up @@ -419,18 +419,23 @@ block size happens to improve performance for the CPU-only section(s).
:code:`--max-memory`
~~~~~~~~~~~~~~~~~~~~

HTTomo supports running on both:
HTTomo supports execution on both:

- a compute cluster, where RAM on the host machine is often quite large
- a personal machine, where RAM is not nearly as large
- compute clusters, where large amounts of system RAM are typically available
- personal workstations or laptops, where available RAM is often more limited

This is done by a mechanism within HTTomo to hold data in RAM wherever there is
enough RAM to do the required processing, and write data to a file if there is not
enough RAM.
To accommodate these different environments, HTTomo dynamically manages intermediate
data during pipeline execution. Whenever sufficient RAM is available, data is kept
in memory to maximise performance. If memory requirements exceed the available
RAM, HTTomo automatically stores intermediate data on disk and reloads it as needed.

The :code:`--max-memory` option specifies the amount of RAM available to HTTomo.
It sets the limit in terms of the maximum amount of the CPU memory per process available
on the system. For instance, if you're running HTTomo on 1 process serially, you need
to pass the value of the CPU memory available on your system, e.g., :code:`--max-memory 32G` would
set it to 32 Gigabyte. Providing an accurate value allows HTTomo to optimise memory usage
while avoiding out-of-memory errors.

The :code:`--max-memory` flag is for telling HTTomo how much RAM the machine has,
so then it can switch to using a file during execution of the pipeline if
necessary.

:code:`--save-snapshots`
~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
2 changes: 1 addition & 1 deletion httomo/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ def check(pipeline: Union[Path, str], in_data_file: Optional[Path] = None):
"--max-memory",
type=click.STRING,
default="0",
help="Limit the amount of memory used by the pipeline to the given memory (supports strings like 3.2G or bytes)",
help="The maximum amount of the CPU memory per process available on the system (supports strings like 3.2G or bytes)",
)
@click.option(
"--save-snapshots",
Expand Down
Loading