diff --git a/docs/source/howto/how_to_run/at_diamond.rst b/docs/source/howto/how_to_run/at_diamond.rst index 7d2c63aa1..d1eaeb1a3 100644 --- a/docs/source/howto/how_to_run/at_diamond.rst +++ b/docs/source/howto/how_to_run/at_diamond.rst @@ -36,8 +36,17 @@ This will allow HTTomo to be run on the local machine like so: Parallel ~~~~~~~~ -A parallel run of HTTomo at Diamond would usually be done on a compute cluster. -However, there are cases where a parallel run on a local machine on cropped data -is also useful, so that has also been described below. +Parallel execution of HTTomo at Diamond is typically performed on a compute cluster. +The HTTomo launcher is integrated with the SLURM workload manager via REST APIs and is +configured to submit jobs to the :code:`wilson` compute cluster. -TODO (:code:`httomo_mpi_local`?) +To run HTTomo, either log in to a Wilson compute node directly or load the HTTomo environment on a workstation +with :code:`module load httomo`. + +Once the environment is loaded, HTTomo jobs can be submitted and executed in parallel as described below. + +.. code-block:: console + + $ httomo_mpi IN_FILE YAML_CONFIG OUT_DIR +` +Also look for help with :code:`httomo_mpi --help`. \ No newline at end of file diff --git a/docs/source/howto/run_httomo.rst b/docs/source/howto/run_httomo.rst index dd6d1f631..af347a661 100644 --- a/docs/source/howto/run_httomo.rst +++ b/docs/source/howto/run_httomo.rst @@ -419,18 +419,23 @@ block size happens to improve performance for the CPU-only section(s). :code:`--max-memory` ~~~~~~~~~~~~~~~~~~~~ -HTTomo supports running on both: +HTTomo supports execution on both: -- a compute cluster, where RAM on the host machine is often quite large -- a personal machine, where RAM is not nearly as large +- compute clusters, where large amounts of system RAM are typically available +- personal workstations or laptops, where available RAM is often more limited -This is done by a mechanism within HTTomo to hold data in RAM wherever there is -enough RAM to do the required processing, and write data to a file if there is not -enough RAM. +To accommodate these different environments, HTTomo dynamically manages intermediate +data during pipeline execution. Whenever sufficient RAM is available, data is kept +in memory to maximise performance. If memory requirements exceed the available +RAM, HTTomo automatically stores intermediate data on disk and reloads it as needed. + +The :code:`--max-memory` option specifies the amount of RAM available to HTTomo. +It sets the limit in terms of the maximum amount of the CPU memory per process available +on the system. For instance, if you're running HTTomo on 1 process serially, you need +to pass the value of the CPU memory available on your system, e.g., :code:`--max-memory 32G` would +set it to 32 Gigabyte. Providing an accurate value allows HTTomo to optimise memory usage +while avoiding out-of-memory errors. -The :code:`--max-memory` flag is for telling HTTomo how much RAM the machine has, -so then it can switch to using a file during execution of the pipeline if -necessary. :code:`--save-snapshots` ~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/httomo/cli.py b/httomo/cli.py index 4775c00a4..7344598f3 100644 --- a/httomo/cli.py +++ b/httomo/cli.py @@ -171,7 +171,7 @@ def check(pipeline: Union[Path, str], in_data_file: Optional[Path] = None): "--max-memory", type=click.STRING, default="0", - help="Limit the amount of memory used by the pipeline to the given memory (supports strings like 3.2G or bytes)", + help="The maximum amount of the CPU memory per process available on the system (supports strings like 3.2G or bytes)", ) @click.option( "--save-snapshots",