Installation#

Dependencies#

The minimal set of dependencies for the PyLops-MPI project is:

Additionally, to use the NCCL engine, the following additional dependencies are required:

We highly encourage using the Anaconda Python distribution or its standalone package manager Conda. However, if this is not possible, some of the dependencies must be installed prior to installing PyLops-MPI.

Download and Install MPI#

Visit the official MPI website to download an appropriate MPI implementation for your system. Follow the installation instructions provided by the MPI vendor.

Verify MPI Installation#

After installing MPI, verify its installation by opening a terminal and running the following command:

>> mpiexec --version

Install NCCL (optional)#

To obtain highly-optimized performance on GPU clusters, PyLops-MPI also supports the Nvidia’s collective communication calls (NCCL). Two additional dependencies are required, CuPy and NCCL, which can be installed using pip:

>> pip install cupy-cuda12x nvidia-nccl-cu12

Note

Replace 12x with your CUDA version (e.g., 11x for CUDA 11.x).

Step-by-step installation for users#

Currently PyLops-MPI can only be installed using pip; simply type the following command in your terminal to install the PyPI distribution:

>> pip install pylops-mpi

Note that when installing via pip, only required dependencies are installed.

Step-by-step installation for developers#

Fork PyLops-MPI#

Fork the PyLops-MPI repository and clone it by executing the following in your terminal:

>> git clone https://github.com/YOUR-USERNAME/pylops-mpi.git

We recommend installing dependencies into a separate environment. For that end, we provide a Makefile with useful commands for setting up the environment.

Install dependencies#

Pip#

If you prefer a pip installation, we provide the following command

>> make dev-install

Note that, differently from the conda command, the above will not create a virtual environment. Make sure you create and activate your environment previously.

Similarly, if you want to enable NCCL but prefer using pip, you must first check the CUDA version of your system:

>> nvidia-smi

The Makefile is pre-configured with CUDA 12.x. If you use this version, run

>> make dev-install_nccl

Otherwise, you can change the command in Makefile to an appropriate CUDA version i.e., If you use CUDA 11.x, change cupy-cuda12x and nvidia-nccl-cu12 to cupy-cuda11x and nvidia-nccl-cu11 and run the command.

Run tests#

To ensure that everything has been setup correctly, run tests:

>> make tests

Make sure no tests fail, this guarantees that the installation has been successful.

If PyLops-MPI is installed with NCCL, also run tests:

>> make tests_nccl

Run examples and tutorials#

Since the sphinx-gallery creates examples/tutorials using only a single process, it is highly recommended to test the examples/tutorials using n processes.

run examples:

>> make run_examples

run tutorials:

>> make run_tutorials

Make sure all the examples and tutorials python scripts are executed without any errors.

Add remote (optional)#

To keep up-to-date on the latest changes while you are developing, you may optionally add the PyLops-MPI repository as a remote. Run the following command to add the PyLops-MPI repo as a remote named upstream:

>> git remote add upstream https://github.com/PyLops/pylops-mpi

From then on, you can pull changes (for example, in the main branch) with:

>> git pull upstream main

Final steps#

PyLops-MPI does not enforce the use of a linter as a pre-commit hook, but we do highly encourage using one before submitting a Pull Request. A properly configured linter (flake8) can be run with:

>> make lint

In addition, it is highly encouraged to build the docs prior to submitting a Pull Request. Apart from ensuring that docstrings are properly formatted, they can aid in catching bugs during development.

Build the docs with:

>> make doc

or Update the docs with:

>> make docupdate