Installation#
Dependencies#
The minimal set of dependencies for the PyLops-MPI project is:
MPI (Message Passing Interface)
Python 3.10 or greater
Additionally, to use the NCCL engine, the following additional dependencies are required:
We highly encourage using the Anaconda Python distribution or its standalone package manager Conda. However, if this is not possible, some of the dependencies must be installed prior to installing PyLops-MPI.
Download and Install MPI#
Visit the official MPI website to download an appropriate MPI implementation for your system. Follow the installation instructions provided by the MPI vendor.
Verify MPI Installation#
After installing MPI, verify its installation by opening a terminal and running the following command:
>> mpiexec --version
Install NCCL (optional)#
To obtain highly-optimized performance on GPU clusters, PyLops-MPI also supports the Nvidia’s collective communication calls (NCCL). Two additional dependencies are required, CuPy and NCCL, which can be installed using pip:
>> pip install cupy-cuda12x nvidia-nccl-cu12
Note
Replace 12x with your CUDA version (e.g., 11x for CUDA 11.x).
Step-by-step installation for users#
Currently PyLops-MPI can only be installed using pip
; simply type the following
command in your terminal to install the PyPI distribution:
>> pip install pylops-mpi
Note that when installing via pip, only required dependencies are installed.
Step-by-step installation for developers#
Fork PyLops-MPI#
Fork the PyLops-MPI repository and clone it by executing the following in your terminal:
>> git clone https://github.com/YOUR-USERNAME/pylops-mpi.git
We recommend installing dependencies into a separate environment. For that end, we provide a Makefile with useful commands for setting up the environment.
Install dependencies#
Conda (recommended)#
For a conda
environment, run
>> make dev-install_conda
This will create and activate an environment called pylops_mpi
, with all
required and optional dependencies.
If you want to enable NCCL in PyLops-MPI, run this instead
>> make dev-install_conda_nccl
Pip#
If you prefer a pip
installation, we provide the following command
>> make dev-install
Note that, differently from the conda
command, the above will not create a virtual environment.
Make sure you create and activate your environment previously.
Similarly, if you want to enable NCCL but prefer using pip, you must first check the CUDA version of your system:
>> nvidia-smi
The Makefile is pre-configured with CUDA 12.x. If you use this version, run
>> make dev-install_nccl
Otherwise, you can change the command in Makefile to an appropriate CUDA version
i.e., If you use CUDA 11.x, change cupy-cuda12x
and nvidia-nccl-cu12
to
cupy-cuda11x
and nvidia-nccl-cu11
and run the command.
Run tests#
To ensure that everything has been setup correctly, run tests:
>> make tests
Make sure no tests fail, this guarantees that the installation has been successful.
If PyLops-MPI is installed with NCCL, also run tests:
>> make tests_nccl
Run examples and tutorials#
Since the sphinx-gallery creates examples/tutorials using only a single process, it is highly recommended to test the examples/tutorials using n processes.
run examples:
>> make run_examples
run tutorials:
>> make run_tutorials
Make sure all the examples and tutorials python scripts are executed without any errors.
Add remote (optional)#
To keep up-to-date on the latest changes while you are developing, you may optionally add the PyLops-MPI repository as a remote. Run the following command to add the PyLops-MPI repo as a remote named upstream:
>> git remote add upstream https://github.com/PyLops/pylops-mpi
From then on, you can pull changes (for example, in the main branch) with:
>> git pull upstream main
Final steps#
PyLops-MPI does not enforce the use of a linter as a pre-commit hook, but we do highly encourage using one before submitting a Pull Request.
A properly configured linter (flake8
) can be run with:
>> make lint
In addition, it is highly encouraged to build the docs prior to submitting a Pull Request. Apart from ensuring that docstrings are properly formatted, they can aid in catching bugs during development.
Build the docs with:
>> make doc
or Update the docs with:
>> make docupdate