Installing PVade

On a Unix base machine

PVade is a software that uses FEniCSx for its Finite Element Computation. For more information about FEniCSx please refer to https://github.com/FEniCS/dolfinx.

In addition to FEniCSx, PVade uses multiple python packages as part of the pre- and post-processing steps. PVade dependencies are included in environment.yaml.

In order to start using PVade, we can use Conda/Mamba for the creation of an environement containg all the necessary dependencies.

In order to obtain Mamba we can use the following ressource https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html.

In order to obtain Conda we can use the following ressource https://conda.io/projects/conda/en/latest/user-guide/install/index.html.

Note

We recommand using Mamba for its fast installation of the environement.

Building Mamba Environment

To use this software, begin by creating a Conda environment using the provided environment.yaml file:

mamba env create -n my_env_name -f environment.yaml

where my_env_name can be replaced with a short name for your Conda environment. When the environment finishes installing, activate it with:

mamba activate my_env_name

from within your activate Conda environment, a simulation can be executed with:

python main.py --command_line_arg value

We can test the successful installation of PVade and it’s MPI implementation by running the following example

mpirun -np $num_cores python -u $PVade/example/poissoneq.py 64  cg none 1

The example solve a Poisson’s equation in 3 dimensions using 64 elements and 1 order Lagrange shape functions with cg as the ksp solver and no preconditioners. For more details about the poisson’s problem we refer the use to the folowing link https://jsdokken.com/dolfinx-tutorial/chapter1/fundamentals.html

Accessing the DOLFINx programming environment on Windows: a review

Current as of: May 2024.

[Instructions for DOLFINx](https://github.com/FEniCS/dolfinx/blob/main/README.md)

  1. Using WSL2

A Windows 11 computer should have WSL natively. In a command prompt, wsl --status will tell you if you have WSL version 2, which is generally preferred.

To install a Ubuntu distro in WSL 2, in a command prompt, run:

wsl --install

You’ll be instructed to create a username and password. After install, your prompt will be logged in to Ubuntu on WSL. You can also access this prompt in the future by searching your apps for Ubuntu.

In Ubuntu, follow the instructions to install the DOLFINx environment.:

add-apt-repository ppa:fenics-packages/fenics
apt update
apt install fenicsx

This may take up to an hour to install.

Following installation, you can access your environment by opening Ubuntu. You can use apt or pip to install any additional packages you need. If you want to access this environment in VSCode, you can use the [WSL extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl), which works very similar to remotely accessing HPC systems through SSH with the [SSH extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh).

I found this method to be the easiest to set up, access, and understand.

  1. Using Docker

[Install](https://docs.docker.com/desktop/install/windows-install/) Docker. Docker Desktop needs a license, as NREL is a large organization. I don’t know if these are available. However, a Docker Destop license is not necessary for installing a DOLFINx environment in Docker.

Docker uses images, which are recipes for programming environments, and containers, which are instances based on an image. Containers can be created and destroyed, entered and exited.

To get up and running with docker, run

docker run --name fenicsx -ti dolfinx/dolfinx:nightly

The image, dolfinx/dolfinx:nightly, can be replaced with other options listed in the instructions. In most cases I would recommend dolfinx/dolfinx:stable, but I used the nightly version to avoid a bug with DOLFINx that hopefully would be resolved by the time this is relevant.

This instruction creates a container from the image, and the first time it is run, it also pulls the image data. This may take up to 20 minutes. If you run this command again, it will create a new container from the same image. Note that you would want to leave out or change the –name fenicsx tag, and that it would take far less time the second time, as the image is already pulled.

A container runs one command. By default, that command is to open a terminal. When you exit that terminal, the container also exits.

You may re-enter a Docker container by running:

docker start -i fenics

Exiting and re-entering containers will keep data like files downloaded and packages installed, on top of the original image. Removing and re-creating a new container will not keep this data, only the original image.

To install additional packages, you may use pip while you are inside your docker container.

There are ways to keep files from one container to another or from your container to your local computer. There are also ways to modify and update the docker container with additional packages. I do not know them.

Docker works on Windows. If you understand Docker, I’d recommend using Docker for DOLFINx. If you don’t, I’d recommend WSL2.

3. Using Conda Windows supports Miniconda and Anaconda. Current DOLFINx instructions read:

conda create -n fenicsx-env
conda activate fenicsx-env
conda install -c conda-forge fenics-dolfinx mpich pyvista
On conda-forge, pyvista supports Windows, but mpich and fenics-dolfinx do not. If that changed, you could use conda environments on Windows by:

  1. Installing Miniconda or Anaconda

  2. Opening Anaconda prompt

  3. Following the above instructions to create and set up the environment

It’s unclear whether mpich and fenics-dolfinx cannot be ported to Windows, or just haven’t yet been. At present, though, they aren’t, so a conda DOLFINx environment through conda-forge is not possible.

  1. Using Spack

Windows only technically supports spack. In theory, one would use spack on Windows according to [these instructions](https://spack.readthedocs.io/en/latest/getting_started.html#spack-on-windows). Your procedure would look like:

  1. Installing prerequisites VSCode with C++ compiler options, Python, Git

2. Cloning spack

git clone https://github.com/spack/spack.git

  1. Opening a spack prompt, by running bin\spack_cmd.bat.

4. Setting up spack

spack compiler find
spack external find cmake
spack external find ninja

5. Setting up your spack environment

spack env create fenicsx-env
spack env activate fenicsx-env
spack add fenics-dolfinx+adios2 py-fenics-dolfinx cflags="-O3" fflags="-O3"
spack install

I got no further than step 1; I couldn’t find options to install VSCode with the needed C++ compiler. Many spackages are also not supported by Windows, so you likely would also not be able to run the spack add line in step 5.

Development to get spack to work on Windows is underway. Development to port the relevant spackages to Windows is not. Verdict: DOLFINx with Spack on Windows is impossible at current stages, and a pain even if possible.

On NREL HPC machine Kestrel

In order to use PVade on Kestrel, we can use one of the two options.

  • A conda/mamba installation

  • Loading FEniCSx as a module

  1. conda/mamba installation

In order to install PVade, it is recommend to use compute node. You can allocate one use it interactively through:

~$ salloc --nodes=1 --time=4:00:00 --partition=$partition_name --account $account_name --mem=0 --exclusive

Make sure you specify the partition name $partition_name and the account name $account_name. Next, we clone the repository from https://github.com/NREL/PVade.git.

~$ git clone https://github.com/NREL/PVade.git

Note

the same can be achieved by downloading the latest release from https://github.com/NREL/PVade/releases

We will refer to $PVade as the location of the cloned repo. We change the directory to $PVade and load mamba.

~$ cd $PVade/
~$ module unload PrgEnv-cray/8.3.3
~$ module load mamba

Note

The same can be achived by using Conda. Mamba was shown to be faster.

We then create an environment my_env_name and activate it.

~$ mamba env create -n PVade_public -f environment.yaml
~$ mamba activate my_env_name

To test the installation we can run an example using the command

mpirun -np $num_cores python -u $PVade/example/poissoneq.py 64  cg none 1

The example solve a Poisson’s equation in 3 dimensions using 64 elements and 1 order Lagrange shape functions with cg as the ksp solver and no preconditioners.

Note

The conda installed PVade requires the use of mpirun instead of srun since it was not configured against Slurm.

  1. Module access

On Kestrel, PVade is installed and available as a module. PVade can be accessed by loading:

module load fenicsx

This instance of PVade leverages a FEniCSx installation that leverages GNU Programming environment and cray-mpich for its mpi communication.