Installation instructions for HPC machines
--------------------------------------------------------------


### Illinois campus cluster
#### Setting up your conda installation

Make sure your `.bashrc` is not loading another conda environment and that `PYTHONPATH` is not set. If it is, clear those things, log out, and log back in.

```
module load anaconda/3
conda create --name fast-mpi4py python=3.8
module load openmpi/3.1.1-gcc-7.2.0
export MPICC=$(which mpicc)
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py
pip install -v --no-binary mpi4py mpi4py
pip install -v --no-binary numpy numpy
pip install pyscf
module load git
pip install git+git://github.com/WagnerGroup/pyqmc --upgrade
pip install snakemake
```

#### Submission script

```
#!/bin/bash
#SBATCH --time=24:00:00                  # Job run time (hh:mm:ss)
#SBATCH --nodes=1                        # Number of nodes
#SBATCH --ntasks-per-node=40             # Number of task (cores/ppn) per node
#SBATCH --job-name=a120
#SBATCH --partition=wagner
#SBATCH --output=myjob.o%j              # Name of batch job output file

module load anaconda/3
module load openmpi/3.1.1-gcc-7.2.0
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py

srun python test.py 
```



#### snakemake setup

I would suggest starting with the setup here: https://github.com/WagnerGroup/Energy-Entropy

```
module load anaconda/3
module load openmpi/3.1.1-gcc-7.2.0
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py
pip install cookiecutter
cookiecutter https://github.com/Snakemake-Profiles/slurm.git

```

The profile will be installed in `~/.config/snakemake/slurm`. I have changed my version of `slurm-submit.py` to include the partition:
```
RESOURCE_MAPPING = {
    "time": ("time", "runtime", "walltime"),
    "mem": ("mem", "mem_mb", "ram", "memory"),
    "mem-per-cpu": ("mem-per-cpu", "mem_per_cpu", "mem_per_thread"),
    "nodes": ("nodes", "nnodes"),
    "partition": ("partition",)
}
```

Then your snakemake can look like the following, if you'd like to run on `qmc_threads` cores on one node and the secondary partition. Note the resources section in the rule.
```
rule VMC_MF:
    input: hf="{dir1}/{dir2}/mf.chk", wffile="{dir1}/{dir2}/vmc_mf_{tol}_{orbs}_{nconfig}.chk"
    output: "{dir1}/{dir2}/eval_mf_{tol}_{orbs}.chk"
    threads: qmc_threads
    resources:
        walltime="4:00:00", partition="secondary"
    run:
        with concurrent.futures.ProcessPoolExecutor(max_workers=qmc_threads) as client:
            functions.evaluate_vmc(input.hf, None, input.wffile, output[0], 
                  slater_kws=None, nblocks=1000, client=client, npartitions=qmc_threads)
```
TODO: change this to work with MPI.

You can run by doing something like the following:
```
module load anaconda/3
module load openmpi/3.1.1-gcc-7.2.0
.  /usr/local/anaconda/5.2.0/python3/etc/profile.d/conda.sh
conda activate fast-mpi4py
hostname >> run.out
nohup nice snakemake --jobs=10  --profile=slurm [filenames] > vmc.out & 
```
You can run this on a login node and it will execute your jobs.



### Summit

Some information is available here: https://www.olcf.ornl.gov/wp-content/uploads/2019/02/STW_Feb_20190211_summit_workshop_python.pdf

Set up your environment:
```
module load python/3.7.0-anaconda3-5.3.0
conda create -n pyqmc3.8 python=3.8
conda init
. .bashrc 
conda install numpy pandas h5py
module load gcc/4.8.5
CC=gcc MPICC=mpicc pip install --no-binary mpi4py install mpi4py
```

Install pyscf and pyqmc.
```
module load gcc/4.8.5
module load cmake
module load openblas/0.3.9-omp

git clone https://github.com/pyscf/pyscf
cd build
cmake ..
make 
cd [pyscf root directory]
pip install .
pip install pyqmc
```