Deep Learning Benchmarks

Dependencies

• python : version 3.6 or newer (tested on 3.6 and 3.9)

Python packages, can be installed with pip (see the --platform option in the usage section):

• matplotlib
• seaborn
• tensorflow : tested with version 2.6 (for --platform TF2 TF2_V1) and 1.15 (for --platform TF1)
• torch : tested with version 1.9 (for --platform Torch)
• jaxlib : tested with version 0.1.71+cuda111 (for --platform Torch)
• jax : tested with version 0.2.21 (for --platform Torch)

To run with docker:

• docker
• docker-compose
• libnvidia-container : for arch (AUR) (required for nvidia-container-toolkit)
• nvidia-container-toolkit : for arch (AUR) (required for gpu accelerated docker using --gpus)
• nvidia-container-runtime : for arch (AUR) (easier docker-compose integration using runtime: nvidia)

See other distribution documentation to get GPU accelerated docker containers.

Usage

Runing the benchmark.py script will run all the possible benchmarks automatically.

Options can be seen using the --help or -h argument :

usage: benchmark.py [-h] [--output OUTPUT] [--no-benchmark] [--no-compare] [--count COUNT] [--platform [PLATFORM ...]]
                    [--data [DATA ...]] [--op [OP ...]] [--list-op] [--list-platform] [--list-data]
                    [--experiment-time EXPERIMENT_TIME]

optional arguments:
  -h, --help            show this help message and exit
  --output OUTPUT       Path to output files (default: output)
  --no-benchmark        Avoid running benchmarks
  --no-compare          Avoid running platform comparaison
  --count COUNT         Number of experiments per benchmark (for stastistical analysis)
  --platform [PLATFORM ...]
                        List of platform to benchmark [TF1, TF2, Torch] (else all are used)
  --data [DATA ...]     List of data type to benchmark [float16, float32, float64] (else all are used)
  --op [OP ...]         List of operation to benchmark (add, mul, div, matmul, etc) (else all are used)
  --list-op             List all possible operation to benchmark (no further action will be done)
  --list-platform       List all possible platform to benchmark (no further action will be done)
  --list-data           List all possible data to benchmark (no further action will be done)
  --experiment-time EXPERIMENT_TIME
                        Change time (in s) per experiment (default=1.000s)

Configuration

In the config/benchmark.py file there are parameters that can be changed:

• ExperimentCategory : Enum class that defines ranges of Mop (Millions of operations) used for comparisons
• EXPERIMENT_TIME : float number to set the amount of time to run each benchmark experiment (in seconds)
• []_ARGS : each kind of benchmark will be run with specific arguments that can be changed here (usually batch sizes and input shapes)

Using Docker

There are 3 services to use with docker-compose (tensorflow_v1, tensorflow and torch), simply run a service:

docker-compose run --rm torch

Arguments can be passed using the BENCH_ARGS environment variable like:

BENCH_ARGS="--op nn_dense_x5 --data float32 --no-compare" docker-compose run --rm tensorflow_v1

Notes

Plots y axis are automatically changed to logarithmic scale if the range is too wide, categorized comparisons are forced to not be in logarithmic scale for clarity (categories are implemented for this reason).

Future

• More operations : Conv1d, Conv2D, DepthWiseConv2D, RNN