PPBench Logo

PPL Bench

Evaluation Framework for Probabilistic Programming Languages

Get StartedIntroduction

Key Features

Modular

Modular

Plug in new models and new Probabilistic Programming Language implementations of the models.

Using Predictive Log Likelihood

Using Predictive Log Likelihood

Using the Predictive Log Likelihood metrics makes PPLBench applicable for all types of PPLs.

Reusable

Reusable

Write a benchmark workflow once, reuse it across all PPLs.

Get started in seconds

  1. 1. Install PPL Bench with supported PPLs:

    pip install 'pplbench[ppls]'

  2. 2. Write a simple JSON config

    Let's store the following config into a file called example.json.

    {
    "model": {
    "class": "logistic_regression.LogisticRegression",
    "args": {
    "n": 2000,
    "k": 5,
    "rho": 3
    }
    },
    "iterations": 500,
    "trials": 2,
    "ppls": [
    {
    "name": "stan",
    "inference": {
    "class": "inference.VI",
    "infer_args": {
    "algorithm": "meanfield"
    }
    },
    "legend": {
    "color": "red",
    "name": "stan-VI-mean"
    }
    },
    {
    "name": "stan",
    "inference": {
    "class": "inference.MCMC"
    },
    "legend": {
    "color": "blue"
    }
    }
    ],
    "save_samples": true
    }

  3. 3. Run PPL Bench:

    pplbench example.json

    You should see the plots like followings in the output directory:

    pll.png

    Full predictive log likelihood

    pll_half.png

    Half predictive log likelihood

  4. 4. Try PPL Bench with other configs

    The examples directory in our GitHub repo provides a list of config files to show how PPL Bench could be used. Let's try examples/logistic_regression.json to see the performance across different PPLs:

    (You'll need to install Jags first before running this config)

    pplbench examples/logistic_regression.json

    The plots shoule look like the followings:

    pll.png

    Full predictive log likelihood

    pll_half.png

    Half predictive log likelihood

  5. To see the schema of the config file used by PPL Bench

    pplbench -h

  6. To learn more about PPL Bench

    Read our Introduction