Skip to content

Running Benchmarks

This page documents the steps to starting a benchmark run

  1. To begin, click on the "Run Benchmark" button as indicated

    Run Benchmark Button

  2. Pick the desired endpoints for this benchmark run. You can also configure them at this point by clicking on their respective "edit" button. Click on "NEXT" to proceed

    Select Endpoints

  3. Pick the desired cookbooks for this benchmark run. You cannot configure the cookbooks at this point.

    Select Cookbooks

    However, you can click "About" to find out more about the cookbook, which will contain more detailed information about the specific cookbook

    Cookbook Details

  4. Before you can start running benchmarks, you have to provide the following information. These will be included in the report generated at the end of the run.

    Name Description Example
    Name (Required) A unique name for you to identify this benchmark run by GPT4 vs Claude on safety benchmarks
    Description Describe the purpose and scope of this benchmark run. Comparing GPT4 and Claude to determine which model is safer as a chatbot
    Run a smaller set The number of prompts per dataset, as specified in the recipe, to be run. Indicating 0 will run the full set.
    * Before running the full recommended set, you may want to run a smaller number of prompts from each recipe to do a sanity check.    		 5

    When ready, click ‘Run’ to start running the benchmarks.

    Benchmark Initialisation

  5. When a benchmark test is running, you can click on ‘See Details’ to recap on what is currently being run. You can also cancel the run

    Progress of Benchmark

  6. A report will be generated once the run is completed. Meanwhile, you can:

    • Start Red Teaming to discover new vulnerabilities
    • Create a custom cookbook by curating your own set of recipes
    • Return back to the Main Page

  7. To view the progress of the run, click on the bell icon and select the specific benchmark run.

    Progress of Multiple Runs

  8. Once a benchmark run is completed, you can click on ‘View Report’

    View Report

  9. One report will be generated for each tested endpoint. Click on the dropdown to toggle the report displayed. You can also download the HTML report and the detailed results as a JSON file.

    Download Benchmark Report

View Run History

  1. You can also view the details of previous runs through 2 methods

    1. By clicking on ‘benchmarking’ icon on the Sidebar and clicking the ‘View Past Runs’ button

    Benchmark View Past Runs

    1. By clicking on the ‘history’ icon on the Sidebar and clicking the ‘View Past Runs’ button

    History View Past Runs

    This is the window that will list the information of the previous runs. For more detailed information of each run, click in the "View Results" button

    Past Runs