add_cookbook
add_cookbook 'My new cookbook' 'I am cookbook description' "['analogical-similarity','auto-categorisation']" |
Add a new cookbook. The 'name' argument will be slugified to create a unique identifier. |
| name (str) | Name of the new cookbook
example: 'My new cookbook' | | description (str) | Description of the new cookbook
example: I am cookbook description' | | recipes (str) | List of recipes to be included in the new cookbook
example: "['analogical-similarity','auto-categorisation']" |
|
add_recipe
add_recipe 'My new recipe' 'I am recipe description' "['category1','category2']" "['bbq-lite-age-ambiguous']" "['bertscore','bleuscore']" -p "['analogical-similarity','mmlu']" -t "['tag1','tag2']" -g "{'A':[80,100],'B':[60,79],'C':[40,59],'D':[20,39],'E':[0,19]}" |
Add a new recipe. The 'name' argument will be slugified to create a unique identifier. |
| name (str) | Name of the new recipe
example: 'My new recipe' | | description (str) | Description of the new recipe
example: 'I am recipe description' | | categories (str) | List of categories to be included in the new recipe (currently in string format). It will be converted into a list in the backend
example: "['category1','category2']" | | -t, --tags (str) | List of tags to be included in the new recipe
example: "['tag1','tag2']" | | datasets (str) | The dataset to be used
example: "['bbq-lite-age-ambiguous']" | | -p, --prompt_templates (str) | List of prompt templates to be included in the new recipe
example: "['analogical-similarity','mmlu']" | | metrics (str) | List of metrics to be included in the new recipe
example: "['bertscore','bleuscore']" | | -g, --grading_scale (str) | Dict of grading scale for the metric to be included in the new recipe
example: "{'A':[80,100],'B':[60,79],'C':[40,59],'D':[20,39],'E':[0,19]}" |
|
delete_cookbook
delete_cookbook my-new-cookbook |
Delete a cookbook. |
| cookbook (str) | Id of the cookbook
example: my-new-cookbook |
|
delete_dataset
delete_dataset bbq-lite-age-ambiguous |
Delete a dataset. |
| dataset (str) | Name of the dataset
example: bbq-lite-age-ambiguous |
|
delete_metric
delete_metric my-new-metric |
Delete a metric. |
| metric (str) | Name of the metric
example: my-new-metric |
|
delete_recipe
delete_recipe my-new-recipe |
Delete a recipe. |
| recipe (str) | Id of the recipe
example: my-new-recipe |
|
delete_result
delete_result my-new-cookbook-runner |
Delete a result. |
| result (str) | Name of the result
example: my-new-cookbook-runner |
|
delete_runner
delete_runner my-new-cookbook-runner |
Delete a runner. |
| runner (str) | Name of the runner
example: my-new-cookbook-runner |
|
list_cookbooks
list_cookbooks -f "risk" |
List all cookbooks. |
| -f, --find (str) | Optional field to find cookbook(s) with keyword
example: risk | | -p, --pagination (str) | Optional tuple to paginate cookbook(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_datasets
list_datasets -f "bbq" |
List all datasets. |
| -f, --find (str) | Optional field to find dataset(s) with keyword
example: bbq | | -p, --pagination (str) | Optional tuple to paginate dataset(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_metrics
list_metrics -f "exact" |
List all metrics. |
| -f, --find (str) | Optional field to find metric(s) with keyword
example: "exact" | | -p, --pagination (str) | Optional tuple to paginate metric(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_recipes
list_recipes -f "mmlu" |
List all recipes. |
| -f, --find (str) | Optional field to find recipe(s) with keyword
example: mmlu | | -p, --pagination (str) | Optional tuple to paginate recipes(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_results
list_results -f "my-runner" |
List all results. |
| -f, --find (str) | Optional field to find result(s) with keyword
example: my-runner | | -p, --pagination (str) | Optional tuple to paginate result(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_runners
list_runners |
List all runners. |
- |
list_runs
list_runs -f "my-run" |
List all runs. |
| -f, --find (str) | Optional field to find run(s) with keyword
example: my-run | | -p, --pagination (str) | Optional tuple to paginate run(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
run_cookbook
run_cookbook "my new cookbook runner" "['chinese-safety-cookbook']" "['openai-gpt35-turbo']" -n 1 -r 1 -s "You are an intelligent AI" |
Run a cookbook. |
| name (str) | Name of cookbook runner
example: "my new cookbook runner" | | cookbooks (str) | List of cookbooks to run
example: "['chinese-safety-cookbook']" | | endpoints (str) | List of endpoints to run
example: "['openai-gpt35-turbo']" | | -n, --num_of_prompts (int) | Number of prompts to run
example: 1 | | -r, --random_seed (int) | Random seed number
example: 1 | | -s, --system_prompt (str) | System Prompt to use
example: "You are an intelligent AI" | | -l, --runner_proc_module (str) | Runner processing module to use. Defaults to use the benchmarking module
| | -o, --result_proc_module (str) | Result processing module to use. Defaults to use the benchmarking-result module
|
|
run_recipe
run_recipe "my new recipe runner" "['bbq','mmlu']" "['openai-gpt35-turbo']" -n 1 -r 1 -s "You are an intelligent AI" |
Run a recipe. |
| name (str) | Name of recipe runner
example: "my new recipe runner" | | recipes (str) | List of recipes to run
example: "['bbq','mmlu']" | | endpoints (str) | List of endpoints to run
example: "['openai-gpt35-turbo']" | | -n, --num_of_prompts (int) | Number of prompts to run
example: 1 | | -r, --random_seed (int) | Random seed number
example: 1 | | -s, --system_prompt (str) | System Prompt to use
example: "You are an intelligent AI" | | -l, --runner_proc_module (str) | Runner processing module to use. Defaults to use the benchmarking module
| | -o, --result_proc_module (str) | Result processing module to use. Defaults to use the benchmarking-result module
|
|
update_cookbook
update_cookbook my-new-cookbook "[('name', 'Updated Cookbook Name'), ('description', 'Updated description'), ('recipes', ['analogical-similarity'])]" |
Update a cookbook. |
| cookbook (str) | Id of the cookbook
example: my-new-cookbook | | update_values (str) | Update cookbook key/value
example: "[('name', 'Updated Cookbook Name'), ('description', 'Updated description'), ('recipes', ['analogical-similarity'])]" |
|
update_recipe
update_recipe my-new-recipe "[('name', 'Updated Recipe Name'), ('tags', ['fairness', 'bbq'])]" |
Update a recipe. |
| recipe (str) | Id of the recipe
example: my-new-recipe | | update_values (str) | Update recipe key/value
example: "[('name', 'Updated Recipe Name'), ('tags', ['fairness', 'bbq'])]" |
|
view_cookbook
view_cookbook my-new-cookbook |
View a cookbook. |
| cookbook (str) | Id of the cookbook
example: my-new-cookbook |
|
view_dataset
view_dataset bbq-lite-age-ambiguous |
View a dataset file. |
| dataset_filename (str) | Name of the dataset file
example: bbq-lite-age-ambiguous |
|
view_metric
view_metric my-new-metric |
View a metric file. |
| metric_filename (str) | Name of the metric file
example: my-new-metric |
|
view_recipe
view_recipe my-new-recipe |
View a recipe. |
| recipe (str) | Id of the recipe
example: my-new-recipe |
|
view_result
view_result my-new-cookbook-runner |
View a result file. |
| result_filename (str) | Name of the result file
example: my-new-cookbook-runner |
|
view_run
view_run my-new-cookbook-runner |
View a runner runs. |
| runner_id (str) | Name of the runner
example: my-new-cookbook-runner |
|
view_runner
view_runner my-new-cookbook-runner |
View a runner. |
| runner (str) | Name of the runner
example: my-new-cookbook-runner |
|
add_dataset
add_dataset 'dataset-name' 'A brief description' 'http://reference.com' 'MIT' 'csv' "{'csv_file_path': '/path/to/your/file.csv'}"
add_dataset 'dataset-name' 'A brief description' 'http://reference.com' 'MIT' 'hf' "{'dataset_name': 'cais/mmlu', 'dataset_config': 'college_biology', 'split': 'test', 'input_col': ['question','choices'], 'target_col': 'answer'}" |
Add a new dataset. The 'name' argument will be slugified to create a unique identifier. |
| name (str) | Name of the new dataset
example: dataset-name | | description (str) | Description of the new dataset
example: 'A brief description' | | reference (str) | Reference of the new dataset
example: 'http://reference.com' | | license (str) | License of the new dataset
example: 'MIT' | | method (str) | Method to convert the new dataset. Choose either 'hf' or 'csv'.
example: hf | | params (literal_eval) | Params of the new dataset in dictionary format.
For 'csv' method: "{'csv_file_path': '/path/to/your/file.csv'}"
For 'hf' method: "{'dataset_name': 'cais_mmlu', 'dataset_config': 'college_biology', 'split': 'test', 'input_col': ['questions','choices'], 'target_col': 'answer'}" |
|
add_endpoint
add_endpoint openai-connector 'OpenAI GPT3.5 Turbo 1106' MY_URI ADD_YOUR_TOKEN_HERE 1 1 "{'temperature': 0.5, 'model': 'gpt-3.5-turbo-1106'}" |
Add a new endpoint. The 'name' argument will be slugified to create a unique identifier. |
| connector_type (str) | Type of connection for the endpoint
example: openai-connector | | name (str) | Name of the new endpoint
example: 'OpenAI GPT3.5 Turbo 1106' | | uri (str) | URI of the new endpoint
example: MY_URI | | token (str) | Token of the new endpoint
example: ADD_YOUR_TOKEN_HERE | | max_calls_per_second (int) | Max calls per second of the new endpoint
example: 1 | | max_concurrency (int) | Max concurrency of the new endpoint
example: 1 | | params (str) | Params of the new endpoint
example: "{'temperature': 0.5, 'model': 'gpt-3.5-turbo-1106'}" |
|
delete_endpoint
delete_endpoint openai-gpt4 |
Delete an endpoint. |
| endpoint (str) | ID of the endpoint
example: openai-gpt4 |
|
delete_prompt_template
delete_prompt_template squad-shifts |
Delete a prompt template. |
| prompt_template (str) | The ID of the prompt template to delete
example: squad-shifts |
|
list_connector_types
list_connector_types -f "openai" |
List all connector types. |
| -f, --find (str) | Optional field to find connector type(s) with keyword
example: openai | | -p, --pagination (str) | Optional tuple to paginate connector type(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_endpoints
list_endpoints -f "gpt" |
List all endpoints. |
| -f, --find (str) | Optional field to find endpoint(s) with keyword
example: gpt | | -p, --pagination (str) | Optional tuple to paginate endpoint(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_prompt_templates
list_prompt_templates -f "toxicity" |
List all prompt templates. |
| -f, --find (str) | Optional field to find prompt template(s) with keyword
example: toxicity | | -p, --pagination (str) | Optional tuple to paginate prompt template(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
update_endpoint
update_endpoint openai-gpt4 "[('name', 'my-special-openai-endpoint'), ('uri', 'my-uri-loc'), ('token', 'my-token-here')]" |
Update an endpoint. |
| endpoint (str) | ID of the endpoint. This field is not editable via CLI after creation.
example: openai-gpt4 | | update_kwargs (str) | Update endpoint key/value
example: "[('name', 'my-special-openai-endpoint'), ('uri', 'my-uri-loc'), ('token', 'my-token-here')]" |
|
view_endpoint
view_endpoint openai-gpt4 |
View an endpoint. |
| endpoint (str) | ID of the endpoint
example: openai-gpt4 |
|
add_bookmark
add_bookmark openai-connector 2 my-bookmarked-prompt |
Bookmark a prompt |
| endpoint (str) | Endpoint which the prompt was sent to.
example: openai-connector | | prompt_id (int) | ID of the prompt (the leftmost column)
example: 2 | | bookmark_name (str) | Name of the bookmark
example: my-bookmarked-prompt |
|
delete_attack_module
delete_attack_module sample_attack_module |
Delete an attack module. |
| attack_module (str) | The ID of the attack module to delete
example: sample_attack_module |
|
delete_bookmark
delete_bookmark my_bookmarked_prompt |
Delete a bookmark |
| bookmark_name (str) | Name of the bookmark
example: my_bookmarked_prompt |
|
delete_context_strategy
delete_context_strategy add_previous_prompt |
Delete a context strategy. |
| context_strategy (str) | The ID of the context strategy to delete
example: add_previous_prompt |
|
delete_session
delete_session my-test-runner |
Delete a session |
| session (str) | The runner ID of the session to delete
example: my-test-runner |
|
export_bookmarks
export_bookmarks "my_list_of_exported_bookmarks" |
Exports bookmarks as a JSON file |
| bookmark_list_name (str) | Name of the exported bookmarks JSON file you want to save as (without the .json extension)
example: my_list_of_exported_bookmarks |
|
list_attack_modules
list_attack_modules -f "text" |
List all attack modules. |
| -f, --find (str) | Optional field to find attack module(s) with keyword
example: text | | -p, --pagination (str) | Optional tuple to paginate attack module(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_bookmarks
list_bookmarks -f my_bookmark |
List all bookmarks. |
| -f, --find (str) | Optional field to find bookmark(s) with keyword
example: my_bookmark | | -p, --pagination (str) | Optional tuple to paginate bookmark(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_context_strategies
list_context_strategies -f "previous_prompt" |
List all context strategies. |
| -f, --find (str) | Optional field to find context strategies with keyword
example: previous_prompt | | -p, --pagination (str) | Optional tuple to paginate context strategies(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
list_sessions
list_sessions -f "my-sessions" |
List all sessions. |
| -f, --find (str) | Optional field to find session(s) with keyword
example: my-sessions | | -p, --pagination (str) | Optional tuple to paginate session(s). E.g. (2,10) returns 2nd page with 10 items in each page.
example: (2,10) |
|
new_session
new_session my-runner -e "['openai-gpt4']" -c add_previous_prompt -p mmlu |
Creates a new red teaming session. |
| runner_id (str) | ID of the runner. Creates a new runner if runner does not exist.
example: my-runner | | -e, --endpoints (str) | List of endpoint(s) for the runner that is only compulsory for creating a new runner.
example: "['openai-gpt4']" | | -c, --context_strategy (str) | Name of the context_strategy to be used - indicate context strategy here if you wish to use with the selected attack.
example: add_previous_prompt | | -p, --prompt_template (str) | Name of the prompt template to be used - indicate prompt template here if you wish to use with the selected attack.
example: mmlu |
|
Automated red teaming
run_attack_module sample_attack_module "this is my prompt" -s "test system prompt" -m bleuscore |
Runs automated red teaming in the current session. |
| attack_module_id (str) | ID of the attack module.
example: sample_attack_module | | prompt (str) | Prompt to be used for the attack.
example: "this is my prompt" | | -s, --system_prompt (str) | System Prompt to be used for the attack. If not specified, the default system prompt will be used.
example: "test system prompt" | | -c, --context_strategy (str) | Name of the context strategy module to be used. If this is set, it will overwrite the context strategy set in the session while running this attack module.
example: add_previous_prompt | | -n, --cs_num_of_prev_prompts (str) | The number of previous prompts to use with the context strategy. If this is set, it will overwrite the number of previous prompts set in the session while running this attack module.
example: 5 | | -p, --prompt_template (str) | Name of the prompt template to be used. If this is set, it will overwrite the prompt template set in the session while running this attack module.
example: mmlu | | -m, --metric (str) | Name of the metric module to be used.
example: bleuscore | | -o, --optional_args (str) | Optional parameters to input into the red teaming module.
example: "{'my_attack_module_custom_field': 1.0}" |
|
use_bookmark
use_bookmark my_bookmark |
Use a bookmarked prompt |
| bookmark_name (str) | Name of the bookmark
example: my_bookmark |
|
use_context_strategy
use_context_strategy my_strategy_one |
Use a context strategy. |
| context_strategy (str) | The ID of the context strategy to use
example: my_strategy_one | | -n, --num_of_prev_prompts (int) | The number of previous prompts to use with the context strategy
example: 6 |
|
use_prompt_template
use_prompt_template 'analogical-similarity' |
Use a prompt template. |
| prompt_template (str) | Name of the prompt template
example: analogical-similarity |
|
use_session
use_session 'my-runner' |
Use an existing red teaming session by specifying the runner ID. |
| runner_id (str) | The ID of the runner which contains the session you want to use.
example: my-runner |
|
view_bookmark
view_bookmark my_bookmarked_prompt |
View a bookmark |
| bookmark_name (str) | Name of the bookmark you want to view
example: my_bookmarked_prompt |
|