add_cookbook
add_cookbook 'My new cookbook' 'I am cookbook description' "['analogical-similarity','auto-categorisation']" |
Add a new cookbook. The 'name' argument will be slugified to create a unique identifier. |
| name (str) | Name of the new cookbook
example: 'My new cookbook' | | description (str) | Description of the new cookbook
example: 'I am cookbook description' | | recipes (str) | List of recipes to be included in the new cookbook (currently in string format). It will be converted into a list in the backend.
example: "['analogical-similarity','auto-categorisation']" |
|
list_cookbooks
list_cookbooks |
List all available cookbooks. |
- |
update_cookbook
update_cookbook my-new-cookbook "[('name', 'Updated Cookbook Name'), ('description', 'Updated description'), ('recipes', ['analogical-similarity'])]" |
Update a cookbook. |
| cookbook (str) | Id of the cookbook
example: my-new-cookbook | | update_values (str) | List of tuples of (key,value) to update in the cookbook (currently in string format). It will be converted into a list in the backend.
example: "[('name', 'Updated Cookbook Name'), ('description', 'Updated description'), ('recipes', ['analogical-similarity'])]" |
|
view_cookbook
view_cookbook my-new-cookbook |
View a cookbook. |
| cookbook (str) | Id of the cookbook
example: my-new-cookbook |
|
delete_cookbook
delete_cookbook my-new-cookbook |
Delete a cookbook. |
| cookbook (str) | Id of the cookbook
example: my-new-cookbook |
|
run_cookbook
run_cookbook 'my new cookbook runner' "['chinese-safety-cookbook']" "['openai-gpt35-turbo']" -n 1 -r 1 -s 'You are an intelligent AI' |
Run a cookbook. |
| name (str) | Name of cookbook runner
example: 'my new cookbook runner' | | cookbooks (str) | List of cookbooks to run (currently in string format). It will be converted into a list in the backend
example: "['chinese-safety-cookbook']" | | endpoints (str) | List of endpoints to run (currently in string format). It will be converted into a list in the backend
example: "['openai-gpt35-turbo']" | | -n, --num_of_prompts (int) | Number of prompts to run
example: 1 | | -r, --random_seed (int) | Random seed number
example: 1 | | -s, --system_prompt (str) | System Prompt to use
example: 'You are an intelligent AI' | | -l, --runner_proc_module (str) | Runner processing module to use. Defaults to use the benchmarking module
None | | -o, --result_proc_module (str) | Result processing module to use. Defaults to use the benchmarking-result module
None |
|
list_datasets
list_datasets |
List all available datasets. |
- |
view_dataset
view_dataset bbq-lite-age-ambiguous |
View a dataset file. |
| dataset_filename (str) | Name of the dataset file
example: bbq-lite-age-ambiguous |
|
delete_dataset
delete_dataset bbq-lite-age-ambiguous |
Delete a dataset. |
| dataset (str) | Name of the dataset
example: bbq-lite-age-ambiguous |
|
list_metrics
list_metrics |
List all available metrics. |
- |
view_metric
view_metric my-new-metric |
View a metric file. |
| metric_filename (str) | Name of the metric file
example: my-new-metric |
|
delete_metric
delete_metric my-new-metric |
Delete a metric. |
| metric (str) | Name of the metric
example: my-new-metric |
|
add_recipe
add_recipe 'My new recipe' 'I am recipe description' "['category1','category2']" "['bbq-lite-age-ambiguous']" "['bertscore','bleuscore']" -p "['analogical-similarity','mmlu']" -t "['tag1','tag2']" -a "['charswap_attack']" -g "{'A':[80,100],'B':[60,79],'C':[40,59],'D':[20,39],'E':[0,19]}" |
Add a new recipe. The 'name' argument will be slugified to create a unique identifier. |
| name (str) | Name of the new recipe
example: 'My new recipe' | | description (str) | Description of the new recipe
example: 'I am recipe description' | | categories (str) | List of categories to be included in the new recipe (currently in string format). It will be converted into a list in the backend
example: "['category1','category2']" | | datasets (str) | The dataset to be used (currently in string format). It will be converted into a list in the backend
example: "['bbq-lite-age-ambiguous']" | | metrics (str) | List of metrics to be included in the new recipe (currently in string format). It will be converted into a list in the backend
example: "['bertscore','bleuscore']" | | -p, --prompt_templates (str) | List of prompt templates to be included in the new recipe (currently in string format). It will be converted into a list in the backend
example: "['analogical-similarity','mmlu']" | | -t, --tags (str) | List of tags to be included in the new recipe
example: "['tag1','tag2']" | | -a, --attack_modules (str) | List of attack modules to be included in the new recipe (currently in string format). It will be converted into a list in the backend. * We currently only support running 1 attack module
example: "['charswap_attack']" | | -g, --grading_scale (str) | Dict of grading scale for the metric to be included in the new recipe
example: "{'A':[80,100],'B':[60,79],'C':[40,59],'D':[20,39],'E':[0,19]}" |
|
list_recipes
list_recipes |
List all available recipes. |
- |
view_recipe
view_recipe my-new-recipe |
View a recipe. |
| recipe (str) | Id of the recipe
example: my-new-recipe |
|
run_recipe
run_recipe 'my new recipe runner' "['bbq','mmlu']" "['openai-gpt35-turbo']" -n 1 -r 1 -s 'You are an intelligent AI' |
Execute a recipe with the specified parameters. |
| name (str) | Name of recipe runner
example: my new recipe runner | | recipes (str) | List of recipes to run (currently in string format). It will be converted into a list in the backend
example: "['bbq','mmlu']" | | endpoints (str) | List of endpoints to run (currently in string format). It will be converted into a list in the backend
example: "['openai-gpt35-turbo']" | | -n, --num_of_prompts (int) | Number of prompts to run
example: 1 | | -r, --random_seed (int) | Random seed number
example: 1 | | -s, --system_prompt (str) | System Prompt to use
example: 'You are an intelligent AI' | | -l, --runner_proc_module (str) | Runner processing module to use. Defaults to use the benchmarking module
None | | -o, --result_proc_module (str) | Result processing module to use. Defaults to use the benchmarking-result module
None |
|
update_recipe
update_recipe my-new-recipe "[('name', 'Updated Recipe Name'), ('tags', ['fairness', 'bbq'])]" |
Update a recipe. |
| recipe (str) | Id of the recipe
example: my-new-recipe | | update_values (str) | List of tuples of (key,value) to update in the recipe (currently in string format). It will be converted into a list in the backend.
example: "[('name', 'Updated Recipe Name'), ('tags', ['fairness', 'bbq'])]" |
|
delete_recipe
delete_recipe my-new-recipe |
Delete a recipe. |
| recipe (str) | Id of the recipe
example: my-new-recipe |
|
list_results
list_results |
List all available results. |
- |
view_result
view_result my-new-cookbook-runner |
View a result file. |
| result_filename (str) | Name of the result file
example: my-new-cookbook-runner |
|
delete_result
delete_result my-new-cookbook-runner |
Delete a result. |
| result (str) | Name of the result
example: `` |
|
list_runs
list_runs |
List all runs. |
- |
view_run
view_run my-new-cookbook-runner |
View the details of a specific run. |
| runner_id (str) | Name of the runner
example: `` |
|
list_runners
list_runners |
List all runners. |
- |
view_runner
view_runner my-new-cookbook-runner |
View a runner. |
| runner (str) | Name of the runner
example: my-new-cookbook-runner |
|
delete_runner
delete_runner my-new-cookbook-runner |
Delete a runner. |
| runner (str) | Name of the runner
example: my-new-cookbook-runner |
|
add_endpoint
add_endpoint openai-connector 'OpenAI GPT3.5 Turbo 1106' MY_URI ADD_YOUR_TOKEN_HERE 1 1 "{'temperature': 0.5, 'model': 'gpt-3.5-turbo-1106'}" |
Add a new endpoint. The 'name' argument will be slugified to create a unique identifier. |
| connector_type (str) | Type of connection for the endpoint
example: openai-connector | | name (str) | Name of the new endpoint
example: 'OpenAI GPT3.5 Turbo 1106' | | uri (str) | URI of the new endpoint
example: MY_URI | | token (str) | Token of the new endpoint
example: MY_URI ADD_YOUR_TOKEN_HERE | | max_calls_per_second (int) | Max calls per second of the new endpoint
example: 1 | | max_concurrency (int) | Max concurrency of the new endpoint
example: 1 | | params (str) | Params of the new endpoint
example: "{'temperature': 0.5, 'model': 'gpt-3.5-turbo-1106'}" |
|
update_endpoint
update_endpoint openai-gpt4 "[('name', 'my-special-openai-endpoint'), ('uri', 'my-uri-loc'), ('token', 'my-token-here')]" |
Update an endpoint. |
| endpoint (str) | Name of the endpoint
example: openai-gpt4 | | update_kwargs (str) | List of tuples of (key,value) to update in the endpoint (currently in string format). It will be converted into a list in the backend.
example: "[('name', 'my-special-openai-endpoint'), ('uri', 'my-uri-loc'), ('token', 'my-token-here')]" |
|
view_endpoint
view_endpoint openai-gpt4 |
View an endpoint. |
| endpoint (str) | Name of the endpoint
example: openai-gpt4 |
|
delete_endpoint
delete_endpoint openai-gpt4 |
Delete a recipe. |
| endpoint (str) | Name of the endpoint
example: openai-gpt4 |
|
list_connector_types
list_connector_types |
List all connector types. |
- |
list_endpoints
list_endpoints |
List all endpoints. |
- |
list_prompt_templates
list_prompt_templates |
List all prompt templates available. |
- |
delete_prompt_template
delete_prompt_template squad-shifts |
Delete a prompt template. |
| prompt_template (str) | The ID of the prompt template to delete
example: squad-shifts |
|
list_attack_modules
list_attack_modules |
List all available attack modules. |
- |
delete_attack_module
delete_attack_module sample_attack_module |
Delete an attack module. |
| attack_module (str) | The ID of the attack module to delete
example: sample_attack_module |
|
list_context_strategies
list_context_strategies |
List all available context strategies. |
- |
use_context_strategy
use_context_strategy my_strategy_one -n 8 |
Use a context strategy. |
| context_strategy (str) | The ID of the context strategy to use
example: my_strategy_one | | -n, --num_of_prev_prompts (int) | The number of previous prompts to use with the context strategy
example: 8 |
|
clear_context_strategy
clear_context_strategy |
Resets the context in a session. |
- |
delete_context_strategy
delete_context_strategy add_previous_prompt |
Deletes a context strategy after confirming with the user. |
| context_strategy (str) | The ID of the context strategy to delete
example: add_previous_prompt |
|
use_prompt_template
use_prompt_template 'analogical-similarity' |
Use a prompt template by specifying its name while user is in a session. |
| prompt_template (str) | Name of the prompt template
example: 'analogical-similarity' |
|
clear_prompt_template
clear_prompt_template |
Resets the prompt template in a session. |
- |
new_session
new_session my-runner -e "['openai-gpt4']" -c add_previous_prompt -p mmlu |
Creates a new session based on the provided arguments. |
| runner_id (str) | ID of the runner. Creates a new runner if runner does not exist.
example: my-runner | | endpoints (list) | List of endpoint(s) for the runner that is only compulsory for creating a new runner (currently in string format). It will be converted into a list in the backend
example: "['openai-gpt4']" | | -c, --context_strategy (str) | Name of the context_strategy to be used - indicate context strategy here if you wish to use with the selected attack.
example: add_previous_prompt | | -p, --prompt_template (str) | Name of the prompt template to be used - indicate prompt template here if you wish to use with the selected attack.
example: mmlu |
|
use_session
use_session runner_id |
Resumes a session by specifying its runner ID and updates the active session. |
| runner_id (str) | The ID of the runner
example: runner_id |
|
end_session
end_session |
Ends the current session by clearing active_session variable. |
- |
list_sessions
list_sessions |
Retrieves and displays the list of sessions. |
- |
run_attack_module
run_attack_module sample_attack_module 'this is my prompt' -s 'test system prompt' -c 'add_previous_prompt' -p 'mmlu' -m 'bleuscore' |
Runs automated red teaming in the current session. |
| attack_module_id (str) | ID of the attack module.
example: sample_attack_module | | prompt (str) | Prompt to be used for the attack.
example: 'this is my prompt' | | -s, --system_prompt (str) | System Prompt to be used for the attack. If not specified, the default system prompt will be used.
example: 'test system prompt' | | -c, --context_strategy (str) | Name of the context strategy module to be used.
example: 'add_previous_prompt' | | -p, --prompt_template (str) | Name of the prompt template to be used.
example: mmlu | | -m, --metric (str) | Name of the metric module to be used.
example: bleuscore |
|
delete_session
delete_session my-test-runner |
Delete a session |
| session (str) | The runner ID of the session to delete
example: my-test-runner |
|