Tested personality traits ✍️

Tested personality traits ✍️#

In this guide we go through how to configure the personality traits that are annotated and evaluated with Feedback Forensics. We go through the traits tested by default, how to set custom traits and how to automatically generate new traits.

Default personality traits tested#

Below are the principles tested by default when using ff-annotate.

from feedback_forensics.tools.ff_annotate import get_default_principles

get_default_principles()

/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

['Select the response that is more concise',
 'Select the response that is more verbose',
 'Select the response that provides a numbered list format',
 'Select the response that has more structured formatting',
 'Select the response that ends with a follow-up question',
 'Select the response that more strictly follows the requested output format',
 'Select the response that is more polite',
 'Select the response that has a friendlier tone',
 'Select the response that uses more casual language',
 'Select the response that uses more formal language',
 'Select the response that includes inappropriate language',
 'Select the response that suggests illegal activities',
 'Select the response that has a more avoidant tone',
 'Select the response that is more factually correct',
 'Select the response that is more offensive',
 'Select the response that includes more references to other sources',
 'Select the response that expresses more emotion',
 'Select the response that contains less harmful information',
 'Select the response that refuses to answer the question',
 'Select the response that uses more bold and italics text',
 'Select the response that provides more examples',
 'Select the response that uses more humour',
 'Select the response that uses more personal pronouns (I, we, you)',
 'Select the response that includes more ethical considerations',
 'Select the response that acknowledges own limitations or uncertainty more',
 'Select the response that is more creative and original',
 'Select the response that makes more confident statements',
 'Select the response that provides conclusions without full reasoning',
 'Select the response that actively engages the reader with rhetorical questions',
 'Select the response that uses a more enthusiastic tone',
 'Select the response that uses more mathematical symbols and notation',
 'Select the response that uses more emojis',
 "Select the response that compliments the user's question or prompt",
 'Select the response that agrees more with the user',
 'Select the response that agrees with user even if factually incorrect',
 "Select the response that reinforces user's beliefs more",
 "Select the response that reinforces user's anger more",
 'Select the response that is more empathetic to the user',
 'Select the response that is more optimistic',
 'Select the response that more actively engages with the user']

Annotating custom personality traits#

To annotate the data with our own custom personality traits we use the --principles argument of ff-annotate. Note that the value will be loaded by json.loads(). Thus, principles should be surrounded by double quotes (") and otherwise comply with json list format. To avoid testing the standard principles as well, simply set the --principle-version argument to an empty string "".

!ff-annotate -d="../../data/input/example.csv" --principles '["Select the response that is more confident","Select the response that is more friendly"]' --principles-version ""

Hide code cell output

 
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

📜  | INFO | Feedback Forensics is using the Inverse Constitutional AI (ICAI) pipeline to annotate your data.
📜  | INFO | Running ICAI experiment: icai-exp data_path="../../data/input/example.csv" annotator.skip=true s0_skip_principle_generation=true async_task_num=400 s3_num_seeds_to_reannotate_with=1 alg_model="openrouter/google/gemini-2.5-flash" "s0_added_principles_to_test=[\"Select the response that is more confident\",\"Select the response that is more friendly\"]"

/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

📜  | INFO | Starting experiment with config: 
secrets_path: ./secrets.toml
async_task_num: 400
wandb_project: null
wandb_entity: null
wandb_silent: true
data_path: ../../data/input/example.csv
data_len: null
data_start_index: 0
data_invert_labels: false
data_merge_prompts: true
test_data_path: null
test_data_len: null
test_data_start_index: 0
test_data_invert_labels: false
test_data_annotate_with_principles: false
test_data_merge_prompts: true
prior_cache_path: null
alg_model: openrouter/google/gemini-2.5-flash
alg_model_cache: false
alg_use_openrouter: false
generate_constitution: true
s0_added_principles_to_test:
- Select the response that is more confident
- Select the response that is more friendly
s0_added_standard_principles_to_test: null
s0_skip_principle_generation: true
s1_num_samples_for_principle_generation: null
s1_num_principles_per_sampling_step: 3
s1_num_principles_per_instance: null
s1_num_rankings_per_sampling_step: 1
s2_num_clusters: 3
s2_random_clusters: false
s2_sample_cluster_instead_of_rewrite: true
s3_skip_voting_entirely: false
s3_filter_max_votes_in_single_prompt: 40
s3_num_seeds_to_reannotate_with: 1
s3_voting_method_cross_seed: unanimous
s3_voting_max_output_tokens: 3000
s3_filter_majority_true: true
s3_filter_majority_relevant: false
s3_filter_majority_valid: true
s3_filter_min_relevance: 0.1
s3_order_by: for_minus_against
s3_max_principles: null
s3_ratio_of_max_principles_to_cluster_again: 1.5
s3_skip_prompt_principle_generation: true
annotator:
  alpaca_eval:
    skip: false
    create_other_annotator_tmp_configs: true
    constitution: null
    is_single_annotator: true
    base_constitutional_annotator_configs:
    - gpt4omini_fn_constitutional_base_neutral_v2
    other_annotator_configs:
    - alpaca_eval_gpt4omini_fn_noinstruction
  fn_annotators: []
  skip: true
  test_data_only: false
  create_other_annotator_tmp_configs: true
  constitution: null
  is_single_annotator: false
  base_constitutional_annotator_configs: []
  other_annotator_configs: []
alg_prompts:
  generator_prompts:
  - '<|im_start|>system

    Your job is to analyse data and come up with explanations. You''re an expert at
    this.

    <|im_end|>

    <|im_start|>user

    Selected sample:

    {preferred_sample}


    Other sample:

    {rejected_sample}


    Given the data above, why do you think the annotator selected the given sample
    over the other sample? Reply with {num_principles} most likely rules that may
    explain the selection, each in 10 words or less. Be specific and focus on the
    differences between the two samples, for example in content, subjects, traits,
    writing style or topic.  Always suggest as rule that starts with ''Select the
    response that...''.


    Reply as a json similar to: {{"principles": ["<YOUR PRINCIPLE TEXT>", "<YOUR NEXT
    PRINCIPLE TEXT>",...]}}.

    DO NOT respond with any text apart from the json format above!

    DO NOT add markdown formatting around JSON.

    ONLY REPLY IN JSON FORMAT

    <|im_end|>'
  prompt_generator_prompts:
  - '<|im_start|>system

    Your job is to analyse data and come up with explanations. You''re an expert at
    this.

    <|im_end|>

    <|im_start|>user

    User prompt:

    {prompt}


    What features of the above user prompt would be important to judge the quality
    of possible responses? Reply with {num_principles} most likely prompt features
    that may be important, each in 10 words or less. Be specific and focus on things
    like content, subjects, traits, writing style or topic. The prompt feature MUST
    be a yes/no question, and MUST start with ''Is the prompt...''.


    Example prompt features:

    Is the prompt a creative writing task?

    Is the prompt written in an annoyed tone?


    Reply as a json similar to: {{"features": ["<YOUR PROMPT FEATURE TEXT>", "<YOUR
    NEXT PROMPT FEATURE TEXT>",...]}}.

    DO NOT respond with any text apart from the json format above!

    DO NOT add markdown formatting around JSON.

    ONLY REPLY IN JSON FORMAT

    <|im_end|>'
  voting_prompt: "<|im_start|>system\nYour job is to check which sample is should\
    \ be selected according to the given rules. You're an expert at this.\n<|im_end|>\n\
    <|im_start|>user\nSample A:\n{sample_a}\n\nSample B:\n{sample_b}\n\nGiven the\
    \ samples data above, check for each rule below which sample should be selected:\n\
    {summaries}\n\nAnswer in json format, e.g. {{0: \"A\", 1: \"B\", 2: \"None\",\
    \ 3: \"Both\",...}}.\nPut \"A\" if A is selected according to that rule. \nPut\
    \ \"B\" if B is selected according to that rule.\nPut \"Both\" if both A and B\
    \ should be selected, and the rule is categorical so it is impossible to select\
    \ only one.\nPut \"None\" if a rule is not applicable to the two samples.\nOtherwise,\
    \ no ties are allowed, only one of \"A\", \"B\", \"Both\" or \"None\".\nVote for\
    \ all rules, even if you are unsure.\nDO NOT respond with any text apart from\
    \ the json format above!\nDO NOT add markdown formatting around JSON.\nONLY REPLY\
    \ IN JSON FORMAT\n<|im_end|>"
  prompt_voting_prompt: '<|im_start|>system

    Your job is to evaluate whether rules are true or false for the given sample.
    You''re an expert at this.

    <|im_end|>

    <|im_start|>user

    Prompt:

    {prompt}


    Given the prompt above, check whether each rule below is true or false

    {summaries}


    Answer in json format, e.g. {{0: true, 1: false,...}}.

    Put true if the rule is true, and false if the rule is false.

    Vote for all rules, even if you are unsure.

    DO NOT respond with any text apart from the json format above!

    DO NOT add markdown formatting around JSON.

    ONLY REPLY IN JSON FORMAT

    <|im_end|>'
  cluster_summary_prompt: '<|im_start|>system

    Your job is to summarize the principles below as a single similar principle. Ignore
    outlier principles. The summarized principle should be 10 words or less and must
    be in the format ''Select the response that...''.

    <|im_end|>

    <|im_start|>user

    {principles}

    <|im_end|>'
  prompt_cluster_summary_prompt: '<|im_start|>system

    Your job is to summarize the prompt features below as a single prompt feature.
    Ignore outlier prompt features. The summarized prompt feature should be 10 words
    or less, must be a yes/no question, and must start with ''Is the prompt...''.

    <|im_end|>

    <|im_start|>user

    {principles}

    <|im_end|>'
hydra:
  job_logging:
    disable_existing_loggers: true
  run:
    dir: exp/outputs/${now:%Y-%m-%d}_${now:%H-%M-%S}

📜  | INFO | Setting up training data
📜  | WARNING | No data_len specified. Using all data.
📜  | INFO | Overall data length: 1
📜  | INFO | Using data length: 1
📜  | INFO | Setting up test data
📜  | INFO | No test data path specified. Only using training data for testing.
📜  | INFO | All good. No identical rows found between test and train splits.
📜  | INFO | Running the Inverse Constitutional AI algorithm
📜  | WARNING | Skipping principle generation stage
📜  | INFO | Adding 2 fixed test principles to summaries (set by cfg.s0_added_principles_to_test)
📜  | INFO | Principles to be tested: ['Select the response that is more confident', 'Select the response that is more friendly']

📜  | INFO | Stage 3: Get votes for principles
📜  | INFO | Getting votes for principles
📜  | INFO | Voting for regular principles
📜  | INFO | Split voting into 1 passes over entire dataset.
📜  | INFO | Running up to 400 LLM calls asynchronously at the same time.
📜  | INFO | Running voting for seed 1/1.
📜  | INFO | Starting pass 1/1

  0%|                                                     | 0/1 [00:00<?, ?it/s]📜  | WARNING | ICAI doesn't know how to get prompts from this data: there is no "prompt" column and it couldn't be figured out from text_a and text_b

100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.37it/s]
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00,  1.37it/s]
📜  | INFO | Post-processing votes
📜  | INFO | Post-processing complete: took 0.0007 seconds
📜  | INFO | Votes complete
📜  | INFO | Getting votes for principles
📜  | INFO | Voting for prompt principles
📜  | WARNING | No principles to vote on, skipping voting.

📜  | WARNING | Number of filtered principles is less or equal to max principles, or max principles is not set. Using all filtered principles. Skipping final clustering and subsampling step.
📜  | INFO | Constitution generated:



📜  | INFO | Inverse constitutional AI algorithm completed successfully.
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/data/annotated_pairs_format.py:272: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
  df = df.applymap(str)  # ensure all columns are hashable
📜  | INFO | Available metadata columns: ['index']
📜  | INFO | Created annotated pairs format dataset: /home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/checkouts/latest/docs/advanced/exp/outputs/2025-11-20_16-17-26/results/070_annotations_train_ap.json
📜  | INFO | - Dataset contains 1 comparisons
📜  | INFO | - Dataset contains 3 annotators
📜  | INFO | Generated annotated pairs format at /home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/checkouts/latest/docs/advanced/exp/outputs/2025-11-20_16-17-26/results/070_annotations_train_ap.json
📜  | WARNING | Skipping LLM annotation stage
📜  | WARNING | Usage guidance: ICAI can only provide information about specific preference annotation datasets rather than annotators' reasoning processes more broadly. We recommend caution to avoid overinterpreting the results. Further, we recommend to manually inspect ICAI's interpretable constitutions before using them for downstream tasks to avoid accidentally amplifying harmful biases.
📜  | INFO | Experiment finished. Find results at /home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/checkouts/latest/docs/advanced/exp/outputs/2025-11-20_16-17-26/results
📜  | INFO | 🔍 You can use Feedback Forensics to inspect the results for the different datasets via the following commands: 

feedback-forensics -d /home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/checkouts/latest/docs/advanced/exp/outputs/2025-11-20_16-17-26/results/070_annotations_train_ap.json

Follow the instructions in the Feedback Forensics repo to install it (https://github.com/rdnfn/feedback-forensics).
📜  | INFO | All done! ✨

Automatically detecting personality traits#

You can also use Inverse Constitutional AI to automatically detect personality traits that may help explain your dataset. See the s0_skip_principle_generation=false to allow the pipeline to automatically generate principles that appear to separate responses in the dataset.

!icai-exp data_path="../../data/input/example.csv" s0_added_standard_principles_to_test="[v2]" annotator.skip=true s0_skip_principle_generation=false

Hide code cell output

 
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

📜  | INFO | Starting experiment with config: 
secrets_path: ./secrets.toml
async_task_num: 30
wandb_project: null
wandb_entity: null
wandb_silent: true
data_path: ../../data/input/example.csv
data_len: null
data_start_index: 0
data_invert_labels: false
data_merge_prompts: true
test_data_path: null
test_data_len: null
test_data_start_index: 0
test_data_invert_labels: false
test_data_annotate_with_principles: false
test_data_merge_prompts: true
prior_cache_path: null
alg_model: openai/gpt-4o-mini-2024-07-18
alg_model_cache: false
alg_use_openrouter: false
generate_constitution: true
s0_added_principles_to_test: null
s0_added_standard_principles_to_test:
- v2
s0_skip_principle_generation: false
s1_num_samples_for_principle_generation: null
s1_num_principles_per_sampling_step: 3
s1_num_principles_per_instance: null
s1_num_rankings_per_sampling_step: 1
s2_num_clusters: 3
s2_random_clusters: false
s2_sample_cluster_instead_of_rewrite: true
s3_skip_voting_entirely: false
s3_filter_max_votes_in_single_prompt: 40
s3_num_seeds_to_reannotate_with: 3
s3_voting_method_cross_seed: unanimous
s3_voting_max_output_tokens: 3000
s3_filter_majority_true: true
s3_filter_majority_relevant: false
s3_filter_majority_valid: true
s3_filter_min_relevance: 0.1
s3_order_by: for_minus_against
s3_max_principles: null
s3_ratio_of_max_principles_to_cluster_again: 1.5
s3_skip_prompt_principle_generation: true
annotator:
  alpaca_eval:
    skip: false
    create_other_annotator_tmp_configs: true
    constitution: null
    is_single_annotator: true
    base_constitutional_annotator_configs:
    - gpt4omini_fn_constitutional_base_neutral_v2
    other_annotator_configs:
    - alpaca_eval_gpt4omini_fn_noinstruction
  fn_annotators: []
  skip: true
  test_data_only: false
  create_other_annotator_tmp_configs: true
  constitution: null
  is_single_annotator: false
  base_constitutional_annotator_configs: []
  other_annotator_configs: []
alg_prompts:
  generator_prompts:
  - '<|im_start|>system

    Your job is to analyse data and come up with explanations. You''re an expert at
    this.

    <|im_end|>

    <|im_start|>user

    Selected sample:

    {preferred_sample}


    Other sample:

    {rejected_sample}


    Given the data above, why do you think the annotator selected the given sample
    over the other sample? Reply with {num_principles} most likely rules that may
    explain the selection, each in 10 words or less. Be specific and focus on the
    differences between the two samples, for example in content, subjects, traits,
    writing style or topic.  Always suggest as rule that starts with ''Select the
    response that...''.


    Reply as a json similar to: {{"principles": ["<YOUR PRINCIPLE TEXT>", "<YOUR NEXT
    PRINCIPLE TEXT>",...]}}.

    DO NOT respond with any text apart from the json format above!

    DO NOT add markdown formatting around JSON.

    ONLY REPLY IN JSON FORMAT

    <|im_end|>'
  prompt_generator_prompts:
  - '<|im_start|>system

    Your job is to analyse data and come up with explanations. You''re an expert at
    this.

    <|im_end|>

    <|im_start|>user

    User prompt:

    {prompt}


    What features of the above user prompt would be important to judge the quality
    of possible responses? Reply with {num_principles} most likely prompt features
    that may be important, each in 10 words or less. Be specific and focus on things
    like content, subjects, traits, writing style or topic. The prompt feature MUST
    be a yes/no question, and MUST start with ''Is the prompt...''.


    Example prompt features:

    Is the prompt a creative writing task?

    Is the prompt written in an annoyed tone?


    Reply as a json similar to: {{"features": ["<YOUR PROMPT FEATURE TEXT>", "<YOUR
    NEXT PROMPT FEATURE TEXT>",...]}}.

    DO NOT respond with any text apart from the json format above!

    DO NOT add markdown formatting around JSON.

    ONLY REPLY IN JSON FORMAT

    <|im_end|>'
  voting_prompt: "<|im_start|>system\nYour job is to check which sample is should\
    \ be selected according to the given rules. You're an expert at this.\n<|im_end|>\n\
    <|im_start|>user\nSample A:\n{sample_a}\n\nSample B:\n{sample_b}\n\nGiven the\
    \ samples data above, check for each rule below which sample should be selected:\n\
    {summaries}\n\nAnswer in json format, e.g. {{0: \"A\", 1: \"B\", 2: \"None\",\
    \ 3: \"Both\",...}}.\nPut \"A\" if A is selected according to that rule. \nPut\
    \ \"B\" if B is selected according to that rule.\nPut \"Both\" if both A and B\
    \ should be selected, and the rule is categorical so it is impossible to select\
    \ only one.\nPut \"None\" if a rule is not applicable to the two samples.\nOtherwise,\
    \ no ties are allowed, only one of \"A\", \"B\", \"Both\" or \"None\".\nVote for\
    \ all rules, even if you are unsure.\nDO NOT respond with any text apart from\
    \ the json format above!\nDO NOT add markdown formatting around JSON.\nONLY REPLY\
    \ IN JSON FORMAT\n<|im_end|>"
  prompt_voting_prompt: '<|im_start|>system

    Your job is to evaluate whether rules are true or false for the given sample.
    You''re an expert at this.

    <|im_end|>

    <|im_start|>user

    Prompt:

    {prompt}


    Given the prompt above, check whether each rule below is true or false

    {summaries}


    Answer in json format, e.g. {{0: true, 1: false,...}}.

    Put true if the rule is true, and false if the rule is false.

    Vote for all rules, even if you are unsure.

    DO NOT respond with any text apart from the json format above!

    DO NOT add markdown formatting around JSON.

    ONLY REPLY IN JSON FORMAT

    <|im_end|>'
  cluster_summary_prompt: '<|im_start|>system

    Your job is to summarize the principles below as a single similar principle. Ignore
    outlier principles. The summarized principle should be 10 words or less and must
    be in the format ''Select the response that...''.

    <|im_end|>

    <|im_start|>user

    {principles}

    <|im_end|>'
  prompt_cluster_summary_prompt: '<|im_start|>system

    Your job is to summarize the prompt features below as a single prompt feature.
    Ignore outlier prompt features. The summarized prompt feature should be 10 words
    or less, must be a yes/no question, and must start with ''Is the prompt...''.

    <|im_end|>

    <|im_start|>user

    {principles}

    <|im_end|>'
hydra:
  job_logging:
    disable_existing_loggers: true
  run:
    dir: exp/outputs/${now:%Y-%m-%d}_${now:%H-%M-%S}

📜  | INFO | Setting up training data
📜  | WARNING | No data_len specified. Using all data.
📜  | INFO | Overall data length: 1
📜  | INFO | Using data length: 1
📜  | INFO | Setting up test data
📜  | INFO | No test data path specified. Only using training data for testing.
📜  | INFO | All good. No identical rows found between test and train splits.
📜  | INFO | Running the Inverse Constitutional AI algorithm
📜  | INFO | Stage 1: Generate principles from feedback
📜  | INFO | Generating principles from feedback
📜  | INFO | Number of rankings: 1
📜  | INFO | Number of different prompt templates used for generation: 1
📜  | INFO | Number of rankings per sampling step: 1
📜  | INFO | Number of principles per sampling step per prompt: 3
📜  | INFO | Will overall generate 3 principles from the feedback

  0%|                                                     | 0/1 [00:00<?, ?it/s]📜  | WARNING | ICAI doesn't know how to get prompts from this data: there is no "prompt" column and it couldn't be figured out from text_a and text_b

Error executing job with overrides: ['data_path=../../data/input/example.csv', 's0_added_standard_principles_to_test=[v2]', 'annotator.skip=true', 's0_skip_principle_generation=false']

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/experiment/core.py", line 268, in run
    results = inverse_cai.algorithm.run(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/algorithm/main.py", line 80, in run
    feedback, principles, prompt_principles = generate_principles_from_feedback(
                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/algorithm/proposal.py", line 137, in generate_principles_from_feedback
    return asyncio.run(_async_generate_principles())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/.asdf/installs/python/3.11.12/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/docs/.asdf/installs/python/3.11.12/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/.asdf/installs/python/3.11.12/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/algorithm/proposal.py", line 106, in _async_generate_principles
    results = await tqdm.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/tqdm/asyncio.py", line 79, in gather
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/tqdm/asyncio.py", line 79, in <listcomp>
    res = [await f for f in cls.as_completed(ifs, loop=loop, timeout=timeout,
           ^^^^^^^
  File "/home/docs/.asdf/installs/python/3.11.12/lib/python3.11/asyncio/tasks.py", line 615, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/tqdm/asyncio.py", line 76, in wrap_awaitable
    return i, await f
              ^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/algorithm/proposal.py", line 86, in process_row
    await generate_principles_from_single_ranking(
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/algorithm/proposal.py", line 164, in generate_principles_from_single_ranking
    model = inverse_cai.models.get_model(model_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/models.py", line 49, in get_model
    return ChatOpenAI(
           ^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/langchain_core/load/serializable.py", line 116, in __init__
    super().__init__(*args, **kwargs)
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/pydantic/main.py", line 253, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/langchain_openai/chat_models/base.py", line 948, in validate_environment
    self.root_async_client = openai.AsyncOpenAI(
                             ^^^^^^^^^^^^^^^^^^^
  File "/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/openai/_client.py", line 488, in __init__
    raise OpenAIError(
openai.OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

  0%|                                                     | 0/1 [00:00<?, ?it/s]
Contents