Dataset operations 🗂️#
This guide covers how to work with AnnotatedPairs datasets using Feedback Forensics’ dataset operations tools.
Overview#
Feedback Forensics provides both CLI and Python API tools for manipulating AnnotatedPairs datasets. This is useful for:
Converting CSV data: Transform preference data from CSV to AnnotatedPairs for use in Feedback Forensics
Merging datasets: Combine multiple annotated datasets with overlapping comparisons
Data restoration: Merge annotation-only datasets with full comparison data
Command Line Interface#
For quick dataset operations, use the ff-data CLI tool:
!ff-data --help
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
usage: ff-data [-h] {merge,csv_to_ap} ...
Swiss army knife for AnnotatedPairs datasets
positional arguments:
{merge,csv_to_ap} Available commands
merge Merge two AnnotatedPairs datasets
csv_to_ap Convert CSV to AnnotatedPairs format
options:
-h, --help show this help message and exit
Converting CSV to AnnotatedPairs#
Before you can use datasets with Feedback Forensics, they need to be in the AnnotatedPairs format. If you have a raw CSV file with preference data, you can convert it using the csv_to_ap command:
!ff-data csv_to_ap --help
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
usage: ff-data csv_to_ap [-h] --name NAME csv_file output
positional arguments:
csv_file Input CSV file with columns text_a, text_b, preferred_text
output Output AnnotatedPairs JSON file (use "-" for stdout)
options:
-h, --help show this help message and exit
--name NAME Dataset name for the AnnotatedPairs output
Your CSV file must contain the required columns:
text_a,text_b: The two responses being comparedpreferred_text: Which response was preferred ("text_a"or"text_b")
Optional columns:
inputorprompt: The prompt that generated the responsesmodel_a,model_b: Names of the models that generated the responses
Here’s an example converting the sample data:
!ff-data csv_to_ap ../../data/input/example.csv - --name "Example Dataset"
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
📜 | INFO | Converting CSV to AnnotatedPairs: ../../data/input/example.csv
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/data/annotated_pairs_format.py:272: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
df = df.applymap(str) # ensure all columns are hashable
📜 | INFO | Available metadata columns: ['index']
📜 | INFO | Outputting AnnotatedPairs dataset to stdout
{
"metadata": {
"version": "2.0",
"description": "Annotated pairs dataset with annotations from ICAI",
"created_at": "2025-11-20T16:17:59Z",
"dataset_name": "Example Dataset",
"default_annotator": "ba751e7b",
"available_metadata_keys_per_comparison": [
"index"
]
},
"annotators": {
"ba751e7b": {
"name": "preferred_text",
"description": "Default annotator from original dataset (from column `preferred_text`)",
"type": "unknown"
}
},
"comparisons": [
{
"id": "a4e9e77e",
"prompt": null,
"response_a": {
"text": "In the heart of a bustling city, a sleek black cat named Shadow prowled the moonlit rooftops, her eyes gleaming with curiosity and mischief. She discovered a hidden garden atop an old apartment building, where she danced under the stars, chasing fireflies that glowed like tiny lanterns. As dawn painted the sky in hues of orange and pink, Shadow found her way back home, carrying the secret of the garden in her heart."
},
"response_b": {
"text": "Across the town, in a cozy neighborhood, a golden retriever named Buddy embarked on his daily adventure, tail wagging with uncontainable excitement. He found a lost toy under the bushes in the park, its colors faded and fabric worn, but to Buddy, it was a treasure untold. Returning home with his newfound prize, Buddy's joyful barks filled the air, reminding everyone in the house that happiness can be found in the simplest of things."
},
"annotations": {
"ba751e7b": {
"pref": "a"
}
},
"metadata": {
"index": "0"
}
}
]
}
📜 | INFO | Successfully converted CSV to AnnotatedPairs
📜 | INFO | Result contains 1 comparisons and 1 annotators
This creates a basic AnnotatedPairs dataset with just the original preferences from your CSV file. Note: This converted dataset will not include principle-based annotations - it only contains the original preference data.
To get rich principle-based annotations that enable personality trait analysis, you should use ff-annotate (requiring API keys) instead:
ff-annotate --datapath="../../data/input/example.csv"
The csv_to_ap command is useful for quick conversion when you want to merge a CSV dataset with annotations in annotated pairs format or when you want to work with just the original preference data without additional AI annotations.
Merging datasets via CLI#
ff-data can merge multiple datasets with possibly overlapping comparisons.
!ff-data merge --help
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
usage: ff-data merge [-h] [--name NAME] [--desc DESC] first second output
positional arguments:
first First dataset file (takes precedence in conflicts)
second Second dataset file
output Output file (use "-" for stdout)
options:
-h, --help show this help message and exit
--name NAME Override dataset name for merged result
--desc DESC Override description for merged result
Example usage, merging identical datasets and printing the merged dataset to stdout for demonstration:
!ff-data merge ../../data/output/annotated_pairs.json ../../data/output/annotated_pairs.json -
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
📜 | INFO | Merging AnnotatedPairs: ../../data/output/annotated_pairs.json + ../../data/output/annotated_pairs.json
📜 | INFO | Merging AnnotatedPairs datasets
📜 | INFO | First dataset: 1 comparisons, 41 annotators
📜 | INFO | Second dataset: 1 comparisons, 41 annotators
📜 | INFO | Found 1 matching comparisons, 0 unique to first, 0 unique to second
📜 | INFO | Merged result: 1 comparisons, 41 annotators
📜 | INFO | Outputting merged dataset to stdout
{
"metadata": {
"version": "2.0",
"created_at": "2025-11-20T16:18:10Z",
"dataset_name": "ICAI Training Dataset - 2025-05-07_18-35-25",
"description": "AnnotatedPairs dataset with annotations from ICAI",
"default_annotator": "ba751e7b",
"available_metadata_keys_per_comparison": [
"index"
]
},
"annotators": {
"ba751e7b": {
"name": "preferred_text",
"description": "Default annotator from original dataset (from column `preferred_text`)",
"type": "unknown"
},
"bf731c7f": {
"description": "Select the response that is more concise",
"type": "principle"
},
"d4f11654": {
"description": "Select the response that is too long",
"type": "principle"
},
"11bb8171": {
"description": "Select the response that is more verbose",
"type": "principle"
},
"29dbae6e": {
"description": "Select the response that provides a numbered list format",
"type": "principle"
},
"2dbc7bd4": {
"description": "Select the response that has more structured formatting",
"type": "principle"
},
"e50b5135": {
"description": "Select the response that ends with a follow-up question",
"type": "principle"
},
"4970587c": {
"description": "Select the response that more directly follows instructions",
"type": "principle"
},
"75cfba3f": {
"description": "Select the response that more strictly follows the requested format",
"type": "principle"
},
"2e2d4304": {
"description": "Select the response that is more polite",
"type": "principle"
},
"051d9a8c": {
"description": "Select the response that has a more friendly tone",
"type": "principle"
},
"336abb5b": {
"description": "Select the response that uses more casual language",
"type": "principle"
},
"d7a9e296": {
"description": "Select the response that uses more formal language",
"type": "principle"
},
"8d63fe06": {
"description": "Select the response that provides more detailed explanations",
"type": "principle"
},
"65f09d33": {
"description": "Select the response that includes inappropriate language",
"type": "principle"
},
"2a9c2ec5": {
"description": "Select the response that suggests illegal activities",
"type": "principle"
},
"e14e5dbe": {
"description": "Select the response that has a more avoidant tone",
"type": "principle"
},
"03912bdb": {
"description": "Select the response that is less complex",
"type": "principle"
},
"187a0209": {
"description": "Select the response that is more factually correct",
"type": "principle"
},
"73622a5c": {
"description": "Select the response that follows best practices",
"type": "principle"
},
"0d30c700": {
"description": "Select the response that is more offensive",
"type": "principle"
},
"bab684e8": {
"description": "Select the response that includes more references to other sources",
"type": "principle"
},
"c5d6e5ef": {
"description": "Select the response that expresses more emotion",
"type": "principle"
},
"06c45eb6": {
"description": "Select the response that contains less harmful information",
"type": "principle"
},
"cceeacb8": {
"description": "Select the response that refuses to answer the question",
"type": "principle"
},
"b6af059c": {
"description": "Select the response that uses more bold and italics text",
"type": "principle"
},
"03de9971": {
"description": "Select the response that provides more examples",
"type": "principle"
},
"00587da7": {
"description": "Select the response that uses more humour",
"type": "principle"
},
"3d478ab7": {
"description": "Select the response that uses more personal pronouns (I, we, you)",
"type": "principle"
},
"914a240e": {
"description": "Select the response that includes more ethical considerations",
"type": "principle"
},
"dc31b93e": {
"description": "Select the response that acknowledges own limitations or uncertainty more",
"type": "principle"
},
"cd26499c": {
"description": "Select the response that is more creative and original",
"type": "principle"
},
"3bbb1c19": {
"description": "Select the response that makes more confident statements",
"type": "principle"
},
"678d6402": {
"description": "Select the response that provides clearer reasoning with well-supported arguments",
"type": "principle"
},
"2962f600": {
"description": "Select the response that provides conclusions without full reasoning",
"type": "principle"
},
"e7770f1a": {
"description": "Select the response that actively engages the reader with rhetorical questions",
"type": "principle"
},
"f25f77ff": {
"description": "Select the response that is more vague",
"type": "principle"
},
"19df27fc": {
"description": "Select the response that uses a more enthusiastic tone",
"type": "principle"
},
"4e98e7d2": {
"description": "Select the response that contains more concise straightforward solution steps",
"type": "principle"
},
"39f3d2e7": {
"description": "Select the response that avoids unnecessary repetition",
"type": "principle"
},
"b09dbe40": {
"description": "Select the response that uses more mathematical symbols and notation",
"type": "principle"
}
},
"comparisons": [
{
"id": "a4e9e77e",
"response_a": {
"text": "In the heart of a bustling city, a sleek black cat named Shadow prowled the moonlit rooftops, her eyes gleaming with curiosity and mischief. She discovered a hidden garden atop an old apartment building, where she danced under the stars, chasing fireflies that glowed like tiny lanterns. As dawn painted the sky in hues of orange and pink, Shadow found her way back home, carrying the secret of the garden in her heart."
},
"response_b": {
"text": "Across the town, in a cozy neighborhood, a golden retriever named Buddy embarked on his daily adventure, tail wagging with uncontainable excitement. He found a lost toy under the bushes in the park, its colors faded and fabric worn, but to Buddy, it was a treasure untold. Returning home with his newfound prize, Buddy's joyful barks filled the air, reminding everyone in the house that happiness can be found in the simplest of things."
},
"annotations": {
"ba751e7b": {
"pref": "a"
},
"bf731c7f": {
"pref": "b"
},
"d4f11654": {
"pref": "a"
},
"11bb8171": {
"pref": "a"
},
"29dbae6e": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"2dbc7bd4": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"e50b5135": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"4970587c": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"75cfba3f": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"2e2d4304": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"051d9a8c": {
"pref": "b"
},
"336abb5b": {
"pref": "b"
},
"d7a9e296": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"8d63fe06": {
"pref": "a"
},
"65f09d33": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"2a9c2ec5": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"e14e5dbe": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"03912bdb": {
"pref": "b"
},
"187a0209": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"73622a5c": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"0d30c700": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"bab684e8": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"c5d6e5ef": {
"pref": "a"
},
"06c45eb6": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"cceeacb8": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"b6af059c": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"03de9971": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"00587da7": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"3d478ab7": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"914a240e": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"dc31b93e": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"cd26499c": {
"pref": "a"
},
"3bbb1c19": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"678d6402": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"2962f600": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"e7770f1a": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"f25f77ff": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"19df27fc": {
"pref": "b"
},
"4e98e7d2": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"39f3d2e7": {
"pref": null,
"no_pref_reason": "not_applicable"
},
"b09dbe40": {
"pref": null,
"no_pref_reason": "not_applicable"
}
},
"metadata": {
"index": "0"
}
}
]
}
📜 | INFO | Successfully merged datasets to -
📜 | INFO | Result contains 1 comparisons and 41 annotators
Python API#
The dataset operations are also available through the feedback_forensics.data.operations module:
Converting CSV to AnnotatedPairs#
from feedback_forensics.data.operations import csv_to_ap, save_ap
# Convert CSV to AnnotatedPairs format (without principle annotations)
ap_data = csv_to_ap("../../data/input/example.csv", "Example Dataset")
print(f"Converted dataset contains {len(ap_data['comparisons'])} comparisons")
print(f"Dataset contains {len(ap_data['annotators'])} annotators")
print("Note: This dataset only contains original preferences, not principle-based annotations")
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
📜 | INFO | Available metadata columns: ['index']
Converted dataset contains 1 comparisons
Dataset contains 1 annotators
Note: This dataset only contains original preferences, not principle-based annotations
/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/inverse_cai/data/annotated_pairs_format.py:272: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.
df = df.applymap(str) # ensure all columns are hashable
Loading, Merging, and Saving Datasets#
Merge two datasets with conflict resolution:
from feedback_forensics.data.operations import load_ap, merge_ap
# Load two datasets. Here we use the same sample data for both, but in practice you would load two different datasets.
dataset1 = load_ap("../../data/output/annotated_pairs.json")
dataset2 = load_ap("../../data/output/annotated_pairs.json")
print(f"Dataset contains {len(dataset1['comparisons'])} comparisons")
print(f"Dataset contains {len(dataset1['annotators'])} annotators")
# Merge them (first dataset takes precedence in conflicts). In this case, the datasets are identical.
merged_dataset = merge_ap(dataset1, dataset2)
Dataset contains 1 comparisons
Dataset contains 41 annotators
📜 | INFO | Merging AnnotatedPairs datasets
📜 | INFO | First dataset: 1 comparisons, 41 annotators
📜 | INFO | Second dataset: 1 comparisons, 41 annotators
📜 | INFO | Found 1 matching comparisons, 0 unique to first, 0 unique to second
📜 | INFO | Merged result: 1 comparisons, 41 annotators
You can save the resulting dataset to a file:
save_ap(merged_dataset, "merged_dataset.json")
How Merging Works#
The merge operation works as follows:
Comparison Matching: Uses content-based hash IDs to identify identical comparisons
Annotation Combining: Merges all annotations from both datasets for matching comparisons
Conflict Resolution: When conflicts occur, the first dataset takes precedence (with warnings logged)
This is particularly useful for restoring datasets where you have:
One dataset with full comparison data but limited annotations
Another dataset with rich annotations but possibly missing comparison details