Command Line Interface

Command Line Interface#

feedback-forensics#

Main CLI command for launching the analysis interface.

!feedback-forensics --help

/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

usage: feedback-forensics [-h] [-d DATAPATH] [--dir DIR] [--load-web-datasets]

Launch the Feedback Forensics visualisation app.

options:
  -h, --help            show this help message and exit
  -d DATAPATH, --datapath DATAPATH
                        Path to AnnotatedPairs dataset to analyse. Can be used
                        multiple times (e.g. -d path1 -d path2).
  --dir DIR             Path to directory containing AnnotatedPairs datasets
                        to analyse. Will be recursively searched for .json
                        files. Can be used multiple times (e.g. --dir dirpath1
                        --dir dirpath2).
  --load-web-datasets   Load datasets from HuggingFace. Does not require
                        HF_TOKEN to be set.

ff-annotate#

Command for annotating datasets with ICAI.

!ff-annotate --help

/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

usage: ff-annotate [-h] -d DATAPATH [-v PRINCIPLES_VERSION] [-p PRINCIPLES]
                   [-o OUTPUT_DIR] [-m MODEL]

CLI command for annotating datasets for Feedback Forensics with ICAI pipeline.

options:
  -h, --help            show this help message and exit
  -d DATAPATH, --datapath DATAPATH
                        Path to AnnotatedPairs JSON file or CSV file with
                        columns text_a, text_b, and preferred_text.
  -v PRINCIPLES_VERSION, --principles-version PRINCIPLES_VERSION
                        Version of standard principles to test (default: v1).
                        A principle is an instruction for the AI annotator to
                        select a text according to a given personality trait.
  -p PRINCIPLES, --principles PRINCIPLES
                        List of custom principles to test (default: None).
                        E.g. "['Select the response that is more confident',
                        'Select the response that is more friendly']"
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Path to output directory (default:
                        exp/outputs/DATETIME)
  -m MODEL, --model MODEL
                        Model to use to annotate the data (default:
                        openrouter/google/gemini-2.5-flash)

ff-data#

Command for data operations.

!ff-data --help

/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

usage: ff-data [-h] {merge,csv_to_ap} ...

Swiss army knife for AnnotatedPairs datasets

positional arguments:
  {merge,csv_to_ap}  Available commands
    merge            Merge two AnnotatedPairs datasets
    csv_to_ap        Convert CSV to AnnotatedPairs format

options:
  -h, --help         show this help message and exit

!ff-data csv_to_ap --help

/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

usage: ff-data csv_to_ap [-h] --name NAME csv_file output

positional arguments:
  csv_file     Input CSV file with columns text_a, text_b, preferred_text
  output       Output AnnotatedPairs JSON file (use "-" for stdout)

options:
  -h, --help   show this help message and exit
  --name NAME  Dataset name for the AnnotatedPairs output

!ff-data merge --help

/home/docs/checkouts/readthedocs.org/user_builds/feedback-forensics/envs/latest/lib/python3.11/site-packages/alpaca_eval/utils.py:20: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources

usage: ff-data merge [-h] [--name NAME] [--desc DESC] first second output

positional arguments:
  first        First dataset file (takes precedence in conflicts)
  second       Second dataset file
  output       Output file (use "-" for stdout)

options:
  -h, --help   show this help message and exit
  --name NAME  Override dataset name for merged result
  --desc DESC  Override description for merged result
Contents