Issues

• ClassificationAccuracy scores every correct answer as 0 when the dataset's labels contain uppercase characters

  aws/fmeval#354 · Filter by author shaurya416 opened on Sep 23, 2026

• Community package: fmeval-openeval-adapter — export datasets/results to a portable interchange format

  aws/fmeval#353 · Filter by author adhabnr-ux opened on Sep 12, 2026

• Cannot load dataset from s3

  aws/fmeval#338 · Filter by author AmrSaber opened on Jan 7, 2025
  2

• Problem running package with python 3.13

  aws/fmeval#337 · Filter by author AmrSaber opened on Jan 6, 2025
  2

• [Feature] Increase model coverage of the BERT Score metric by adding torchmetrics implementation

  aws/fmeval#332 · Filter by author achad4 opened on Nov 8, 2024
  2

  

• [Feature] Lambda model runner

  aws/fmeval#331 · Filter by author achad4 opened on Nov 8, 2024
  1

• Option to disable BERTScore in QAAccuracy

  aws/fmeval#330 · Filter by author athewsey opened on Oct 28, 2024
  3

• [Feature] Add callback mechanism to evaluation

  aws/fmeval#284 · Filter by author acere opened on May 29, 2024
  1

• [Feature] Add system metrics collected during evaluation to eval_output

  aws/fmeval#280 · Filter by author acere opened on May 24, 2024
  4

• [Feature] Streaming results/progress and summary metrics for faster feedback

  aws/fmeval#278 · Filter by author athewsey opened on May 17, 2024
  1

• [Feature] EvalAlgorithmInterface.evaluate should accept a list of DataConfigs for consistency

  aws/fmeval#269 · Filter by author athewsey opened on May 3, 2024
  1

• [Feature] JSON export/import for EvalOutput classes

  aws/fmeval#268 · Filter by author athewsey opened on May 3, 2024
  3

• [Feature] Image fields for multi-modal models

  aws/fmeval#247 · Filter by author athewsey opened on Apr 9, 2024
  1

• [Feature] Multi-variable prompt templates

  aws/fmeval#246 · Filter by author athewsey opened on Apr 9, 2024
  2

• [Feature] Support string output path in EvalAlgorithmInterface.evaluate(save="...")

  aws/fmeval#165 · Filter by author athewsey opened on Jan 9, 2024
  3

• [Feature] LLM-based (QA Accuracy) eval algorithm

  aws/fmeval#163 · Filter by author athewsey opened on Jan 9, 2024
  2