answer-equivalence-dataset

KO | EN

gitlite — search

#javascript #hacktoberfest #react #typescript #llm #android #rust #deep-learning #nodejs #windows #cli #ios #mcp #open-source

answer-equivalence-dataset

(★ 30)

This dataset contains human judgements about answer equivalence. The data is based on SQuAD (Stanford Question Answering Dataset), and contains 9k human judgements of answer candidates generated by Albert on the SQuAD train set, and an additional 14k human judgements for answer candidates produced by BiDAF, Luke, and XLNet on the SQuAD dev set.

answer-equivalence-dataset Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Answer Equivalence Dataset

This dataset is introduced and described in [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](http://arxiv.org/abs/2202.07654).

## Download the data

| AE Split  | # AE Examples | # Ratings |
|-----------|---------------|-----------|
| [Train](https://github.com/google-research-datasets/answer-equivalence-dataset/blob/main/v1/train.jsonl) | 9,090 | 9,090 |
| [Dev](https://github.com/google-research-datasets/answer-equivalence-dataset/blob/main/v1/ae_dev.jsonl) | 2,734 | 4,446 |
| [Test](https://github.com/google-research-datasets/answer-equivalence-dataset/blob/main/v1/ae_test.jsonl) | 5,831 | 9,724 |
| Total  | 17,655 | 23,260 |

| Split by system  | # AE Examples | # Ratings |
|------------------|---------------|-----------|
| [BiDAF dev predictions](https://github.com/google-research-datasets/answer-equivalence-dataset/blob/main/v1/dev_by_system/dev_bidaf.jsonl) | 5622 | 7522 |
| [XLNet dev predictions](https://github.com/google-research-datasets/answer-equivalence-dataset/blob/main/v1/dev_by_system/dev_xlnet.jsonl) | 2448 | 7932 |
| [Luke dev predictions](https://github.com/google-research-datasets/answer-equivalence-dataset/blob/main/v1/dev_by_system/dev_luke.jsonl) | 2240 | 4590 |
| Total  | 8,565 | 14,170 |

## BERT Matching (BEM) model

The BEM model from the paper, finetuned on this dataset, is available on [tfhub](https://tfhub.dev/google/answer_equivalence/bem/1).

This [colab](https://colab.research.google.com/github/google-research-datasets/answer-equivalence-dataset/blob/main/Answer_Equivalence_BEM_example.ipynb) demonstrates how to use it.

## How to cite AE?

```
@article{bulian-etal-2022-tomayto,
  author    = {Jannis Bulian and
		Christian Buck  and
		Wojciech Gajewski and
		Benjamin B{\"o}rschinger and
		Tal Schuster},
  title     = {Tomayto, Tomahto. Beyond Token-level Answer Equivalence 
               for Question Answering Evaluation},
  journal   = {CoRR},
  volume    = {abs/2202.07654},
  year      = {2022},
  ee        = {http://arxiv.org/abs/2202.07654},
}
```

## Disclaimer

This is not an official Google product.

## Contact information

For help or issues, please submit a GitHub issue or contact the authors by email.

answer-equivalence-dataset

answer-equivalence-dataset Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

answer-equivalence-dataset

answer-equivalence-dataset Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README