SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP

This is the official repository for the dataset and code of the paper: "SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP", accepted at EMNLP 2025 (Main Conference).

Our dataset is constructed based on resources from The ACL OCL Corpus (Shaurya Rohatgi, 2022) . The dataset can be accessed at: SciNLP

It contains four files:

• train/test/dev.json: Train, dev and test data set of SciNLP data set.
• KG_triples.txt: Triples extracted from the ACL Anthology PDFs by using the HGERE model based on SciNLP training

The data format for models is JSON. Each line of the input file contains one document in the following format.

{
  # document ID (please make sure doc_key can be used to identify a certain document)
  "doc_key": "2020.acl-main.21",

  # sentences in the document, each sentence is a list of tokens
  "sentences": [
    [...],
    [...],
    ["Prior", "works", "performed", "SQG", "on", "CoQA", ...],
    ...
  ],

  # entities (boundaries and entity type) in each sentence
  "ner": [
    [...],
    [...],
    [[3, 3, "task"], [5, 5, "dataset"], ...], #the boundary positions are indexed in the document level
    ...,
  ],

  # relations (two spans and relation type) in each sentence
  "relations": [
    [...],
    [...],
    [[3, 3, 5, 5, "evaluatedOn"] ...],
    ...
  ]
}

Our used supervised methods include:

• PURE
• PL-Marker
• HGERE

SciNLP is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. The dataset follows the same policy as The ACL OCL Corpus : for non-commercial academic research use only.

If you use this dataset, please cite the following works:

@inproceedings{scinlp,
  author    = {Decheng Duan, Yingyi Zhang, Jitong Peng and Chengzhi Zhang},
  title     = {SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP2025)},
  year      = {2025},
  address   = {Suzhou, China}
}

@Misc{acl_anthology_corpus,
    author =       {Shaurya Rohatgi},
    title =        {ACL Anthology Corpus with Full Text},
    howpublished = {Github},
    year =         {2022},
    url =          {https://github.com/shauryr/ACL-anthology-corpus}
}