Menu
×
West Ashley Library
Closed
Phone: (843) 766-6635
Wando Mount Pleasant Library
Closed
Phone: (843) 805-6888
Village Library
Closed
Phone: (843) 884-9741
St. Paul's/Hollywood Library
Closed
Phone: (843) 889-3300
Otranto Road Library
Closed
Phone: (843) 572-4094
Mt. Pleasant Library
Closed
Phone: (843) 849-6161
McClellanville Library
Closed
Phone: (843) 887-3699
Keith Summey North Charleston Library
Closed
Phone: (843) 744-2489
John's Island Library
Closed
Phone: (843) 559-1945
Hurd/St. Andrews Library
Closed
Phone: (843) 766-2546
Folly Beach Library
Closed
Phone: (843) 588-2001
Edisto Island Library
Closed
Phone: (843) 869-2355
Dorchester Road Library
Closed
Phone: (843) 552-6466
John L. Dart Library
Closed
Phone: (843) 722-7550
Baxter-Patrick James Island
Closed
Phone: (843) 795-6679
Main Library
2 p.m. – 5 p.m.
Phone: (843) 805-6930
Bees Ferry West Ashley Library
Closed
Phone: (843) 805-6892
Edgar Allan Poe/Sullivan's Island Library
Closed for renovations
Phone: (843) 883-3914
Mobile Library
Closed
Phone: (843) 805-6909
Today's Hours
West Ashley Library
Closed
Phone: (843) 766-6635
Wando Mount Pleasant Library
Closed
Phone: (843) 805-6888
Village Library
Closed
Phone: (843) 884-9741
St. Paul's/Hollywood Library
Closed
Phone: (843) 889-3300
Otranto Road Library
Closed
Phone: (843) 572-4094
Mt. Pleasant Library
Closed
Phone: (843) 849-6161
McClellanville Library
Closed
Phone: (843) 887-3699
Keith Summey North Charleston Library
Closed
Phone: (843) 744-2489
John's Island Library
Closed
Phone: (843) 559-1945
Hurd/St. Andrews Library
Closed
Phone: (843) 766-2546
Folly Beach Library
Closed
Phone: (843) 588-2001
Edisto Island Library
Closed
Phone: (843) 869-2355
Dorchester Road Library
Closed
Phone: (843) 552-6466
John L. Dart Library
Closed
Phone: (843) 722-7550
Baxter-Patrick James Island
Closed
Phone: (843) 795-6679
Main Library
2 p.m. – 5 p.m.
Phone: (843) 805-6930
Bees Ferry West Ashley Library
Closed
Phone: (843) 805-6892
Edgar Allan Poe/Sullivan's Island Library
Closed for renovations
Phone: (843) 883-3914
Mobile Library
Closed
Phone: (843) 805-6909
Patron Login
menu
Item request has been placed!
×
Item request cannot be made.
×
Processing Request
Analyzing Dataset Annotation Quality Management in the Wild.
Item request has been placed!
×
Item request cannot be made.
×
Processing Request
- Author(s): Klie, Jan-Christoph (AUTHOR); Castilho, Richard Eckart de (AUTHOR); Gurevych, Iryna (AUTHOR)
- Source:
Computational Linguistics. Sep2024, Vol. 50 Issue 3, p817-866. 50p. - Source:
- Additional Information
- Subject Terms:
- Abstract: Data quality is crucial for training accurate, unbiased, and trustworthy machine learning models as well as for their correct evaluation. Recent work, however, has shown that even popular datasets used to train and evaluate state-of-the-art models contain a non-negligible amount of erroneous annotations, biases, or artifacts. While practices and guidelines regarding dataset creation projects exist, to our knowledge, large-scale analysis has yet to be performed on how quality management is conducted when creating natural language datasets and whether these recommendations are followed. Therefore, we first survey and summarize recommended quality management practices for dataset creation as described in the literature and provide suggestions for applying them. Then, we compile a corpus of 591 scientific publications introducing text datasets and annotate it for quality-related aspects, such as annotator management, agreement, adjudication, or data validation. Using these annotations, we then analyze how quality management is conducted in practice. A majority of the annotated publications apply good or excellent quality management. However, we deem the effort of 30% of the studies as only subpar. Our analysis also shows common errors, especially when using inter-annotator agreement and computing annotation error rates. [ABSTRACT FROM AUTHOR]
- Abstract: Copyright of Computational Linguistics is the property of MIT Press and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Contact CCPL
Copyright 2022 Charleston County Public Library Powered By EBSCO Stacks 3.3.0 [350.3] | Staff Login
No Comments.