Difference between revisions of "Data Jamboree 1/Annotation Experiment"
(→Completeness of annotations) |
(→Variability of EQ statements) |
||
| Line 17: | Line 17: | ||
Completed Annotations* | Completed Annotations* | ||
! % Consistency with Key | ! % Consistency with Key | ||
| − | ! | + | ! Variable component of annotation |
|- | |- | ||
| 1 || 4 || 100 || | | 1 || 4 || 100 || | ||
| Line 46: | Line 46: | ||
Participants annotated only one character identically. Variation in the other annotations was due to several reasons. Most commonly, participants differed in how they post-composed entity terms, particularly in how entities were related in post-composition (for example, use of part_of/has_part). Participants also varied in choosing the appropriate quality term among many similar choices. Another point of difference was due to confusion in the definition of an anatomy term, pointing to the importance of consistently naming bone terms in the TAO. | Participants annotated only one character identically. Variation in the other annotations was due to several reasons. Most commonly, participants differed in how they post-composed entity terms, particularly in how entities were related in post-composition (for example, use of part_of/has_part). Participants also varied in choosing the appropriate quality term among many similar choices. Another point of difference was due to confusion in the definition of an anatomy term, pointing to the importance of consistently naming bone terms in the TAO. | ||
| − | The results of the annotation experiment highlight the need for annotation standards, and stream-lining of the software interface so that curators are not faced with so many similar choices for quality terms and relations. These needs are currently being addressed. | + | The results of the annotation experiment highlight the need for annotation standards, and stream-lining of the software interface so that curators are not faced with so many similar choices for quality terms and relations. These needs are currently being addressed. |
====References==== | ====References==== | ||
Revision as of 19:07, 24 April 2008
Contents
Background and Participant Preparation
An annotation experiment was conducted on day 2 of the Phenoscape Data Jamboree in order to assess curation consistency among the four trained participants. Training consisted of a hands-on group annotation exercise on day 1, and individual work on each participant's own publications with assistance from project personnel on days 1 and 2. An Annotation Guide with examples of character types commonly encountered in the fish systematic literature was also given to participants. Participants were given 2 hours to annotate 10 characters (plus one extra credit) taken from three publications (see References).
Results and Conclusions
Completeness of annotations
Three of the four participants attempted annotations for all 11 characters, while one participant finished only 7 characters. All participants recorded the character number and textual description, and selected the appropriate voucher specimen for each annotation. Only two of the four participants recorded evidence codes for each annotation.
Variability of EQ statements
A summary of annotation consistency among participants is presented in the table below (incomplete annotations due to software issues are excluded).
| Character # | # Participants with
Completed Annotations* |
% Consistency with Key | Variable component of annotation |
|---|---|---|---|
| 1 | 4 | 100 | |
| 2 | 3 | 0 | post-composition of Q term for relative length |
| 3 | 3 | 0 | incorrect recording of count values |
| 4 | 4 | 0 | TAO term definition confusion (bone vs. cartilage) |
| 5 | 3 | 33 | E post-composition; choice of appropriate Q |
| 6 | 4 | 0 | E post-composition |
| 7 | 4 | 50 | E post-composition |
| 8 | 3 | 33 | choice of appropriate Q term |
| 9 | 3 | 0 | E post-composition; choice of appropriate Q term |
| 10 | 2 | 50 | choice of appropriate Q term |
| EC | 2 | 25 | anatomy term post-composition; choice of appropriate quality term |
- *incomplete annotations due to software issues were excluded
Participants annotated only one character identically. Variation in the other annotations was due to several reasons. Most commonly, participants differed in how they post-composed entity terms, particularly in how entities were related in post-composition (for example, use of part_of/has_part). Participants also varied in choosing the appropriate quality term among many similar choices. Another point of difference was due to confusion in the definition of an anatomy term, pointing to the importance of consistently naming bone terms in the TAO.
The results of the annotation experiment highlight the need for annotation standards, and stream-lining of the software interface so that curators are not faced with so many similar choices for quality terms and relations. These needs are currently being addressed.
References
- Hilton EJ. 2003. Comparative osteology and phylogenetic systematics of fossil and living bony-tongue fishes (Actinopterygii, Teleostei, Osteoglossomorpha). Zoological Journal of the Linnean Society 137: 1-100.
- Sanger TJ, McCune AR. 2002. Comparative osteology of the Danio (Cyprinidae: Ostariophysi) axial skeleton with comments on Danio relationships based on molecules and morphology. Zoological Journal of the Linnean Society 135:529-546.
- Zanata AM, Vari RP. 2005. The family Alestidae (Ostariophysi, Characiformes): a phylogenetic analysis of a trans-Atlantic clade. Zoological Journal of the Linnean Society 145: 1-144.
