Difference between revisions of "Data Jamboree 2/Annotation Experiment"

From phenoscape
(Results:)
 
(4 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Curation consistency experiment: Review of Results ==
+
===Background:===
 +
The curation experiment was given to five curators on the third day of the [[Data_Jamboree_2/Agenda | second data jamboree]]. The goals of the experiment were to assess curation consistency among a group of new curators, and to identify areas of improvement in curator training, ontology development, and software improvement.
  
Character #2
+
===Participant Training:===
* difference between depth, length, width
+
Only one of the five curators had experience using Phenex prior to the data roundup. Training for all curators consisted of:
* some failed to post-compose (e.g., increased length)
 
* need to standardize shape, length, depth, and width definitions
 
* should then use higher level character (such as 'shape')
 
  
Character #3
+
* A hands-on, group curation exercise led and assisted by experienced curators given on the first day of the data roundup
* some use 'increased count' which is-a count
+
* Two days of individual curation work on publications related to each participant's area of taxonomic expertise, with assistance from project personnel
 +
* For reference, participants were given an [[Guide_to_Character_Annotation | Annotation Guide]] with examples of character types commonly encountered in the fish systematic literature
  
Character #4
+
===Results:===
* question of whether basihyal bone precludes presence (or implies absence) of cartilage
 
* question of how to annotate this looks more difficult than in reality is
 
* basihyal cartilage absent implies basihyal bone absent (because the latter develops from the former)
 
** in fact it can also be that the cartilage is absent b/c it has developed into the bone (completely ossified)
 
** hence need to add that basihyal is absent too
 
* graph view can be very helpful
 
  
Character #5
+
A summary of results for each character is presented below, and a comparison to the [[Data_Jamboree_1/Annotation_Experiment | first consistency experiment]] given at the Data Jamboree I is presented in this  [[Data_Jamboree_2/Annotation_Experiment#Comparison_of_Consistency_Experiment_I_vs._II | table.]]  Additional notes on the discussion of results with curators and advisors can be found [[Data_Jamboree_2/Notes#Curation_consistency_experiment:_Review_of_Results | here.]]
* software should prevent filling in relative entities for qualities that aren't relational
 
* post-composed relative entity because of uncertainty of whether the full (existing) term would be compatible with the definition the author may have been using (for a different clade)
 
* difficulty to find the 'bony projection' by auto-completion (because it doesn't pop up when typing the beginnings of 'projection')
 
* problem of capturing the 'overlaps with' aspect in addition to 'anterior to'
 
* full text search of the definitions would be very useful
 
  
Character #6
+
====Character Summary and Suggestions for Improvement====
* hypural is also contained within the upper lobe of the caudal fin
 
* difficulty of finding and comparing definitions of relationship types
 
  
Character #7
+
''1. Presence or absence of intercalar: (0) present; (1) absent.3''
* Software should ensure that entity starts with an entity term if post-composed, not a quality
+
*Consistency: 5/5
 +
*Summary: all curators annotated this character identically
 +
''2. Opercle depth to width ratio: (0) less than two; (1) about two or greater than two. Essentially, this character distinguishes between those taxa with a short, relatively broad opercle and those with a tall relatively slender opercle.2''
 +
*Consistency: 1/5
 +
*Summary of variable annotations:
 +
**The majority of curators used various size qualities (e.g., increased width; decreased depth) to describe differences in shape.
 +
*Suggestions to improve consistency:
 +
** Definitions of size terms need to be improved; also, size and it's children do not share a parent with shape.
 +
** Annotation of characters with detailed size information should be annotated to higher level (shape in this case).
 +
'''''3.''' Number of unbranched plus branched pelvic-fin rays: (0) 11; (1) nine; (2) more than 11.3''
 +
*Consistency: 3/5
 +
*Summary of variable annotations:
 +
**Increased count used as quality term for state 2 by one curator; this is OK because parent is_a count
 +
**Incorrect entity chosen by two curators (pelvic fin actinotrichium instead of pelvic fin lepidotrichium)
 +
**Quality and Count left blank by one curator
 +
''4. Basihyal: (0) present and ossified; (1) present and cartilaginous; (2) absent.1''
 +
*Consistency:
 +
*Summary of variable annotations:
 +
** most curators recorded both presence and absence of bone and cartilage terms for each state
 +
** bone composition (cartilaginous vs. ossified) used by one curator for quality
 +
*Suggestions and group discussion:
 +
** for state 2: basihyal cartilage absent implies basihyal bone absent (because the latter develops from the former)
 +
*** in fact it can also be that the cartilage is absent b/c it has developed into the bone (completely ossified)
 +
*** hence need to add that basihyal is absent too
 +
** graph view can be very helpful to visualize develops_from relationships
 +
''5. Position of anterior margin of nasal: (0) falling short of lateral process of mesethmoid (= lateral ethmoid wing of Weitzman, 1962); (1) extending anteriorly to overlie or extend beyond lateral process of mesethmoid.3''
 +
*Consistency: 1/5
 +
*Summary:
 +
**Incorrect entity (naris vs. nasal bone) used in post-composition
 +
**Monadic qualty incorrectly used rather than relational quality
 +
**Post-composed entity term chosen rather than approporiate pre-composed term
 +
**Relational structural quality used rather than relational positional quality
 +
*Suggestions:
 +
** software should prevent filling in relative entities for qualities that aren't relational
 +
** full text search of the definitions would be very useful
 +
''6. Number of hypurals in upper lobe of caudal fin:  (0) four; (1) three.2''
 +
*Consistency: 2/5
 +
**Incorrect post-composition for entity (in various ways; see individual annotations below)
 +
**Quality and Count left blank
 +
**contained_in used for post-composition
 +
*Suggestions:
 +
**display definitions for relationship types (contained_in vs. part_of)
 +
**hypural can also be "contained in" upper lobe of caudal fin
 +
''7. Presence or absence of medially directed, spine-like process on ventral surface of post-temporal: (0) present; (1) absent.3''
 +
*Consistency: 3/5
 +
*Summary:
 +
**post-composed entity incorrectly started with quality term
 +
**post-composed entity term lacking spatial information
 +
*Suggestions:
 +
** Software should ensure that entity starts with a spatial or anatomy term if post-composed, not a quality
 +
''8. Presence or absence of contact between frontal and pterotic: (0) frontal and pterotic bones in contact; (1) pterotic excluded from contact with frontal by sphenotic.3''
 +
*Consistency: 5/5 for state 0; 4/5 for state 1
 +
*Summary:
 +
**curator mistakenly(?) used same quality for both states
 +
*Suggestions:
 +
** Record free-text information not used in annotation in the comments field (e.g., sphenotic needs to go into the comment field)
 +
''9. Orientation of infrapharyngobranchial 1: (0) proximal tip anteriorly directed; (1) proximal tip posteriorly directed.1''
 +
*Consistency: 2/5
 +
**monadic quality term used incorrectly (related entity used in annotation)
 +
**less specific entity term used in post-composition (pharyngobranchial vs. pharyngobranchial 1)
 +
**spatial term incorrectly used as quality term
 +
**spatial term incorrectly used as related entity
 +
*Suggestions:
 +
** software should prevent filling in spatial or monadic terms for quality if related entity field is filled.
 +
** if too complex to express the exact nature of the orientatin, then annotate quality as "orientation"
 +
''10. Dermosphenotic (0) triangular; (1) triradiate; (2) tubular.1''
 +
*Consistency: 5/5
 +
**Summary: all curators annotated this character equivalently
 +
**Discussion about approporiateness of tripartite for state 1
  
Character #8
+
References:
* sphenotic needs to go into the comment field
 
  
Character #9
+
1Hilton, EJ. 2003. Comparative osteology and phylogenetic systematics of fossil and living bony-tongue fishes (Actinopterygii, Teleostei, Osteoglossomorpha). Zoological Journal of the Linnean Society 137:1-100.
* too complex to express the exact nature of the orientation, so choose just 'orientation'
 
  
Character #10
+
2Sanger, TJ and AR McCune. 2002. Comparative osteology of the Danio (Cyprinidae: Ostariophysi) axial skeleton with comments on Danio relationships based on molecules and morphology. Zoological Journal of the Linnean Society 135: 529-546.
* triradiate versus tripartite
+
 
 +
3Zanata AM and Vari RP. 2005. The family Alestidae (Ostariophysi, Characiformes): a phylogenetic analysis of a trans-Atlantic clade. Zoological Journal of the Linnean Society 145: 1-144.
 +
 
 +
 
 +
====Comparison of Consistency Experiment I vs. II====
 +
{| border="1" cellspacing="0" cellpadding="3"
 +
! Character #
 +
! # Participants with
 +
Completed Annotations (n=4; Exp. I)
 +
! % Consistency with Key (Exp. I)
 +
! # Participants with
 +
Completed Annotations (n=5; Exp. II)
 +
! % Consistency with Key (Exp. II)
 +
|-
 +
| 1 || 4 || 100 || 5 || 100
 +
|-
 +
| 2 || 3 || 0  || 5 || 20
 +
|-
 +
| 3 || 3 || 0  || 4 || 60
 +
|-
 +
| 4 || 4 || 0  || 5 ||
 +
|-
 +
| 5 || 3 || 33  || 5 || 20
 +
|-
 +
| 6 || 4 || 0  || 5 || 40
 +
|-
 +
| 7 || 4 || 50  || 5 || 60
 +
|-
 +
| 8 || 3 || 33  || 5 || 100
 +
|-
 +
| 9 || 3 || 0  || 5 || 40
 +
|-
 +
| 10 || 2 || 50  || 5 || 100
 +
|-
 +
|
 +
|}
 +
 
 +
====Raw data from participants====
 +
* [[Media:Consistency-expt-II-rawdata.xls|download excel file]]
  
 
[[Category:Data Jamboree 2]]
 
[[Category:Data Jamboree 2]]
 
[[Category:Curation]]
 
[[Category:Curation]]

Latest revision as of 14:46, 8 October 2008

Background:

The curation experiment was given to five curators on the third day of the second data jamboree. The goals of the experiment were to assess curation consistency among a group of new curators, and to identify areas of improvement in curator training, ontology development, and software improvement.

Participant Training:

Only one of the five curators had experience using Phenex prior to the data roundup. Training for all curators consisted of:

  • A hands-on, group curation exercise led and assisted by experienced curators given on the first day of the data roundup
  • Two days of individual curation work on publications related to each participant's area of taxonomic expertise, with assistance from project personnel
  • For reference, participants were given an Annotation Guide with examples of character types commonly encountered in the fish systematic literature

Results:

A summary of results for each character is presented below, and a comparison to the first consistency experiment given at the Data Jamboree I is presented in this table. Additional notes on the discussion of results with curators and advisors can be found here.

Character Summary and Suggestions for Improvement

1. Presence or absence of intercalar: (0) present; (1) absent.3

  • Consistency: 5/5
  • Summary: all curators annotated this character identically

2. Opercle depth to width ratio: (0) less than two; (1) about two or greater than two. Essentially, this character distinguishes between those taxa with a short, relatively broad opercle and those with a tall relatively slender opercle.2

  • Consistency: 1/5
  • Summary of variable annotations:
    • The majority of curators used various size qualities (e.g., increased width; decreased depth) to describe differences in shape.
  • Suggestions to improve consistency:
    • Definitions of size terms need to be improved; also, size and it's children do not share a parent with shape.
    • Annotation of characters with detailed size information should be annotated to higher level (shape in this case).

3. Number of unbranched plus branched pelvic-fin rays: (0) 11; (1) nine; (2) more than 11.3

  • Consistency: 3/5
  • Summary of variable annotations:
    • Increased count used as quality term for state 2 by one curator; this is OK because parent is_a count
    • Incorrect entity chosen by two curators (pelvic fin actinotrichium instead of pelvic fin lepidotrichium)
    • Quality and Count left blank by one curator

4. Basihyal: (0) present and ossified; (1) present and cartilaginous; (2) absent.1

  • Consistency:
  • Summary of variable annotations:
    • most curators recorded both presence and absence of bone and cartilage terms for each state
    • bone composition (cartilaginous vs. ossified) used by one curator for quality
  • Suggestions and group discussion:
    • for state 2: basihyal cartilage absent implies basihyal bone absent (because the latter develops from the former)
      • in fact it can also be that the cartilage is absent b/c it has developed into the bone (completely ossified)
      • hence need to add that basihyal is absent too
    • graph view can be very helpful to visualize develops_from relationships

5. Position of anterior margin of nasal: (0) falling short of lateral process of mesethmoid (= lateral ethmoid wing of Weitzman, 1962); (1) extending anteriorly to overlie or extend beyond lateral process of mesethmoid.3

  • Consistency: 1/5
  • Summary:
    • Incorrect entity (naris vs. nasal bone) used in post-composition
    • Monadic qualty incorrectly used rather than relational quality
    • Post-composed entity term chosen rather than approporiate pre-composed term
    • Relational structural quality used rather than relational positional quality
  • Suggestions:
    • software should prevent filling in relative entities for qualities that aren't relational
    • full text search of the definitions would be very useful

6. Number of hypurals in upper lobe of caudal fin: (0) four; (1) three.2

  • Consistency: 2/5
    • Incorrect post-composition for entity (in various ways; see individual annotations below)
    • Quality and Count left blank
    • contained_in used for post-composition
  • Suggestions:
    • display definitions for relationship types (contained_in vs. part_of)
    • hypural can also be "contained in" upper lobe of caudal fin

7. Presence or absence of medially directed, spine-like process on ventral surface of post-temporal: (0) present; (1) absent.3

  • Consistency: 3/5
  • Summary:
    • post-composed entity incorrectly started with quality term
    • post-composed entity term lacking spatial information
  • Suggestions:
    • Software should ensure that entity starts with a spatial or anatomy term if post-composed, not a quality

8. Presence or absence of contact between frontal and pterotic: (0) frontal and pterotic bones in contact; (1) pterotic excluded from contact with frontal by sphenotic.3

  • Consistency: 5/5 for state 0; 4/5 for state 1
  • Summary:
    • curator mistakenly(?) used same quality for both states
  • Suggestions:
    • Record free-text information not used in annotation in the comments field (e.g., sphenotic needs to go into the comment field)

9. Orientation of infrapharyngobranchial 1: (0) proximal tip anteriorly directed; (1) proximal tip posteriorly directed.1

  • Consistency: 2/5
    • monadic quality term used incorrectly (related entity used in annotation)
    • less specific entity term used in post-composition (pharyngobranchial vs. pharyngobranchial 1)
    • spatial term incorrectly used as quality term
    • spatial term incorrectly used as related entity
  • Suggestions:
    • software should prevent filling in spatial or monadic terms for quality if related entity field is filled.
    • if too complex to express the exact nature of the orientatin, then annotate quality as "orientation"

10. Dermosphenotic (0) triangular; (1) triradiate; (2) tubular.1

  • Consistency: 5/5
    • Summary: all curators annotated this character equivalently
    • Discussion about approporiateness of tripartite for state 1

References:

1Hilton, EJ. 2003. Comparative osteology and phylogenetic systematics of fossil and living bony-tongue fishes (Actinopterygii, Teleostei, Osteoglossomorpha). Zoological Journal of the Linnean Society 137:1-100.

2Sanger, TJ and AR McCune. 2002. Comparative osteology of the Danio (Cyprinidae: Ostariophysi) axial skeleton with comments on Danio relationships based on molecules and morphology. Zoological Journal of the Linnean Society 135: 529-546.

3Zanata AM and Vari RP. 2005. The family Alestidae (Ostariophysi, Characiformes): a phylogenetic analysis of a trans-Atlantic clade. Zoological Journal of the Linnean Society 145: 1-144.


Comparison of Consistency Experiment I vs. II

Character # # Participants with

Completed Annotations (n=4; Exp. I)

% Consistency with Key (Exp. I) # Participants with

Completed Annotations (n=5; Exp. II)

% Consistency with Key (Exp. II)
1 4 100 5 100
2 3 0 5 20
3 3 0 4 60
4 4 0 5
5 3 33 5 20
6 4 0 5 40
7 4 50 5 60
8 3 33 5 100
9 3 0 5 40
10 2 50 5 100

Raw data from participants