Edit this page on GitHub

Data Jamboree 1/Annotation Experiment

Background and Participant Preparation

An annotation experiment was conducted on day 2 of the April 2008 Data Jamboree in order to assess curation consistency among the four trained participants. Training consisted of a hands-on group annotation exercise on day 1, and individual work on each participant’s own publications with assistance from project personnel on days 1 and 2. An Annotation Guide with examples of character types commonly encountered in the fish systematic literature was also given to participants. For the experiment, participants were given 2 hours to annotate 10 characters (plus one extra credit) taken from three publications.

Results and Conclusions

Completeness of annotations

Three of the four participants attempted annotations for all 11 characters, while one participant finished only 7 characters. All participants recorded the character number and textual description, and selected the appropriate voucher specimen for each annotation. Only two of the four participants recorded evidence codes for each annotation.

Variability of EQ statements

A summary of annotation consistency among participants is presented in the table below (incomplete annotations due to software issues are excluded).

Character # # Participants with Completed Annotations*  % Consistency with Key Variable component of annotation
1 4 100  
2 3 0 post-composition of Q term for relative length
3 3 0 incorrect recording of count values
4 4 0 TAO term definition confusion (bone vs. cartilage)
5 3 33 E post-composition; choice of appropriate Q
6 4 0 E post-composition
7 4 50 E post-composition
8 3 33 choice of appropriate Q term
9 3 0 E post-composition; choice of appropriate Q term
10 2 50 choice of appropriate Q term
EC 2 25 E post-composition; choice of appropriate Q term

Participants annotated only one character identically. Variation in other annotations was due to several reasons:

The results of the annotation experiment provide detailed areas for improvement in establishing curation standards to assist in annotation, and stream-lining of the software interface so that curators are not faced with similar and inapplicable choices for terms and relations. Based on these results, curation standards and improvement to the Annotation Guide are in development, and improvements to the Phenote interface are planned.

Instructions to Participants

In the following exercise, we would like to assess annotation consistency among data jamboree participants working independently on a common set of characters. These characters are taken directly from the literature of comparative morphology of fishes.

Download the annotation-experiment folder from the Phenoscape server.

Use the taxon list and the character matrix given in Table 1 to annotate characters 1-11 below. For the purpose of this exercise, character states noted in Table 1 were observed in all specimens listed in the materials list. The name of this publication is: Lapp and Dahdul, 2008.

Table 1. Character matrix

Taxon

1

2

3

4

5

6

7

8

9

10

EC

Chanos chanos

0

0

0

0

0

0

0

0

0

0

0

Diplomystes chilensis

0

0

0

1

0

0

0

0

0

1

0

Hiodon alosoides

1

1

1

2

1

1

1

1

1

2

1

1. Presence or absence of intercalar: (0) present; (1) absent.

2. Opercle depth to width ratio: (0) less than two; (1) about two or greater than two. Essentially, this character distinguishes between those taxa with a short, relatively broad opercle and those with a tall relatively slender opercle.

3. Number of unbranched plus branched pelvic-fin rays: (0) 11; (1) nine; (2) more than 11.

4. Basihyal: (0) present and ossified; (1) present and cartilaginous; (2) absent.

5. Position of anterior margin of nasal: (0) falling short of lateral process of mesethmoid (= lateral ethmoid wing of Weitzman, 1962); (1) extending anteriorly to overlie or extend beyond lateral process of mesethmoid.

6. Number of hypurals in upper lobe of caudal fin: (0) four; (1) three.

7. Presence or absence of medially directed, spine-like process on ventral surface of post-temporal: (0) present; (1) absent.

8. Presence or absence of contact between frontal and pterotic: (0) frontal and pterotic bones in contact; (1) pterotic excluded from contact with frontal by sphenotic.

9. Orientation of infrapharyngobranchial 1: (0) proximal tip anteriorly directed; (1) proximal tip posteriorly directed.

10. Infraorbital 6: (0) triangular; (1) triradiate; (2) tubular.

Extra Credit: Form of parasphenoid: (0) relatively straight anteroposteriorly and with posterior portion of bone aligned approximately along longitudinal axis of anterior portion of vertebral column; (1) with distinct anteroposterior curvature and with posterior portion of bone located distinctly ventral to longitudinal axis through anterior portion of vertebral column. (Zanata & Vari, 2005)

References

Raw Data from Participants

Character Number

Character State

Participant

Entity Name

Quality Name

Additional Entity Name

Count

Measurement

Unit Name

1

0

key

intercalar

present

1

0

1

intercalar

present

1

0

2

intercalar

present

1

0

3

intercalar

present

1

0

4

intercalar

present

1

1

key

intercalar

absent

1

1

1

intercalar

absent

1

1

2

intercalar

absent

1

1

3

intercalar

absent

1

1

4

intercalar

absent

2

0

key

opercle

depth^relative_to(width)

<2

ratio

2

0

1

opercle

depth^relative_to(width)

opercle

<2

ratio

2

0

2

opercle

2

0

3

opercle

increased width

decreased height

2

0

4

opercle

relational shape quality

<2

2

1

key

opercle

depth^relative_to(width)

>, = 2

2

1

1

opercle

depth^relative_to(width)

>, = 2

ratio

2

1

2

opercle

2

1

3

opercle

increased height

decreased height

2

1

4

opercle

relational shape quality

2 or >

3

0

key

pelvic fin lepidotrichium

count

11

3

0

1

pelvic fin lepidotrichium

count

11

3

0

2

pelvic fin lepidotrichium

count

11

3

0

3

pelvic fin lepidotrichium

count

11

3

0

4

accessory pretectal nucleus

count

11

3

1

key

pelvic fin lepidotrichium

count

9

3

1

1

pelvic fin lepidotrichium

count

9

3

1

2

pelvic fin lepidotrichium

count

9

3

1

3

pelvic fin lepidotrichium

count

9

3

1

4

pelvic fin

count

>11

3

1

key

pelvic fin

count

>11

3

2

1

pelvic fin lepidotrichium

count

11

3

2

2

pelvic fin lepidotrichium

count

11

3

2

3

pelvic fin lepidotrichium

count

11

3

2

4

pelvic fin

count

11

4

0

key

basihyal

present

4

0

1

basihyal^is_a(bone)

present

4

0

2

basihyal cartilage

ossified

4

0

3

basihyal

ossified

4

0

4

basihyal

ossified

4

1

key

basihyal cartilage

present

4

1

1

basihyal cartilage

present

4

1

2

basihyal cartilage

cartilaginous

4

1

3

basihyal

cartilaginous

4

1

4

basihyal

cartilaginous

4

2

key

basihyal cartilage

absent

4

2

1

basihyal

absent

4

2

2

basihyal cartilage

absent

4

2

3

basihyal

absent

4

2

4

basihyal

absent

5

0

key

nasal bone

separated from

mesethmoid bone

5

0

1

nasal bone^has_boundary(posterior)

position

mesethmoid bone

5

0

2

anterior margin^part_of(nasal bone)

separated from

mesethmoid bone

5

0

3

nasal bone

length^ends_earlier_than(length)

mesethmoid bone^part_of(lateral ethmoid process)

5

0

4

nasal bone

posterior to

5

0

4

nasal bone

posterior to

5

1

key

nasal bone

overlaps with

lateral ethmoid ventral wing

5

1

1

basihyal

composition

5

1

2

anterior margin^part_of(nasal bone)

overlap with

mesethmoid bone

5

1

3

nasal bone

length^ends_during(length)

mesethmoid bone^part_of(lateral ethmoid process)

5

1

4

nasal bone

lateral to

6

0

key

hypural^part_of(caudal fin)

count

4

6

0

1

hypural

count

caudal fin^has_integral_part(caudal fin upper lobe)

4

6

0

2

hypural

count

caudal fin upper lobe

4

6

0

3

caudal fin upper lobe^contains(hypural)

count

4

6

0

4

caudal fin upper lobe^arises_from(medial surface)

count

4

6

1

key

hypural^part_of(caudal fin)

count

3

6

1

1

hypural

count

caudal fin^has_integral_part(caudal fin upper lobe)

3

6

1

2

hypural

count

caudal fin upper lobe

3

6

1

3

caudal fin upper lobe^contains(hypural)

count

3

6

1

4

caudal fin upper lobe

count

3

7

0

key

bony projection^part_of(ventral surface)^part_of(posttemporal)

present

7

0

1

posttemporal

present

posttemporal^has_part(specialized hemal arch and spine)

4

7

0

2

bony projection^part_of(ventral surface)^part_of(posttemporal)

present

7

0

3

bony projection^contained_in(posttemporal)

present

7

0

4

bony projection^part_of(posttemporal)

absent

7

0

key

bony projection^part_of(ventral surface)^part_of(posttemporal)

absent

7

1

1

posttemporal

present

posttemporal^has_part(specialized hemal arch and spine)

3

7

1

2

bony projection^part_of(ventral surface)^part_of(posttemporal)

absent

7

1

3

bony projection^contained_in(posttemporal)

absent

7

1

4

bony projection^part_of(posttemporal)

present

8

0

key

frontal bone

in contact with

pterotic

8

0

2

frontal bone

in contact with

pterotic

8

0

3

frontal bone

pterotic

8

0

3

frontal bone

pterotic

8

0

4

frontal bone

pterotic

8

1

key

frontal bone

separated from

pterotic

8

1

2

frontal bone

separated from

pterotic

8

1

3

frontal bone

pterotic

8

1

4

frontal bone

pterotic

8

1

4

frontal bone

pterotic

9

0

key

proximal region^part_of(pharyngobranchial 1 bone)

anteriorly rotated

9

0

2

proximal region^part_of(pharyngobranchial 1 bone)

anterioralized

9

0

3

pharyngobranchial 1 bone

9

0

4

anatomical region^arises_from(medial surface)^part_of(pharyngobranchial 1 bone)

anteriorly rotated

9

1

key

proximal region^part_of(pharyngobranchial 1 bone)

posteriorly rotated

9

1

2

proximal region^part_of(pharyngobranchial 1 bone)

posterioralized

9

1

3

pharyngobranchial 1 bone

9

1

4

anatomical region^arises_from(medial surface)^part_of(pharyngobranchial 1 bone)

posteriorly rotated

10

0

key

infraorbital 6

triangular

10

0

2

infraorbital 6

triangular

10

0

3

infraorbital 6

shape

10

0

4

infraorbital 4

posteriorly rotated

10

1

key

infraorbital 6

tripartite

10

1

2

infraorbital 6

tripartite

10

1

3

infraorbital 6

shape

10

1

4

infraorbital 4

posteriorly rotated

10

2

key

infraorbital 6

cylindrical

10

2

2

infraorbital 6

cylindrical

10

2

3

infraorbital 6

shape

10

2

4

infraorbital 4

anteriorly rotated

ECa

0

key

anterior/posterior axis^part_of(parasphenoid)

straight angle

ECa

0

4

parasphenoid

straight angle

ECa

0

2

anterior/posterior axis^part_of(parasphenoid)

straight angle

ECa

1

key

anterior/posterior axis^part_of(parasphenoid)

curved ventral

ECa

1

4

parasphenoid

curved ventral

ECa

1

2

anterior/posterior axis^part_of(parasphenoid)

curved

vertebral column

ECb

0

key

posterior region^part_of(parasphenoid)

aligned with

anterior region^part_of(vertebral column)

ECb

0

2

posterior region^part_of(parasphenoid)

aligned with

vertebral column

ECb

1

key

posterior region^part_of(parasphenoid)

ventral to

anterior region^part_of(vertebral column)

ECb

1

2

posterior region^part_of(parasphenoid)

ventral to