2. Presentation of Ambiguity Rates and Error Rates (fine-grained mode of calculation)

In this section, we examine ambiguities and errors using a ‘fine-grained’ mode of calculation, treating each error as of equal importance to any other error. In section 4, we look at the same data in terms of a ‘coarse-grained’ mode of calculation, ignoring errors and ambiguities involving subcategories of the same part of speech.

2.1 Overall estimated ambiguity and error rates: based on the 50,000 word sample

As the following table shows, the ambiguity rate varies considerably between written and spoken texts. (However, note that the calculation for speech is based on a small sample of 5,000 words.)

Table 1: Estimated ambiguity and error rates for the whole corpus (fine-grained calculation)

It will be noted that written texts on the whole have a higher ambiguity rate, whereas spoken texts have a slightly greater error rate.

The success of an automatic tagger is sometimes represented in terms of the information-retrieval measures of precision and recall, rather than ambiguity rate and error rate as in Table 1. Precision is the extent to which incorrect tags are successfully discarded from the output. Recall is the extent to which all correct tags are successfully retained in the output of the tagger, allowing, however, for more than one reading to occur for one word (i.e. ambiguous tagging is permitted). According to these measures, the success of the tagging is as follows:

However, from now on we will continue to use ‘ambiguity rate’ and ‘error rate’, which appear to us more transparent.

2.2 Estimated ambiguity and error rates for each tag (fine-grained mode of calculation)

The estimates for individual tags are again based on the 50,000 sample, and the ambiguity rate for each tag is based on the number of ambiguity tags which begin with a given tag. The table also specifies the estimated likelihood that a given tag, in the first position of the ambiguity tag, is the correct tag.

Table 2: Estimated ambiguity rates and error rates (by tag) (fine-grained calculation)

2.3 Estimated error rates specifying the incorrect tag and the correct tag (fine-grained calculation)

The next table, Table 3, gives the frequency, as a percentage, of error-prone tag-pairs where XXX is the incorrect tag and YYY is the correct tag which should have occurred in its place. In the third column, the number of the specified error-type is listed, as a frequency count from the sample of 50,000 words. In the fourth column, this is expressed as a percentage of all the tagging errors of word category XXX (in Table 2 column (f)). The fifth column answers the question: if tag XXX occurs, what is the likelihood that it is an error for tag YYY? Where the number of occurrences of a given error-type is less than 5 (i.e. 1 in 10,000 words), they are ignored. Hence, Table 3 is not exhaustive: only the more likely error-types are listed. In the second column, we add, where useful, the individual words which trigger these errors.

Similar to before, the asterisk * indicates a ‘less serious’ error, in which the erroneous and correct tags belong to the same major category or part of speech. As the Table shows, the most frequent specific error types are within the verb category: VVB ? VVI (55, or 9.8% of all VVB tags) and VVD ? VVN (44, or 4.5% of all VVD tags).

Yet a further way of looking at the ambiguities and errors in the corpus is to make a coarse-grained calculation in counting these phenomena. In a fine-grained measurement, which is the one assumed up to now, each tag is considered to define its own word class which is different from all other word classes. Using the coarse-grained calculation, on the other hand, we consider words to belong to different word classes (parts of speech) only when the major category is different. If we consider the pair NN1 (singular and common noun) and NP0 (proper noun), the coarse-grained calculation says that the ambiguity tag NN1-NP0 or NP0-NN1 does not show tagging uncertainty, since both the proposed tags agree in categorizing the word as the same part of speech (a noun). So this does not add to the ambiguity rate. Similarly, the coarse-grained point of view on error is that, if a word is tagged as NN1 when it should be NP0, or vice versa, then this is not error, because both tags are within the noun category. To summarize: in the fine-grained calculation, minor differences of word-class count towards the ambiguity and error rates; in the coarse-grained calculation, they do not.

In this section, the same calculations are made as in section 3, except that errors and ambiguities which are confined within a major category (noun, verb, etc.) are ignored. In practice, most of the errors and ambiguities of this kind come from the difficulty the tagger finds in recognizing the difference between NN1 (singular common noun) and NP0 (proper noun), between VVD (past tense lexical verb) and VVN (past participle lexical verb), and between VVB (finite present tense base form, lexical verb) and VVI (infinitive lexical verb). Thus the ambiguity tags NN1-NP0, VVD-VVN and their mirror images do not occur in the relevant table (Table 5) below. However, since there are no ambiguity tags for VVB and VVI, the problem of distinguishing these two shows up only in the error calculation. The three tables in this section, Tables 4-6, correspond to Tables 1-3 above respectively:

Table 4: Estimated ambiguity and error rates for the whole corpus (coarse-grained calculation)

It will be noted from Table 4 that this method of calculation reduces the overall ambiguity rate by c.1 per cent, and the overall error rate by c.0.5 per cent. We will not present coarse-grained tables corresponding to Tables 2 and 3 above: these tables would be unchanged from the fine-grained calculation, except that the rows marked with an asterisk (*) would be deleted, and the other calculations changed as necessary.

Table 5: Estimated error rates for the whole corpus after ambiguities have been automatically eliminated (assuming the first part of each ambiguity tag to be correct)

The following table gives an error count (c) for each tag: i.e. the number of errors in the 50,000 word sample where that tag was the erroneous tag. [Cf. the "safer" error count in Table 3, column (f).] In addition, each tag has a correction count (d): i.e. the number of erroneous tags for which that tag was the correct tag. If we subtract the Error count (c) from the Tag count (b), and add the Correction count (d) to the result, we arrive at the "Real tag count" (e) representing the number of occurrences of that tag in the corrected sample corpus. Not included in the table is the small number of ‘multi-word’ errors which resulted in two tags being replaced by one (error count), or one tag being replaced by two (correction count), due to the incorrect non-use or use of multi-word tags. The last column divides the error count by the tag count to provide the error rate (as a percentage).

It is clear from this table that the amount of error in the tagging of the corpus varies greatly from one tag to another. The most error prone-tag, by a large margin, is VVB, with more than 17 per cent error, while many of the tags are associated with no errors at all, and well over half the tags have less than a 1 per cent error.

The final table, Table 7, gives figures for the third level of detail, where we itemise individual tag pairs XXX, YYY, where XXX is the incorrect tag, and YYY is the correct one which should have appeared but did not. Only those pairings which account for 5 or more errors are listed. This table differs from Table 3 in that here the second tags of ambiguity tags are not taken into account ("riskier mode"). It will be seen that the errors which occur tend to fall into a relatively small number of major categories.

Table 7: Estimated frequency of selected tag-pairs, where XXX is the incorrect tag, and YYY is the correct one (after the second tags of ambiguity tags have been eliminated automatically) based on the sample of 50,000 words

Some of the error types above are associated with one or two particular words, and where these occur they are listed. For example, the AV0 - EX0 type of error occurs invariably with the one word there.

Appendix:

List of text samples used for the manually-conducted 50,000-word error analysis

Each sample consisted of 2,000 words of each of the BNC texts below, except that two samples, one of written and one of spoken English, consisted of 1,000 words only. These samples are marked "*" in the list below. The reason for using half-length samples in two cases was to maintain the proportion of written and spoken data as 90% - 10%, so as to keep the proportions of the sample the same as the proportions in the BNC as a whole. The BNC text files are cited by the three-character code used in the BNC Users Reference Guide.

Written: Imaginative writing G0S, ADY, H7P, GW0, FSF

Written: Informative writing

Natural Science JXN

Applied Science HWV, CEG

Social Science CLH, EE8, *A6Y

World Affairs A4J, CMT, EE2, EB7

Commerce and finance HGP, B27

Arts C9U, G1N

Belief and Thought CA9

Leisure EX0, ADR, CE4

Spoken: Demographic KBG

Context-governed D8Y, *FXH