CLAWS4: THE TAGGING OF THE BRITISH NATIONAL CORPUS

Geoffrey Leech, Roger Garside and Michael Bryant

UCREL
Lancaster University
United Kingdom

1. INTRODUCTION

The British National Corpus (BNC) consists of c.100 million words of English written texts and spoken transcriptions, sampled from a comprehensive range of text types. The BNC includes 10 million words of spoken language, c.45% of which is impromptu conversation (see Crowdy, forthcoming). It also includes an immense variety of written texts, including unpublished materials. The grammatical tagging of the corpus has therefore required the "super-robustness" of a tagger which can adapt well to virtually all kinds of text. The tagger also has had to be versatile in dealing with different tagsets (sets of grammatical category labels - see 3 below) and accepting text in varied input formats. For the purposes of the BNC, the tagger has been required both to accept and to output text in a corpus- oriented TEI-conformant mark-up format known as CDIF (Corpus Document Interchange Format), but within this format many variant formats (affecting, for example, segmentation into words and sentences) can be readily accepted. In addition, CLAWS allows variable output formats: for the current tagger, these include (a) a vertically-presented format suitable for manual editing, and (b) a more compact horizontally- presented format often more suitable for end- users. Alternative output formats are also allowed with (c) so-called "portmanteau tags", i.e. combinations of two alternative tags, where the tagger calculates there is insufficient evidence for safe disambiguation, and (d) with simplified "plain text" mark-up for the human reader.

CLAWS4, the BNC tagger[Footnote 2], incorporates many features of adaptability such as the above. It also incorporates many refinements of linguistic analysis which have built up over 14 years: particularly in the construction and content of the idiom-tagging component (see 2 below). At the same time, there are still many improvements to be made: the claim that "you can put together a tagger from scratch in a couple of months" (recently heard at a research conference) is, in our view, absurdly optimistic.

2. THE DESIGN OF THE GRAMMATICAL TAGGER (CLAWS4)

Section a:

segmentation of text into word and sentence units.

Section b:

initial (non-contextual) part-of-speech assignment [using a lexicon, word-ending list, and various sets of rules for tagging unknown items]

Section c:

rule-driven contextual part-of-speech assignment

Section d:

probabilistic tag disambiguation [Markov process]

Section c':

[second pass of Stage C]

Section e:

output in intermediate form

The pre-processing section (a) is not trivial, since, in any large and varied corpus, there is a need to handle unusual text structures (such as those of many popular and technical magazines), less usual graphic features (e.g. non-roman alphabetic characters, mathematical symbols), and features of conversation transcriptions: e.g. false starts, incomplete words and utterances, unusual expletives, unplanned repetitions, and (sometimes multiple) overlapping speech.

Sections (b) and (d) apply essentially a Hidden Markov Model (HMM) to the assignment and disambiguation of tags. But the intervening section (c) has become increasingly important as CLAWS4 has developed the need for versatility across a range of text types. This task of rule- driven contextual part-of-speech assignment began in 1981 as an "idiom-tagging" program for dealing, in the main, with parts of speech extending over more than one orthographic word (e.g. complex prepositions such as according to and complex conjunctions such as so that). In the more fully developed form it now has, this section utilises several different idiom lexicons dealing, for example, with (i) general idioms such as as much as (which is on one analysis a single coordinator, and on another analysis, a word sequence), (ii) complex names such as Dodge City and Mrs Charlotte Green (where the capital letter alone would not be enough to show that Dodge and Green are proper nouns), (iii) foreign expressions such as annus horribilis.

These idiom lexicons (over 3000 entries in all) can match on both tags and word-tokens, employing a regular expression formalism at the level of the individual item and the sequence of items. Recognition of unspecified words with initial capitals is also incorporated. Conceptually, each entry has two parts: (a) a regular-expression- based "template" specifying a set of conditions on sequences of word-tag pairs, and (b) a set of tag assignments or substitutions to be performed on any sequence matching the set of conditions in (a). Examples of entries from each of the above kinds of idiom lexicon entry are:

1. as AV0, ([ ])5, as CJS, possible AJ0

2. Monte/Mount/Mt NP0, ([WIC])2 NP0, [WIC] NP0

3. ad AV021 AJ021, hoc AV022 AJ022

While the HMM-type process employed in Sections (b) and (d) affirms our faith in probabilistic methods, the growing importance of the contextual part-of-speech assignment in (c) and (c') demonstrates the extent to which it is important to transcend the limitations of the orthographic word, as the basic unit of grammatical tagging, and also to selectively adopt non-probabilistic solutions. The term "idiom-tagging" was never particularly appropriate for these sections, which now handle more generally the interdependence between grammatical and lexical processing which NLP systems ultimately have to cope with, and are also able to incorporate parsing information beyond the range of the one-step Markov process (based on tag bigram frequences) employed in (d)[Footnote 3]. Perhaps the term "phraseological component" would be more appropriate here. The need to combine probabilistic and non-probabilistic methods in tagging has been widely noted (see, e.g., Voutilainen et al. 1992).

3. EXTENDING ADAPTABILITY: SPOKEN DATA AND TAGSETS

The BNC tagging takes place within the context of a larger project, in which a major task (undertaken by OUCS at Oxford) is to encode the texts in a TEI-conformant mark-up (CDIF). Two tagsets have been employed: one, more detailed than the other, is used for tagging a 2-million- word Core Corpus (an epitome of the whole BNC), which is being post-edited for maximum accuracy. Thus tagsets, like text formats and datastructures, are among the features which are task-definable in CLAWS4. In general, the system has been revised to allow many adaptive decisions to be made at run time, and to render it suitable for non-specialist researchers to use.

4. ERROR RATES AND WHAT THEY MEAN

1. Consistency. It is necessary to measure tagging practice against some standard of what is an appropriate tag for a given word in a given context. For example, is horrifying in a horrifying adventure, or washing in a washing machine an adjective, a noun, or a verb participle? Only if this is specified independently, by an annotation scheme, can we feel confident in judging where the tagger is "correct" or "incorrect". For the tagging of the LOB Corpus by the earliest version of CLAWS, the annotation scheme was published in some detail (Johansson et al 1986). We are working on a similar annotation scheme document (at present a growing in-house document) for the tagging of the BNC.

2. Size of Tagset. It might be supposed that tagging with a more fine-grained tagset which contains more tags is more likely to produce error than tagging with a smaller and cruder tagset. In the BNC project, we have used a tagset of 58 tags (the C5 tagset) for the whole corpus, and in addition we have used a larger tagset of 138 tags (the C6 tagset) [Footnote 5] for the Core Corpus of 2 million words. The evidence so far is that this makes little difference to the error rate. But size of tagset must, in the absence of more conclusive evidence, remain a factor to be considered.

3. Discriminative Value of Tags. The difficulty of grammatical tagging is directly related to the number of words for which a given tag distinction is made. This measure may be called "discriminative value". For example, in the C5 tagset, one tag (VDI) is used for the infinitive of just one verb - to do - whereas another tag (VVI) is used for the infinitive of all lexical verbs. On the other hand, VDB is used for finite base forms of to do (including the present tense, imperative, and subjunctive), whereas VVB is used of finite base forms of all lexical verbs. It is clear the tags VDI and VDB have a low discriminative value, whereas VVI and VVB have a high one - since there are thousands of lexical verbs in English. It is also clear that a tagset of the lowest possible discriminative value - one which assigned a single tag to each word and a single word to each tag - would be utterly valueless.

4. Linguistic Quality. This is a very elusive, but crucial concept. How far are the tags in a particular tagset valuable, by criteria either of linguistic theory/description, or of usefulness in NLP? For example, the tag VDI, mentioned in c. above, appears trivial, but it can be argued that this is nevertheless a useful category for English grammar, where the verb do (unlike its equivalent in most other European languages) has a very special function, e.g. in forming questions and negatives. On the other hand, if we had decided to assign a special tag to the verb become, this would have been more questionable. Linguistic quality is, on the face of it, determined only in a judgemental manner. Arguably, in the long term, it can be determined only by the contribution a particular tag distinction makes to success in particular applications, such as speech recognition or machine-aided translation. At present, this issue of linguistic quality is the Achilles' heel of grammatical tagging evaluation, and we must note that without judgement on linguistic quality, evaluation in terms of (2) and (3) is insecurely anchored.

References

Garside, R, G. Leech and G. Sampson (eds), (1987). The Computational Analysis of English: a Corpus-based Approach. London: Longman.

Johansson, S., E. Atwell, R. Garside and G. Leech (1986). The Tagged LOB Corpus: User's Manual. Bergen: Norwegian Computing Centre for the Humanities.

Marshall, I. (1983). Choice of grammatical word-class without global syntactic analysis: tagging words in the LOB Corpus, Computers and the Humanities, 17, 139-50.

Voutilainen, A. , J. Heikkilä and A. Anttila (1992). Constraint Grammar of English: A Performance-Oriented Introduction. University of Helsinki: Department of General Linguistics.