Tagging

Assume:

• Corpus to be tagged
• Tagset (87 to 146 tags): a flat classification, though there may be additional features
• Dictionary: associates words with tags

Tags are usually represented in the corpus text file with some kind of "markup language": slash tags, SGML

Computational problem:

· Preprocessing: "tokenization" (finding word boundaries), normalizing words, morphological parsing (not covered in this chapter)

· Match every word in the part of speech to set of tags from the tagset by dictionary lookup

· If there's more than one possibility, choose a tag. (Or, retain the whole list for later disambiguation.)

Only 11.5% of English word types in the Brown corpus are ambiguous; but over 40% of the tokens are. This is because the most frequent items are the most likely to be ambiguous.

Taggers can achieve 96-97% "correct" tags (relative to hand-tagging).

Humans have a 96-97% agreement rate (up to 100% if they can discuss the tags).

Lexical categories ("parts of speech")

The following list is given by Jurafsky and Martin, and is based on categories commonly used in linguistics. Note that this is a rather different list from the actual tagsets given in the text. Tagsets distinguish morphological forms and include tags for punctuation and symbols; certain very common items have their own tags.

• Open classes
  • Nouns
    • Proper nouns
    • Common nouns
      • Mass nouns
      • Count nouns
  • Verbs
    • Auxiliaries
  • Adjectives
  • Adverbs
    • Directional
    • Degree
    • Locative
    • Manner
    • Temporal
• Closed classes
  • Prepositions
  • Determiners
  • Pronouns
    • Personal pronouns
    • Possessive pronouns
    • Wh-pronouns
  • Conjunctions
    • Complementizers
  • Auxiliary verbs
    • Be
    • Do
    • Have
    • Modals
  • Particles
  • Numerals
  • Interjections
  • Negatives
  • Politeness markers
  • Greetings
  • "Existential there"

Theoretical problems:

Note that this approach makes several questionable assumptions:

• The lexicon can be exhaustively divided into a single set of "parts of speech". Most tagsets are "flat": don't allow cross-classification. This probably doesn't reflect the best theory of how the lexicon is organized: words can be classified in various ways, with no one system as primary. For instance, the verb "think" and the noun "idea" both belong to the category of items that allow a "that-complement".
• There is one correct tagset for a language (independent of syntactic theory!)
• Given the tagset, there's just one best tag for any item (some systems may save multiple tags): categories aren't fuzzy or overlapping.
• This approach works best with relatively isolating languages. Note issues from crosslinguistic variation: heavily inflected languages, languages with low categoriality (those where stems can be inflected in any word class)