MorphSegmenter_Example-Lezgi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic Prediction of Lezgi Morpheme Breaks\n",
    "\n",
    "This program does supervised morphological segmentation of all morphemes and glossing of either just affixes or affixes and stems. This classifier can be used on any language but it expects cleanly segmented, glossed, and POS-tagged (lexical categories) as training data. \n",
    "\n",
    "This program is intended to quickly increase the amount of accessible data from low resource, and often endangered, languages.  It is considered successful if it reaches 80% accuracy. This goal comes from the Pareto Principle - the idea that 20% of one's effort produces 80% of one's results, and vice versa. This program should accurately complete 80% of the annotations, leaving the most interesting and informative 20% for the human linguist to complete.This project was inspired by an ongoing fieldwork project. A native Lezgi speaker who has no background in linguistics has been annotating the collection of texts. She has quickly learned basic morphology and gained FLEx skills. However, simultaneously learning and doing basic linguistic analysis produces inaccurate and inconsistent annotations. It is also time-consuming. Many of the mistakes are due to the repetitive nature of the work. Not every part of speech has inflectional morphology. The annotator is most likely to skip over essential words with simple morphology, such as ergative case-marked arguments, and concentrate on morphologicaly complex words. \n",
    "\n",
    "Once the training is complete, the program should predict morpheme breaks and affix glosses for any text that has been labeled with parts of speech. Identifying parts of speech is required because this seems a reasonable task for a non-linguist native speaker. The data used in this example does include two distinctions in Lezgi that might be difficult without linguistic training. Participles are distinguished from verbs, but Lezgi participles end in a unique letter. Demonstrative pronouns are distinguished from pronouns. This distinction was used primarily because it was already consistently annotated in the data. \n",
    "\n",
    "This example uses Lezgi [lez], a Nakh-Daghestanian language spoken in Russia and Azerbaijan. Lezgi is an agglutinating language that is overwhelmingly suffixing. The training and test data came from a collection of 21 transcribed oral narratives spoken in the Qusar dialect of northwest Azerbaijan. Nine texts with about 2,500 words were used for training data after having been cleanly annotated with morpheme breaks and part of speech. All but three of affixes were glossed. Many of the stems are not glossed. The FlexText XML export labels each morpheme as stem, suffix, or prefix. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing Data\n",
    " \n",
    "This process assumes that 1) the data has been analyzed in FLEx and exported as a FlexText, then saved with an .xml file extension, 2) words have been annotated in FLEx for part of speech, (for the example - verb, participle, adjective, adverb, noun/proper noun, particle, (personal) pronoun, demonstrative, and postposition), 3) morpheme breaks are consistent, and 4) all affixes, but not necessarily stems, are glossed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#API for parsing XML docs\n",
    "import xml.etree.ElementTree as ET\n",
    "from itertools import chain\n",
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelBinarizer\n",
    "import sklearn\n",
    "import pycrfsuite\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function below takes the data from the FLExText XML export. The data is read from the XML file and broken down by morphemes. The POS and a \"label\" is associated with each word. The label for stems is \"stem\" if stems are not being glossed, otherwise it is their gloss as it always is for affixes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def XMLtoWords(filename,stems=False):\n",
    "    '''Takes FLExText text as .xml. Returns data as list: [[[[[[morpheme, gloss], pos],...],lexemes],sents]].\n",
    "    Ignores punctuation. Morph_types can be: stem, suffix, prefix, or phrase when lexical item is made up of two words.'''\n",
    "    \n",
    "    datalists = []\n",
    "\n",
    "    #open XML doc using xml parser\n",
    "    root = ET.parse(filename).getroot()\n",
    "\n",
    "    for text in root:\n",
    "        for paragraphs in text:\n",
    "            #Only get paragraphs, ignore metadata.\n",
    "            if paragraphs.tag == 'paragraphs':\n",
    "                for paragraph in paragraphs:\n",
    "                    #jump straight into items under phrases\n",
    "                    for phrase in paragraph[0]:\n",
    "                        sent = []\n",
    "                        #ignore first item which is the sentence number\n",
    "                        for word in phrase[1]:\n",
    "                            #ignore punctuation tags which have no attributes\n",
    "                            if word.attrib:\n",
    "                                lexeme = []\n",
    "                                for node in word:\n",
    "                                    if node.tag == 'morphemes':\n",
    "                                        for morph in node:\n",
    "                                            morpheme = []\n",
    "                                            #note morph type \n",
    "                                            morph_type = morph.get('type')\n",
    "                                            #Treat MWEs or unlabeled morphemes as stems.\n",
    "                                            if morph_type == None or morph_type == 'phrase':\n",
    "                                                morph_type = 'stem'                                            \n",
    "                                            for item in morph:\n",
    "                                                #get morpheme token\n",
    "                                                if item.get('type') == 'txt':\n",
    "                                                    form = item.text\n",
    "                                                    #without hyphens demarcating affixes\n",
    "                                                    if morph_type == 'suffix':\n",
    "                                                        form = form[1:]\n",
    "                                                    if morph_type == 'prefix':\n",
    "                                                        form = form[:-1]\n",
    "                                                    morpheme.append(form)\n",
    "                                                #get affix glosses\n",
    "                                                if item.get('type') == 'gls' and morph_type != 'stem':\n",
    "                                                    morpheme.append(item.text)\n",
    "                                                #get stem \"gloss\":'stem' or actual gloss\n",
    "                                                    if item.get('type') == 'gls' and morph_type == 'stem':\n",
    "                                                        if not stems:\n",
    "                                                            morpheme.append(morph_type)\n",
    "                                                        else:\n",
    "                                                            morpheme.append(item.text)\n",
    "                                            lexeme.append(morpheme)\n",
    "                                    #get word's POS\n",
    "                                    if node.get('type') == 'pos':\n",
    "                                        lexeme.append(node.text)\n",
    "                                sent.append(lexeme)\n",
    "                        datalists.append(sent)\n",
    "    return datalists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the morphemes are broken down by letter. Each letter is associated with the word's POS tag and a BIO tag is added to its label/gloss. \"B\" denotes the initial letter of a morpheme. \"I\" marks non-initial letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def WordsToLetter(wordlists):\n",
    "    '''Takes data from XMLtoWords: [[[[[[morpheme, gloss], pos],...],words],sents]]. \n",
    "    Returns [[[[[letter, POS, BIO-label],...],words],sents]]'''\n",
    "\n",
    "    letterlists = []\n",
    "    \n",
    "    for phrase in wordlists:\n",
    "        sent = []\n",
    "        for lexeme in phrase:\n",
    "            word = []\n",
    "            #Skip POS label\n",
    "            for morpheme in lexeme[:-1]:\n",
    "                #use gloss as BIO label\n",
    "                label = morpheme[1]\n",
    "                #Break morphemes into letters\n",
    "                for i in range(len(morpheme[0])):\n",
    "                    letter = [morpheme[0][i]]\n",
    "                    #add POS label to each letter\n",
    "                    letter.append(lexeme[-1])\n",
    "                    #add BIO label\n",
    "                    if i == 0:\n",
    "                        letter.append('B-' + label)\n",
    "                    else:\n",
    "                        letter.append('I-' + label)\n",
    "                        #letter.append('I')\n",
    "                    word.append(letter)\n",
    "            sent.append(word)\n",
    "        letterlists.append(sent)\n",
    "    \n",
    "    return letterlists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below two functions above are called and the data is split into train/test sets.\n",
    "\n",
    "With a corpus of a little less than 2,500 words, I originally tried a 90/10 split. The accuracy results ranged from 92% to 97% but the test data was seeing a dozen or less labels. An 80/20 random split results has less range in accuracy (within 2%) but still averages 94%. The number of labels the test data encounters, however, is nearly doubled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#Randomize and split the data\n",
    "traindata,testdata = train_test_split(WordsToLetter(XMLtoWords(\"FLExTxtExport2.xml\")),test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CRFSuite \n",
    "### Define Features\n",
    "\n",
    "It is assumed that a \"phrase\" in FLEx is equivalent to a complete sentence. In reality, some \"phrases\" contain more than one sentence, some contain only a sentence fragment. This means that the word position in the sentence is often inaccurate, but it was retained to take into account Lezgi's strong tendency for verb-final word order. Affixes are rarely more than 3 letters long, so features include the previous and next 1-4 letters. This ensures that the program is viewing at least one letter in the previous/next morpheme. More often it is viewing the whole previous/next 1-2 morphemes. \n",
    "\n",
    "Since Lezgi is primarily suffixing, the position of a letter in a word is counted from the end of the word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def extractFeatures(sent):\n",
    "    '''Takes data as [[[[[letter, POS, BIO-label],...],words],sents]].\n",
    "    Returns list of words with characters as features list: [[[[[letterfeatures],POS,BIO-label],letters],words]]'''\n",
    "    \n",
    "    featurelist = []\n",
    "    senlen = len(sent)\n",
    "    \n",
    "    #each word in a sentence\n",
    "    for i in range(senlen):\n",
    "        word = sent[i]\n",
    "        wordlen = len(word)\n",
    "        lettersequence = ''\n",
    "        #each letter in a word\n",
    "        for j in range(wordlen):\n",
    "            letter = word[j][0]\n",
    "            #gathering previous letters\n",
    "            lettersequence += letter\n",
    "            #ignore digits             \n",
    "            if not letter.isdigit():\n",
    "                features = [\n",
    "                    'bias',\n",
    "                    'letterLowercase=' + letter.lower(),\n",
    "                    'postag=' + word[j][1],\n",
    "                ] \n",
    "                #position of word in sentence and pos tags sequence\n",
    "                if i > 0:\n",
    "                    features.append('prevpostag=' + sent[i-1][0][1])\n",
    "                    if i != senlen-1:\n",
    "                        features.append('nxtpostag=' + sent[i+1][0][1])\n",
    "                    else:\n",
    "                        features.append('EOS')\n",
    "                else:\n",
    "                    features.append('BOS')\n",
    "                    #Don't get pos tag if sentence is 1 word long\n",
    "                    if i != senlen-1:\n",
    "                        features.append('nxtpostag=' + sent[i+1][0][1])\n",
    "                #position of letter in word\n",
    "                if j == 0:\n",
    "                    features.append('BOW')\n",
    "                elif j == wordlen-1:\n",
    "                    features.append('EOW')\n",
    "                else:\n",
    "                    features.append('letterposition=-%s' % str(wordlen-1-j))\n",
    "                #letter sequences before letter\n",
    "                if j >= 4:\n",
    "                    features.append('prev4letters=' + lettersequence[j-4:j].lower() + '>')\n",
    "                if j >= 3:\n",
    "                    features.append('prev3letters=' + lettersequence[j-3:j].lower() + '>')\n",
    "                if j >= 2:\n",
    "                    features.append('prev2letters=' + lettersequence[j-2:j].lower() + '>')\n",
    "                if j >= 1:\n",
    "                    features.append('prevletter=' + lettersequence[j-1:j].lower() + '>')\n",
    "                #letter sequences after letter\n",
    "                if j <= wordlen-2:\n",
    "                    nxtlets = word[j+1][0]\n",
    "                    features.append('nxtletter=<' + nxtlets.lower())\n",
    "                    #print('\\nnextletter:', nxtlet)\n",
    "                if j <= wordlen-3:\n",
    "                    nxtlets += word[j+2][0]\n",
    "                    features.append('nxt2letters=<' + nxtlets.lower())\n",
    "                    #print('next2let:', nxt2let)\n",
    "                if j <= wordlen-4:\n",
    "                    nxtlets += word[j+3][0]\n",
    "                    features.append('nxt3letters=<' + nxtlets.lower())\n",
    "                if j <= wordlen-5:\n",
    "                    nxtlets += word[j+4][0]\n",
    "                    features.append('nxt4letters=<' + nxtlets.lower())\n",
    "                \n",
    "            featurelist.append(features)\n",
    "    \n",
    "    return featurelist\n",
    "\n",
    "def extractLabels(sent):\n",
    "    labels = []\n",
    "    for word in sent:\n",
    "        for letter in word:\n",
    "            labels.append(letter[2])\n",
    "    return labels\n",
    "\n",
    "def extractTokens(sent):\n",
    "    tokens = []\n",
    "    for word in sent:\n",
    "        for letter in word:\n",
    "            tokens.append(letter[0])\n",
    "    return tokens\n",
    "\n",
    "def sent2features(data):\n",
    "    return [extractFeatures(sent) for sent in data]\n",
    "\n",
    "def sent2labels(data):\n",
    "    return [extractLabels(sent) for sent in data]\n",
    "\n",
    "def sent2tokens(data):\n",
    "    return [extractTokens(sent) for sent in data]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train = sent2features(traindata)\n",
    "Y_train = sent2labels(traindata)\n",
    "\n",
    "X_test = sent2features(testdata)\n",
    "Y_test = sent2labels(testdata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "trainer = pycrfsuite.Trainer(verbose=False)\n",
    "\n",
    "for xseq, yseq in zip(X_train, Y_train):\n",
    "    trainer.append(xseq, yseq)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set training parameters. L-BFGS (what is this) is default. Using Elastic Net (L1 + L2) regularization [ditto?]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trainer.set_params({\n",
    "        'c1': 1.0, #coefficient for L1 penalty\n",
    "        'c2': 1e-3, #coefficient for L2 penalty\n",
    "        'max_iterations': 50 #early stopping\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The program saves the trained model to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_filename = 'LING5800_lezgi.crfsuite'\n",
    "trainer.train(model_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<contextlib.closing at 0x21205f22048>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tagger = pycrfsuite.Tagger()\n",
    "tagger.open(model_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's use the trained model to make predications for just one example sentence from the test data. The predicted labels are printed out for comparison above the correct labels. Most examples have 100% accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Letters: х  ъ  с  а  н  т  о  с  т  а  р  л  а  г  ь  а  н  а  р  а  з  и  в  и  л  е  р  х  ь  а  н  а  ч  у  н  м  е  р  е  к  а  д  и  л  а  й  р  а  з  и  а  м  у  к  ь  н  а\n",
      "Predicted: B-stem I-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem I-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem I-stem B-OBL I-OBL B-SUPER B-ELAT I-ELAT B-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR\n",
      "Correct: B-stem I-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem I-stem I-stem I-stem I-stem B-PL I-PL B-stem I-stem I-stem B-AOR I-AOR B-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem I-stem B-OBL I-OBL B-SUPER B-ELAT I-ELAT B-stem I-stem I-stem I-stem B-stem I-stem I-stem I-stem I-stem B-AOR I-AOR\n"
     ]
    }
   ],
   "source": [
    "example_sent = testdata[0]\n",
    "print('Letters:', '  '.join(extractTokens(example_sent)), end='\\n')\n",
    "\n",
    "print('Predicted:', ' '.join(tagger.tag(extractFeatures(example_sent))))\n",
    "print('Correct:', ' '.join(extractLabels(example_sent)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the Model\n",
    "\n",
    "The following function will evaluate how well the model performs. Unlike CRF example found at https://github.com/scrapinghub/python-crfsuite/blob/master/examples/CoNLL%202002.ipynb, this model is not designed to disregard \"O\" labels, since all characters that are not part of a word (e.g. digits and punctuation) are already eliminated during pre-processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def bio_classification_report(y_correct, y_pred):\n",
    "    '''Takes list of correct and predicted labels from tagger.tag. \n",
    "    Prints a classification report for a list of BIO-encoded sequences.\n",
    "    It computes letter-level metrics.'''\n",
    "\n",
    "    labeler = LabelBinarizer()\n",
    "    y_correct_combined = labeler.fit_transform(list(chain.from_iterable(y_correct)))\n",
    "    y_pred_combined = labeler.transform(list(chain.from_iterable(y_pred)))\n",
    "    \n",
    "    tagset = set(labeler.classes_)\n",
    "    tagset = sorted(tagset, key=lambda tag: tag.split('-', 1)[::-1])\n",
    "    class_indices = {cls: idx for idx, cls in enumerate(labeler.classes_)}\n",
    "    \n",
    "    return classification_report(\n",
    "        y_correct_combined,\n",
    "        y_pred_combined,\n",
    "        labels = [class_indices[cls] for cls in tagset],\n",
    "        target_names = tagset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will predict BIO labels in the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Y_pred = [tagger.tag(xseq) for xseq in X_test]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get results for labeled position evaluation. This evaluates how well the classifier performed on each morpheme as a whole and their tags, rather than evaluating character-level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def concatenateLabels(y_list):\n",
    "    '''Return list of morpheme labels [[B-label, I-label,...]morph,[B-label,...]]'''\n",
    "    \n",
    "    morphs_list = []\n",
    "    labels_list = []\n",
    "    morph = []\n",
    "    for sent in y_list:\n",
    "        for label in sent:\n",
    "            labels_list.append(label)\n",
    "            if label[0] == 'I':\n",
    "                #build morpheme shape, adding to first letter\n",
    "                morph.append(label)\n",
    "            else:\n",
    "                # Once processed first morph, add new morphemes & gloss labels to output\n",
    "                if morph:\n",
    "                    morphs_list.append(morph)\n",
    "                #Extract morpheme features\n",
    "                morph = [label]\n",
    "    \n",
    "    return morphs_list, labels_list\n",
    "\n",
    "def countMorphemes(morphlist):\n",
    "    counts = {}\n",
    "    for morpheme in morphlist:\n",
    "        counts[morpheme[0][2:]] = counts.get(morpheme[0][2:], 0) + 1\n",
    "    return counts\n",
    "\n",
    "def eval_labeled_positions(y_correct, y_pred):\n",
    "    \n",
    "    #group the labels by morpheme and get list of morphemes\n",
    "    correctmorphs,_ = concatenateLabels(y_correct)\n",
    "    predmorphs,predLabels = concatenateLabels(y_pred)\n",
    "    #Count instances of each morpheme\n",
    "    test_morphcts = countMorphemes(correctmorphs)\n",
    "    pred_morphcts = countMorphemes(predmorphs)\n",
    "    \n",
    "    correctMorphemects = {}\n",
    "    idx = 0\n",
    "    num_correct = 0\n",
    "    for morpheme in correctmorphs:\n",
    "        correct = True\n",
    "        for label in morpheme:\n",
    "            if label != predLabels[idx]:\n",
    "                correct = False\n",
    "            idx += 1\n",
    "        if correct == True:\n",
    "            num_correct += 1\n",
    "            correctMorphemects[morpheme[0][2:]] = correctMorphemects.get(morpheme[0][2:], 0) + 1\n",
    "    #calculate P, R F1 for each morpheme\n",
    "    results = ''\n",
    "    for firstlabel in correctMorphemects.keys():\n",
    "        lprec = correctMorphemects[firstlabel]/pred_morphcts[firstlabel]\n",
    "        lrecall = correctMorphemects[firstlabel]/test_morphcts[firstlabel]\n",
    "        results += firstlabel + '\\t\\t{0:.2f}'.format(lprec) + '\\t\\t' + '{0:.2f}'.format(lrecall) + '\\t' + '{0:.2f}'.format((2*lprec*lrecall)/(lprec+lrecall)) +'\\t\\t' + str(test_morphcts[firstlabel]) + '\\n'\n",
    "    #overall results\n",
    "    precision = num_correct/len(predmorphs)\n",
    "    recall = num_correct/len(correctmorphs)\n",
    "    \n",
    "    print('\\t\\tPrecision\\tRecall\\tf1-score\\tInstances\\n\\n' + results + '\\ntotal/avg\\t{0:.2f}'.format(precision) + '\\t\\t' + '{0:.2f}'.format(recall) + '\\t' + '{0:.2f}'.format((2*precision*recall)/(precision+recall)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we check the results and print a report of the results. These results are for character level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\tPrecision\tRecall\tf1-score\tInstances\n",
      "\n",
      "ELAT\t\t1.00\t\t1.00\t1.00\t\t3\n",
      "GEN\t\t0.80\t\t0.57\t0.67\t\t7\n",
      "PERF\t\t1.00\t\t1.00\t1.00\t\t4\n",
      "OBL\t\t0.88\t\t0.71\t0.79\t\t21\n",
      "SBST\t\t1.00\t\t1.00\t1.00\t\t2\n",
      "DAT\t\t1.00\t\t0.86\t0.92\t\t14\n",
      "SUB\t\t1.00\t\t1.00\t1.00\t\t1\n",
      "AOR\t\t0.95\t\t1.00\t0.98\t\t20\n",
      "SUPER\t\t1.00\t\t1.00\t1.00\t\t2\n",
      "IMPF\t\t1.00\t\t1.00\t1.00\t\t1\n",
      "TEMP\t\t1.00\t\t0.50\t0.67\t\t2\n",
      "PTP\t\t0.57\t\t1.00\t0.73\t\t4\n",
      "NEG\t\t1.00\t\t1.00\t1.00\t\t1\n",
      "stem\t\t0.99\t\t0.98\t0.98\t\t164\n",
      "INF\t\t1.00\t\t1.00\t1.00\t\t2\n",
      "FOC\t\t0.90\t\t1.00\t0.95\t\t9\n",
      "PL\t\t1.00\t\t1.00\t1.00\t\t4\n",
      "\n",
      "total/avg\t0.94\t\t0.88\t0.91\n"
     ]
    }
   ],
   "source": [
    "eval_labeled_positions(Y_test, Y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "     B-ADSS       1.00      1.00      1.00         1\n",
      "      B-AOR       0.92      1.00      0.96        12\n",
      "      B-DAT       0.86      0.75      0.80         8\n",
      "     B-ELAT       1.00      1.00      1.00         2\n",
      "      B-ERG       0.00      0.00      0.00         2\n",
      "      B-FOC       0.86      1.00      0.92         6\n",
      "      B-FUT       0.00      0.00      0.00         2\n",
      "      B-GEN       1.00      0.75      0.86         4\n",
      "          I       0.95      0.99      0.97       417\n",
      "      B-IMC       0.00      0.00      0.00         1\n",
      "     B-IMPF       1.00      0.75      0.86         4\n",
      "    B-INESS       0.00      0.00      0.00         1\n",
      "      B-INF       1.00      1.00      1.00         1\n",
      "      B-NEG       1.00      1.00      1.00         2\n",
      "  B-NEG.PST       0.00      0.00      0.00         1\n",
      "      B-OBL       0.80      0.67      0.73        12\n",
      "     B-PERF       1.00      0.75      0.86         4\n",
      "       B-PL       0.89      0.67      0.76        12\n",
      "      B-PTP       1.00      1.00      1.00         2\n",
      "     B-PURP       0.00      0.00      0.00         1\n",
      "        B-Q       0.00      0.00      0.00         1\n",
      "     B-SBSS       0.00      0.00      0.00         1\n",
      "      B-SUB       1.00      1.00      1.00         1\n",
      "    B-SUPER       1.00      1.00      1.00         1\n",
      "     B-stem       1.00      0.98      0.99       124\n",
      "\n",
      "avg / total       0.94      0.96      0.95       623\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Sarah R M\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "print(bio_classification_report(Y_test, Y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model, with a 80/20 split, produces an average accuracy of 94% with a less than 2% range over randomized test data. This is significantly above the targeted accuracy of 80%. Table 1 shows the results of one run. \n",
    "\n",
    "|__label__|__precision__|__recall__|__f1-score__|__instances__|\n",
    "|---------|-------------|----------|------------|-------------|\n",
    "|B-AOR|1.00|0.88|0.94|17|\n",
    "|B-DAT|0.92|1.00|0.96|11|\n",
    "|B-ELAT|0.67|1.00|0.80|2|\n",
    "|B-ENT|0.33|0.50|0.40|2|\n",
    "|B-ERG|0.00|0.00|0.00|3|\n",
    "|B-FOC|0.86|1.00|0.92|6|\n",
    "|B-FUT|0.00|0.00|0.00|2|\n",
    "|B-GEN|0.50|0.33|0.40|6|\n",
    "|B-HORT|0.00|0.00|0.00|1|\n",
    "|I|0.95|0.99|0.97|480|\n",
    "|B-INESS|1.00|0.33|0.50|3|\n",
    "|B-MSDR|0.00|0.00|0.00|2|\n",
    "|B-NEG|1.00|1.00|1.00|1|\n",
    "|B-OBL|0.80|0.60|0.69|20|\n",
    "|B-PL|0.50|0.50|0.50|2|\n",
    "|B-POESS|1.00|1.00|1.00|3|\n",
    "|B-PTP|1.00|0.67|0.80|3|\n",
    "|B-SBST|1.00|0.50|0.67|2|\n",
    "|B-SUPER|0.67|1.00|0.80|2|\n",
    "|B-TEMP|0.00|0.00|0.00|1|\n",
    "|B-UNK|0.00|0.00|0.00|1|\n",
    "|B-stem|1.00|0.99|0.99|138|\n",
    "|__avg / total__|__0.94__|__0.94__|__0.94__|__708__|\n",
    "\n",
    "<center>Table 1: Results of morpheme predictions</center>\n",
    "\n",
    "As might be expected, the classifier has less success predicting less frequent labels. This makes the results of the I labels (non-initial letters in a morpheme) surprising, until one considers that transitions between morphemes may not always be clear. Other results become more interesting with some knowledge of Lezgi morphology. The inessive (INESS) and ergative (ERG) case and the oblique stem morpheme (OBL) are identical. The only difference between the first two is the tendency of sentence position, even with Lezgi's free word order. The difference between the latter two is that the ergative morpheme is word final and the the oblique stem is follow by another case morpheme. \n",
    "\n",
    "|__precision__|__recall__|__f1-score__|\n",
    "|-----|------|------|\n",
    "|0.54|0.49|0.49|\n",
    "\n",
    "<center>Table 2: Average score of affix labels only.</center>\n",
    "\n",
    "The classifier has most success identifying stem morphemes (STEM) and non-initial letters (I), the majority of which belong to stem morphemes. It has less success with identifying affixes. The classifier is clearly adept at splitting affixes from stems and this is already helpful to human annotators but it would be less helpful splitting strings of affixes and correcly glossing them. Table 2 shows average precision, recall, and f1-score of affix labels is much less accurate than the overall accuracy. This is most likely due in part to homonymic affixes and in part to the fewer instances of affixes compared to stems. As the more texts are correctly annotated with the help of the model, more data can be fed into the training, hopefully increasing the accuracy and incrementally speeding the annotation process.\n",
    "\n",
    "The data was also run on a bidirectional sequence-to-sequence deep neural network with attention. The hidden layer size was set at 128, the batch size as 32, the teacher forcing ratio at 0.5. The results in Table 3 indicate that with a small amount of data a supervised classifier can produce equal or better results than a neural network.\n",
    "\n",
    "|epochs|accuracy|\n",
    "|------|--------|\n",
    "|50|0.57|\n",
    "|100|0.75|\n",
    "|200|0.90|\n",
    "|300|0.92|\n",
    "|__500__|__0.93__|\n",
    "|600|0.89|\n",
    "|1000|0.91|\n",
    "\n",
    "<center>Table 3: Results of deep neural network</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What the Classifier Learned\n",
    "\n",
    "By using methods of the crfsuite, we can look insider classifier and see what it learned. From the example printout in Table 3, we can see, for example, that the stem, elative (ELAT), imperfective (IMPF), aorist (AOR), perfective (PF), and plural (PL) morphemes most often consist of more than one letter  but superessive (SUPER), oblique (OBL), and subessive (SUB) morphemes usually consist of just one letter. We can also see that temporal converb (TEMP) morpheme often follows the participle (PTP) morpheme, and another case morpheme tends to follow the oblique, superessive, and subessive case morphemes. These patterns correspond to the facts of Lezgi morphology. On the other hand, both Table 4 and Table 5 indciate that is highly likelythat a  genitive case (GEN) morpheme will be a prefix, which is impossible. This indicates that the affix type (prefix or suffix) might be a useful feature to include.\n",
    "\n",
    "|-|-|-|weights|\n",
    "|---|---|----|-----|\n",
    "|B-SUPER| ->| B-ELAT|  4.820010|\n",
    "|B-OBL|  ->| B-SPSS|  3.806645|\n",
    "|B-SUB|  ->| B-ELAT|  3.444584|\n",
    "|B-OBL|  ->| B-DAT|   2.946830|\n",
    "|B-stem| ->| I|       2.258064|\n",
    "|B-OBL|  ->| B-GEN|   2.247354|\n",
    "|I|      ->| B-OBL|   1.913825|\n",
    "|B-stem| ->| B-OBL|   1.862016|\n",
    "|B-ELAT| ->| I|       1.711584|\n",
    "|B-PTP|  ->| B-TEMP|  1.620690|\n",
    "|B-IMPF| ->| I|       1.300227|\n",
    "|B-AOR|  ->| I|       1.252594|\n",
    "|B-PERF| ->| I|       1.135483|\n",
    "|B-PL|   ->| I|       1.043438|\n",
    "|B-GEN|  ->| B-stem|  0.956780|\n",
    "\n",
    "<center>Table 4: Top most likely transitions</center>\n",
    "\n",
    "On the other hand, Table 5, for example, indicates that the negative affix rarely follows a non-initial letter of another morpheme. This is accurate because the negative affix is the only prefix in the language. It is not surprising that the transition still has a greater than zero probability since it is often only one letter long and this letter may be found at the beginning of any word.\n",
    "\n",
    "|-|-|-|weights|\n",
    "|---|---|---|----|\n",
    "|B-ERG|  ->| B-stem|  0.295926|\n",
    "|B-TEMP| ->| I|       0.254567|\n",
    "|B-SBST| ->| I|       0.249661|\n",
    "|I|      ->| B-NEG|   0.221662|\n",
    "|B-INF|  ->| B-stem|  0.196340|\n",
    "|I|      ->| B-DAT|   0.057729|\n",
    "|B-NEG|  ->| B-stem  |0.013683|\n",
    "|I|      ->| B-stem|  0.009557|\n",
    "|I|      ->| B-ERG|   0.000074|\n",
    "|I|      ->| B-SUPER| -0.000692|\n",
    "|I|      ->| B-FOC|   -0.003919|\n",
    "|I|      ->| B-SBST|  -0.023268|\n",
    "|B-OBL|  ->| I|       -0.034257|\n",
    "|B-INESS| ->| I|       -0.157967|\n",
    "|I|      ->| B-GEN|   -1.180139|\n",
    "\n",
    "<center>Table 5: Top most unlikely transitions</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top likely transitions:\n",
      "B-SUPER -> B-ELAT  4.491556\n",
      "B-OBL  -> B-SPSS  3.999741\n",
      "B-SUB  -> B-ELAT  2.783580\n",
      "B-OBL  -> B-GEN   2.581786\n",
      "B-stem -> I       2.441328\n",
      "B-OBL  -> B-DAT   2.354065\n",
      "I      -> B-OBL   2.319193\n",
      "B-ELAT -> I       2.027671\n",
      "B-AOR  -> I       1.810235\n",
      "B-OBL  -> B-SUPER 1.537023\n",
      "B-IMPF -> I       1.262218\n",
      "B-GEN  -> B-stem  1.204093\n",
      "B-PL   -> I       1.181239\n",
      "B-PERF -> I       1.172315\n",
      "B-ERG  -> B-stem  1.059358\n",
      "\n",
      "Top unlikely transitions:\n",
      "B-stem -> B-OBL   0.263636\n",
      "B-stem -> B-PL    0.178466\n",
      "B-POESS -> I       0.174021\n",
      "B-FOC  -> I       0.163086\n",
      "I      -> B-stem  0.081534\n",
      "B-NEG  -> B-stem  0.062830\n",
      "B-INESS -> B-stem  0.062172\n",
      "I      -> B-AOR   0.033769\n",
      "B-ENT  -> I       0.009679\n",
      "B-PTP  -> B-stem  0.005626\n",
      "B-GEN  -> B-FOC   0.000337\n",
      "B-OBL  -> I       -0.004404\n",
      "I      -> B-SBST  -0.031958\n",
      "B-INESS -> I       -0.095406\n",
      "I      -> B-GEN   -0.620703\n"
     ]
    }
   ],
   "source": [
    "info = tagger.info()\n",
    "\n",
    "def print_transitions(trans_features):\n",
    "    '''Print info from the crfsuite.'''\n",
    "    \n",
    "    for (label_from, label_to), weight in trans_features:\n",
    "        print(\"%-6s -> %-7s %0.6f\" % (label_from, label_to, weight))\n",
    "\n",
    "print(\"Top likely transitions:\")\n",
    "print_transitions(Counter(info.transitions).most_common(15))\n",
    "\n",
    "print(\"\\nTop unlikely transitions:\")\n",
    "print_transitions(Counter(info.transitions).most_common()[-15:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make some observations about the state features. For example, Table 6 indicates that the model rightly recognized that the stem is nearly always at the beginning of the word and there are no consistent feature to identify the non-initial letters of various morphemes. \n",
    "\n",
    "|weight|label|feature|\n",
    "|---|---|----|\n",
    "|13.385742| B-stem| BOW|\n",
    "|6.80475| I|      bias|\n",
    "|5.169367| B-PL|   nxt2letters=<ур|\n",
    "|5.142534| B-DAT|  letterLowercase=з|\n",
    "|4.858094| B-NEG|  letterLowercase=ш|\n",
    "|4.568794| B-PTP|  letterLowercase=й|\n",
    "|4.513613| B-PST|  letterLowercase=й|\n",
    "|4.361416| B-ADSS| letterLowercase=в|\n",
    "|4.269127| B-PL|   nxtletter=<р|\n",
    "|4.216564| B-FOC|  nxtletter=<и|\n",
    "|4.203677| B-GEN|  letterLowercase=н|\n",
    "|4.023482| B-INF|  letterLowercase=з|\n",
    "|3.977504| B-IMPF| letterLowercase=з|\n",
    "|3.868088| B-NEG|  letterLowercase=ч|\n",
    "|3.636859| B-FOC|  letterLowercase=н|\n",
    "\n",
    "<center>Table 6: Top positive features</center>\n",
    "\n",
    "Table 7 indicates that certain letter sequences might be less likely to begin a morpheme. One interesting observation that could be easily confirmed by a corpus study is that the focus particle is least likely to occur on a verb than on any other lexical category. \n",
    "\n",
    "|weight|label|feature|\n",
    "|---|----|---|\n",
    "|-0.606766| I|      prev2letters=ча>|\n",
    "|-0.679616| I|      letterLowercase=ч|\n",
    "|-0.704380| I|      prevletter=ш>|\n",
    "|-0.741532| I|      prev2letters=ич>|\n",
    "|-0.833423| B-FOC|  postag=v|\n",
    "|-0.937032| B-FOC|  bias|\n",
    "|-1.029693| I|      prev3letters=вал>|\n",
    "|-1.071785| I|      nxtletter=<й|\n",
    "|-1.073034| I|      prev3letters=гьу>|\n",
    "|-1.126576| I|      prev2letters=ди>|\n",
    "|-1.150632| B-AOR|  bias|\n",
    "|-1.201650| I|      letterLowercase=н|\n",
    "|-1.240373| I|      letterLowercase=з|\n",
    "|-1.250568| I|      prevletter=р>|\n",
    "\n",
    "<center>Table 7: Top negative</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top positive:\n",
      "13.322432 B-stem BOW\n",
      "5.645198 I      bias\n",
      "5.374702 B-DAT  letterLowercase=з\n",
      "5.080712 B-PL   nxt2letters=<ур\n",
      "5.016308 B-NEG  letterLowercase=ш\n",
      "4.591081 B-IMPF letterLowercase=з\n",
      "4.378502 B-AOR  letterLowercase=н\n",
      "4.364921 B-GEN  letterLowercase=н\n",
      "4.319225 B-PL   nxtletter=<р\n",
      "4.172178 B-FOC  letterLowercase=н\n",
      "4.108242 B-FOC  nxtletter=<и\n",
      "4.009077 B-INF  letterLowercase=з\n",
      "3.860550 B-ADSS letterLowercase=в\n",
      "3.858400 B-MSDR letterLowercase=н\n",
      "3.552755 B-INELAT nxtletter=<й\n",
      "\n",
      "Top negative:\n",
      "-0.819617 I      letterLowercase=з\n",
      "-0.855908 B-FOC  bias\n",
      "-1.046632 I      prev2letters=ча>\n",
      "-1.071244 I      prev2letters=уш>\n",
      "-1.171039 I      nxt2letters=<ур\n",
      "-1.234349 I      prev2letters=ил>\n",
      "-1.320892 I      prev2letters=ун>\n",
      "-1.366267 I      prev2letters=ди>\n",
      "-1.404496 I      prev3letters=вал>\n",
      "-1.575353 I      prev3letters=ава>\n",
      "-1.649316 I      prevletter=р>\n",
      "-1.717653 B-AOR  bias\n",
      "-1.778882 I      prev2letters=да>\n",
      "-1.805477 I      letterLowercase=д\n",
      "-2.029026 I      prev2letters=уд>\n"
     ]
    }
   ],
   "source": [
    "def print_state_features(state_features):\n",
    "    for (attr, label), weight in state_features:\n",
    "        print(\"%0.6f %-6s %s\" % (weight, label, attr))    \n",
    "\n",
    "print(\"Top positive:\")\n",
    "print_state_features(Counter(info.state_features).most_common(15))\n",
    "\n",
    "print(\"\\nTop negative:\")\n",
    "print_state_features(Counter(info.state_features).most_common()[-15:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future steps\n",
    "\n",
    "The goal of this project was to find a way to speed the work on annotator and improve their accuracy. Since the model reach over the 80% accuracy goal, there seems little reason to try to improve the features, although an examination of the transitions and state features point to a few adjustments that might increase accuracy. The bigggest problem seems to be the almost 50% reduction in predicting the affix glosses. However, the small number of instances found in the test data indicate that this will be improved as the amount of supervised examples increases. The model as it is can speed this increase.\n",
    "\n",
    "It should be assumed that few annotators will have programming skills. This is especially true for speakers of minority languages which are often are in areas with limited educational opportunities. The results of this classifier should be checked and corrected by trained annotators. Ideally, this program would be exapnded to write the predicted breaks and glosses to an XML file compatible with FLEx or ELAN or another interface familiar to the annotator or easy to learn. In meantime, the data could be output to an CSV file and presented to the annotator as an spreadsheet.\n",
    "\n",
    "Even with carefully annotated training data by a linguist familiar with FLEX and Lezgi morphology, mistakes were made. A few POS tags and affix glosses were missing. This prevents the program from working, but does not tell the user where or what the missing data are. Pre-processing functions should be adjusted so that they present the troublesome morphemes with glosses as a list to the user so that they can be found and corrected using FLEx's concordance feature. "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}