TextLoadUnify buggy without "--unify"

[phikoehn]

When not using the "--unify" option, there still will be an index of unique items built in TextLoadUnify that is based on unique lines. This messes up (at least) the "--score" option of scoring sentence pairs.

So, this:

for line in fin:
    new_ind = len(sent2ind)
    inds.append(sent2ind.setdefault(line, new_ind))
    if args.unify:
        if inds[-1] == new_ind:
            sents.append(line[:-1])
            nu += 1
    else:
        sents.append(line[:-1])
        nu += 1

should be changed to:

for line in fin:
    if args.unify:
        new_ind = len(sent2ind)
        inds.append(sent2ind.setdefault(line, new_ind))
        if inds[-1] == new_ind:
            sents.append(line[:-1])
            nu += 1
    else:
        sents.append(line[:-1])
        inds.append( nu )
        nu += 1