Added function to calculate Corpus-level BLEU and RIBES #1229
Conversation
Syncing with bleeding edge develop branch
Syncing with bleeding edge develop branch
|
@hoontw Thanks for the review! The |
|
Looks much better than before! Just a few more minor tweaks and it is good to go. |
|
@hoontw sorry for the late commits, I was moving around and at last I've reached where I'm supposed to be =) |
|
Thanks @alvations for the changes. It's great to have corpus-level metrics for MT. |
Added function to calculate Corpus-level BLEU and RIBES
|
Now we can do some tuning using the corpus-level metrics, possibly something like https://github.com/alvations/mosesdecoder/blob/master/scripts/training/propy/pro.py. |
| from nltk.util import ngrams | ||
|
|
||
|
|
||
| def bleu(references, hypothesis, weights): | ||
| def sentence_bleu(references, hypothesis, weights=[0.25, 0.25, 0.25, 0.25]): |
There was a problem hiding this comment.
better set weights to a tuple, not a list, to avoid mutable default parameters.
| for i, _ in enumerate(weights, start=1) | ||
| ) | ||
| # Calculates the modified precision *p_n* for each order of ngram. | ||
| p_ns = [] |
There was a problem hiding this comment.
list comprehension would look much better here.
In the process of adding functions to calculate corpus-level BLEU and RIBES, the following changes were made:
[0.25] * 4._modified_precision()to return numerator and denominator instead of a single-sentence BLEU score, i.e.corpus_bleu()sentence_modified_precision()will return the floating point BLEU score for a single sentence._modified_precision()and_brevity_penalty()returns numerators and denominators. E.g. when someone wants to implement BLEU smoothing from http://acl2014.org/acl2014/W14-33/pdf/W14-3346.pdf.nltk.translatenamespacestranlsate.__init__.pyimports thesentence_bleuas the defaultnltk.translate.bleu. So from the user's end, BLEU still works the same way and calculating corpus-level bleu will require importingfrom nltk.translate.bleu_score import corpus_bleu.corpus_ribes()len(). See line 307 from RIBES v1.03.1 (http://www.kecl.ntt.co.jp/icl/lirg/ribes/)