WMT11

Workshop on Machine Translation


The Sixth Workshop on Machine Translation (WMT11) took place from 30 July to 31 July, 2011, in Edinburgh, United Kingdom. It was organised by WMT.

statmt.org/wmt11/program.html

Table of contents
  1. Tasks
    1. Tunable metrics task
  2. Schedule
    1. Day 1
    2. Day 2
  3. Results
    1. News translation
      1. → English
      2. English →

Tasks

Tunable metrics task

The tunable metrics shared task was invitation-only.

Schedule

Day 1

   
09:00 – 09:25 Session 1: Shared Tasks and Evaluation
Chair: Chris Callison-Burch
A Grain of Salt for the WMT Manual Evaluation
Ondrej Bojar, Miloš Ercegovčević, Martin Popel, Omar Zaidan
09:25 – 09:50 A Lightweight Evaluation Framework for Machine Translation Reordering
David Talbot, Hideto Kazawa, Hiroshi Ichikawa, Jason Katz-Brown, Masakazu Seno, Franz Och
09:50 – 10:30 Findings of the 2011 Workshop on Statistical Machine Translation
Chris Callison-Burch, Philipp Koehn, Christof Monz, Omar Zaidan
10:30 – 11:00 ☕️
11:00 – 12:40 Session 2: Shared Metrics and System Combination Tasks
Poster Session Metrics
  Evaluate with Confidence Estimation: Machine ranking of translation outputs using grammatical features
Eleftherios Avramidis, Maja Popović, David Vilar, Aljoscha Burchardt
  AMBER: A Modified BLEU, Enhanced Ranking Metric
Boxing Chen, Roland Kuhn
  TESLA at WMT 2011: Translation Evaluation and Tunable Metric
Daniel Dahlmeier, Chang Liu, Hwee Tou Ng
  Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems
Michael Denkowski, Alon Lavie
  Approximating a Deep-Syntactic Metric for MT Evaluation and Tuning
Matouš Macháček, Ondrej Bojar
  Evaluation without references: IBM1 scores as evaluation metrics
Maja Popović, David Vilar, Eleftherios Avramidis, Aljoscha Burchardt
  Morphemes and POS tags for n-gram based evaluation metrics
Maja Popović
  E-rating Machine Translation
Kristen Parton, Joel Tetreault, Nitin Madnani, Martin Chodorow
  TINE: A Metric to Assess MT Adequacy
Miguel Rios, Wilker Aziz, Lucia Specia
  Regression and Ranking based Optimisation for Sentence Level MT Evaluation
Xingyi Song, Trevor Cohn
  MAISE: A Flexible, Configurable, Extensible Open Source Package for Mass AI System Evaluation
Omar Zaidan
11:00 – 12:40 Poster Session System Combination
  MANY improvements for WMT11
Loïc Barrault
  The UPV-PRHLT combination system for WMT 2011
Jesús González-Rubio, Francisco Casacuberta
  CMU System Combination in WMT 2011
Kenneth Heafield, Alon Lavie
  The RWTH System Combination System for WMT 2011
Gregor Leusch, Markus Freitag, Hermann Ney
  Expected BLEU Training for Graphs: BBN System Description for WMT11 System Combination Task
Antti-Veikko Rosti, Bing Zhang, Spyros Matsoukas, Richard Schwartz
  The UZH System Combination System for WMT 2011
Rico Sennrich
  Description of the JHU System Combination Scheme for WMT 2011
Daguang Xu, Yuan Cao, Damianos Karakos
12:40 – 14:00 🍴
14:00 – 14:25 Session 3: Language Models and Document Alignment
Chair: David Talbot
Multiple-stream Language Models for Statistical Machine Translation
Abby Levenberg, Miles Osborne, David Matthews
14:25 – 14:50 KenLM: Faster and Smaller Language Model Queries
Kenneth Heafield
14:50 – 15:15 Wider Context by Using Bilingual Language Models in Machine Translation
Jan Niehues, Teresa Herrmann, Stephan Vogel, Alex Waibel
15:15 – 15:40 A Minimally Supervised Approach for Detecting and Ranking Document Translation Pairs
Kriste Krstovski, David A. Smith
15:40 – 16:10 ☕️
16:10 – 16:35 Session 4: Syntax and Semantics
Chair: Michel Galley
Agreement Constraints for Statistical Machine Translation into German
Philip Williams, Philipp Koehn
16:35 – 17:00 Fuzzy Syntactic Reordering for Phrase-based Statistical Machine Translation
Jacob Andreas, Nizar Habash, Owen Rambow
17:00 – 17:25 Filtering Antonymous, Trend-Contrasting, and Polarity-Dissimilar Distributional Paraphrases for Improving Statistical Machine Translation
Yuval Marton, Ahmed El Kholy, Nizar Habash
17:25 – 17:50 Productive Generation of Compound Words in Statistical Machine Translation
Sara Stymne, Nicola Cancedda

Day 2

   
09:00 – 10:25 Session 5: Machine Learning and Adaptation
Chair: Phil Blunsom
SampleRank Training for Phrase-Based Machine Translation
Barry Haddow, Abhishek Arun, Philipp Koehn
09:25 – 10:50 Instance Selection for Machine Translation using Feature Decay Algorithms
Ergun Bicici, Deniz Yuret
09:50 – 10:15 Investigations on Translation Model Adaptation Using Monolingual Data
Patrik Lambert, Holger Schwenk, Christophe Servan, Sadaf Abdul-Rauf
10:15 – 10:40 Topic Adaptation for Lecture Translation through Bilingual Latent Semantic Models
Nick Ruiz, Marcello Federico
10:40 – 11:00 ☕️
11:00 – 12:40 Session 6: Shared Translation Task
Poster Presentations
  Personal Translator at WMT2011
Vera Aleksic, Gregor Thurmair
  LIMSI @ WMT11
Alexandre Allauzen, Hélène Bonneau-Maynard, Hai-Son Le, Aurélien Max, Guillaume Wisniewski, François Yvon, Gilles Adda, Josep Maria Crego, Adrien Lardilleux, Thomas Lavergne, Artem Sokolov
  Shallow Semantic Trees for SMT
Wilker Aziz, Miguel Rios, Lucia Specia
  RegMT System for Machine Translation, System Combination, and Evaluation
Ergun Bicici, Deniz Yuret
  Improving Translation Model by Monolingual Data
Ondrej Bojar, Aleš Tamchyna
  The CMU-ARK German-English Translation System
Chris Dyer, Kevin Gimpel, Jonathan H. Clark, Noah A. Smith
  Noisy SMS Machine Translation in Low-Density Languages
Vladimir Eidelman, Kristy Hollingshead, Philip Resnik
  Stochastic Parse Tree Selection for an Existing RBMT System
Christian Federmann, Sabine Hunsicker
  Joint WMT Submission of the QUAERO Project
Markus Freitag, Gregor Leusch, Joern Wuebker, Stephan Peitz, Hermann Ney, Teresa Herrmann, Jan Niehues, Alex Waibel, Alexandre Allauzen, Gilles Adda, Josep Maria Crego, Bianka Buschbeck, Tonio Wandmacher, Jean Senellart
  CMU Syntax-Based Machine Translation at WMT 2011
Greg Hanneman, Alon Lavie
  The Uppsala-FBK systems at WMT 2011
Christian Hardmeier, Jörg Tiedemann, Markus Saers, Marcello Federico, Mathur Prashant
  The Karlsruhe Institute of Technology Translation Systems for the WMT 2011
Teresa Herrmann, Mohammed Mediani, Jan Niehues, Alex Waibel
  CMU Haitian Creole-English Translation System for WMT 2011
Sanjika Hewavitharana, Nguyen Bach, Qin Gao, Vamshi Ambati, Stephan Vogel
  Experiments with word alignment, normalization and clause reordering for SMT between English and German
Maria Holmqvist, Sara Stymne, Lars Ahrenberg
  The Value of Monolingual Crowdsourcing in a Real-World Translation Scenario: Simulation using Haitian Creole Emergency SMS Messages
Chang Hu, Philip Resnik, Yakov Kronrod, Vladimir Eidelman, Olivia Buzek, Benjamin B. Bederson
  The RWTH Aachen Machine Translation System for WMT 2011
Matthias Huck, Joern Wuebker, Christoph Schmidt, Markus Freitag, Stephan Peitz, Daniel Stein, Arnaud Dagnelies, Saab Mansour, Gregor Leusch, Hermann Ney
  ILLC-UvA translation system for EMNLP-WMT 2011
Maxim Khalilov, Khalil Sima’an
  UPM system for the translation task
Verónica López-Ludeña, Rubén San-Segundo
  Two-step translation with grammatical post-processing
David Mareček, Rudolf Rosa, Petra Galuščáková, Ondrej Bojar
  Influence of Parser Choice on Dependency-Based MT
Martin Popel, David Mareček, Nathan Green, Zdenek Zabokrtsky
  The LIGA (LIG/LIA) Machine Translation System for WMT 2011
Marion Potet, Raphaël Rubino, Benjamin Lecouteux, Stéphane Huet, Laurent Besacier, Hervé Blanchon, Fabrice Lefèvre
  Factored Translation with Unsupervised Word Clusters
Christian Rishj, Anders Sgaard
  The BM-I2R Haitian-Créole-to-English translation system description for the WMT 2011 evaluation campaign
Marta R. Costa-jussà, Rafael E. Banchs
  The Universitat d’Alacant hybrid machine translation system for WMT 2011
Víctor M. Sánchez-Cartagena, Felipe Sánchez-Martínez, Juan Antonio Pérez-Ortiz
  LIUM’s SMT Machine Translation Systems for WMT 2011
Holger Schwenk, Patrik Lambert, Loïc Barrault, Christophe Servan, Sadaf Abdul-Rauf, Haithem Afli, Kashif Shah
  Spell Checking Techniques for Replacement of Unknown Words and Data Cleaning for Haitian Creole SMS Translation
Sara Stymne
  Joshua 3.0: Syntax-based Machine Translation with the Thrax Grammar Extractor
Jonathan Weese, Juri Ganitkevitch, Chris Callison-Burch, Matt Post, Adam Lopez
  DFKI Hybrid Machine Translation System for WMT 2011 - On the Integration of SMT and RBMT
Jia Xu, Hans Uszkoreit, Casey Kennington, David Vilar, Xiaojun Zhang
  CEU-UPV English-Spanish system for WMT11
Francisco Zamora-Martinez, Maria Jose Castro-Bleda
  Hierarchical Phrase-Based MT at the Charles University for the WMT 2011 Shared Task
Daniel Zeman
12:40 – 14:00 🍴
14:00 – 15:40 Invited Talk
Crisis MT: Developing A Cookbook for MT in Crisis Situations
William Lewis, Robert Munro, Stephan Vogel
15:40 – 16:10 ☕️
16:10 – 16:35 Session 7: Translation Models
Chair: Chris Dyer
Generative Models of Monolingual and Bilingual Gappy Patterns
Kevin Gimpel, Noah A. Smith
16:35 – 17:00 Extraction Programs: A Unified Approach to Translation Rule Extraction
Mark Hopkins, Greg Langmead, Tai Vo
17:00 – 17:25 Bayesian Extraction of Minimal SCFG Rules for Hierarchical Phrase-based Translation
Baskaran Sankaran, Gholamreza Haffari, Anoop Sarkar
17:25 – 17:50 From n-gram-based to CRF-based Translation Models
Thomas Lavergne, Alexandre Allauzen, Josep Maria Crego, François Yvon

Results

Full results of the shared tasks: Findings of the 2011 Workshop on Statistical Machine Translation

News translation

The results were determined with a relative ranking, the ≥ others (“greater than or equal to others”) score.

It measures how often a system was judged to be better than or equal to any other system.

→ English

Language pair System ≥ others
Czech → UEDIN
ONLINE-B
0.69
0.68
Spanish → ONLINE-B
ONLINE-A
KOC
SYSTRAN
ALACANT
0.72
0.72
0.67
0.66
0.66
French → ONLINE-A
LIMSI
ONLINE-B
KIT
CMU-DENKOWSKI
RBMT-1
0.66
0.66
0.66
0.64
0.62
0.58
German → ONLINE-B
CMU-DYER
ONLINE-A
RWTH-WUEBKER
0.72
0.66
0.66
0.59

English →

Language pair System ≥ others
→ Czech ONLINE-B
CU-MARECEK
UEDIN
CU-POPEL
0.65
0.63
0.59
0.58
→ Spanish ONLINE-B
ONLINE-A
RBMT-3
PROMT
CEU-UPV
UEDIN
UPPSALA
0.74
0.72
0.71
0.70
0.65
0.64
0.61
→ French LIMSI
ONLINE-B
KIT
0.73
0.70
0.69
→ German RBMT-3
ONLINE-B
RBMT-1
DFKI-FEDERMANN
RBMT-4
RBMT-2
ONLINE-A
LIMSI
KIT
COPENHAGEN
0.73
0.73
0.70
0.68
0.66
0.66
0.65
0.65
0.64
0.56

Want to learn more about WMT11?


Edit this article →

Machine Translate is created and edited by contributors like you!

Learn more about contributing →

Licensed under CC-BY-SA-4.0.

Cite this article →