WMT11

Workshop on Machine Translation

Location

Edinburgh, United Kingdom

Shared Tasks

Schedule

Day 1


9:00	Session 1: Shared Tasks and Evaluation Chair: Chris Callison-Burch A Grain of Salt for the WMT Manual Evaluation Ondrej Bojar, Miloš Ercegovčević, Martin Popel, Omar Zaidan
9:25	A Lightweight Evaluation Framework for Machine Translation Reordering David Talbot, Hideto Kazawa, Hiroshi Ichikawa, Jason Katz-Brown, Masakazu Seno, Franz Och
9:50	Findings of the 2011 Workshop on Statistical Machine Translation Chris Callison-Burch, Philipp Koehn, Christof Monz, Omar Zaidan
10:30	☕️
11:00	Session 2: Shared Metrics and System Combination Tasks Poster Session Metrics
	Evaluate with Confidence Estimation: Machine ranking of translation outputs using grammatical features Eleftherios Avramidis, Maja Popović, David Vilar, Aljoscha Burchardt
	AMBER: A Modified BLEU, Enhanced Ranking Metric Boxing Chen, Roland Kuhn
	TESLA at WMT 2011: Translation Evaluation and Tunable Metric Daniel Dahlmeier, Chang Liu, Hwee Tou Ng
	Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems Michael Denkowski, Alon Lavie
	Approximating a Deep-Syntactic Metric for MT Evaluation and Tuning Matouš Macháček, Ondrej Bojar
	Evaluation without references: IBM1 scores as evaluation metrics Maja Popović, David Vilar, Eleftherios Avramidis, Aljoscha Burchardt
	Morphemes and POS tags for n-gram based evaluation metrics Maja Popović
	E-rating Machine Translation Kristen Parton, Joel Tetreault, Nitin Madnani, Martin Chodorow
	TINE: A Metric to Assess MT Adequacy Miguel Rios, Wilker Aziz, Lucia Specia
	Regression and Ranking based Optimisation for Sentence Level MT Evaluation Xingyi Song, Trevor Cohn
	MAISE: A Flexible, Configurable, Extensible Open Source Package for Mass AI System Evaluation Omar Zaidan
	Poster Session System Combination
	MANY improvements for WMT11 Loïc Barrault
	The UPV-PRHLT combination system for WMT 2011 Jesús González-Rubio, Francisco Casacuberta
	CMU System Combination in WMT 2011 Kenneth Heafield, Alon Lavie
	The RWTH System Combination System for WMT 2011 Gregor Leusch, Markus Freitag, Hermann Ney
	Expected BLEU Training for Graphs: BBN System Description for WMT11 System Combination Task Antti-Veikko Rosti, Bing Zhang, Spyros Matsoukas, Richard Schwartz
	The UZH System Combination System for WMT 2011 Rico Sennrich
	Description of the JHU System Combination Scheme for WMT 2011 Daguang Xu, Yuan Cao, Damianos Karakos
12:40	🍴
14:00	Session 3: Language Models and Document Alignment Chair: David Talbot Multiple-stream Language Models for Statistical Machine Translation Abby Levenberg, Miles Osborne, David Matthews
14:25	KenLM: Faster and Smaller Language Model Queries Kenneth Heafield
14:50	Wider Context by Using Bilingual Language Models in Machine Translation Jan Niehues, Teresa Herrmann, Stephan Vogel, Alex Waibel
15:15	A Minimally Supervised Approach for Detecting and Ranking Document Translation Pairs Kriste Krstovski, David A. Smith
15:40	☕️
16:10	Session 4: Syntax and Semantics Chair: Michel Galley Agreement Constraints for Statistical Machine Translation into German Philip Williams, Philipp Koehn
16:35	Fuzzy Syntactic Reordering for Phrase-based Statistical Machine Translation Jacob Andreas, Nizar Habash, Owen Rambow
17:00	Filtering Antonymous, Trend-Contrasting, and Polarity-Dissimilar Distributional Paraphrases for Improving Statistical Machine Translation Yuval Marton, Ahmed El Kholy, Nizar Habash
17:25	Productive Generation of Compound Words in Statistical Machine Translation Sara Stymne, Nicola Cancedda

Day 2


9:00	Session 5: Machine Learning and Adaptation Chair: Phil Blunsom SampleRank Training for Phrase-Based Machine Translation Barry Haddow, Abhishek Arun, Philipp Koehn
9:25	Instance Selection for Machine Translation using Feature Decay Algorithms Ergun Bicici, Deniz Yuret
9:50	Investigations on Translation Model Adaptation Using Monolingual Data Patrik Lambert, Holger Schwenk, Christophe Servan, Sadaf Abdul-Rauf
10:15	Topic Adaptation for Lecture Translation through Bilingual Latent Semantic Models Nick Ruiz, Marcello Federico
10:40	☕️
11:00	Session 6: Shared Translation Task Poster Presentations
	Personal Translator at WMT2011 Vera Aleksic, Gregor Thurmair
	LIMSI @ WMT11 Alexandre Allauzen, Hélène Bonneau-Maynard, Hai-Son Le, Aurélien Max, Guillaume Wisniewski, François Yvon, Gilles Adda, Josep Maria Crego, Adrien Lardilleux, Thomas Lavergne, Artem Sokolov
	Shallow Semantic Trees for SMT Wilker Aziz, Miguel Rios, Lucia Specia
	RegMT System for Machine Translation, System Combination, and Evaluation Ergun Bicici, Deniz Yuret
	Improving Translation Model by Monolingual Data Ondrej Bojar, Aleš Tamchyna
	The CMU-ARK German-English Translation System Chris Dyer, Kevin Gimpel, Jonathan H. Clark, Noah A. Smith
	Noisy SMS Machine Translation in Low-Density Languages Vladimir Eidelman, Kristy Hollingshead, Philip Resnik
	Stochastic Parse Tree Selection for an Existing RBMT System Christian Federmann, Sabine Hunsicker
	Joint WMT Submission of the QUAERO Project Markus Freitag, Gregor Leusch, Joern Wuebker, Stephan Peitz, Hermann Ney, Teresa Herrmann, Jan Niehues, Alex Waibel, Alexandre Allauzen, Gilles Adda, Josep Maria Crego, Bianka Buschbeck, Tonio Wandmacher, Jean Senellart
	CMU Syntax-Based Machine Translation at WMT 2011 Greg Hanneman, Alon Lavie
	The Uppsala-FBK systems at WMT 2011 Christian Hardmeier, Jörg Tiedemann, Markus Saers, Marcello Federico, Mathur Prashant
	The Karlsruhe Institute of Technology Translation Systems for the WMT 2011 Teresa Herrmann, Mohammed Mediani, Jan Niehues, Alex Waibel
	CMU Haitian Creole-English Translation System for WMT 2011 Sanjika Hewavitharana, Nguyen Bach, Qin Gao, Vamshi Ambati, Stephan Vogel
	Experiments with word alignment, normalization and clause reordering for SMT between English and German Maria Holmqvist, Sara Stymne, Lars Ahrenberg
	The Value of Monolingual Crowdsourcing in a Real-World Translation Scenario: Simulation using Haitian Creole Emergency SMS Messages Chang Hu, Philip Resnik, Yakov Kronrod, Vladimir Eidelman, Olivia Buzek, Benjamin B. Bederson
	The RWTH Aachen Machine Translation System for WMT 2011 Matthias Huck, Joern Wuebker, Christoph Schmidt, Markus Freitag, Stephan Peitz, Daniel Stein, Arnaud Dagnelies, Saab Mansour, Gregor Leusch, Hermann Ney
	ILLC-UvA translation system for EMNLP-WMT 2011 Maxim Khalilov, Khalil Sima’an
	UPM system for the translation task Verónica López-Ludeña, Rubén San-Segundo
	Two-step translation with grammatical post-processing David Mareček, Rudolf Rosa, Petra Galuščáková, Ondrej Bojar
	Influence of Parser Choice on Dependency-Based MT Martin Popel, David Mareček, Nathan Green, Zdenek Zabokrtsky
	The LIGA (LIG/LIA) Machine Translation System for WMT 2011 Marion Potet, Raphaël Rubino, Benjamin Lecouteux, Stéphane Huet, Laurent Besacier, Hervé Blanchon, Fabrice Lefèvre
	Factored Translation with Unsupervised Word Clusters Christian Rishj, Anders Sgaard
	The BM-I2R Haitian-Créole-to-English translation system description for the WMT 2011 evaluation campaign Marta R. Costa-jussà, Rafael E. Banchs
	The Universitat d'Alacant hybrid machine translation system for WMT 2011 Víctor M. Sánchez-Cartagena, Felipe Sánchez-Martínez, Juan Antonio Pérez-Ortiz
	LIUM’s SMT Machine Translation Systems for WMT 2011 Holger Schwenk, Patrik Lambert, Loïc Barrault, Christophe Servan, Sadaf Abdul-Rauf, Haithem Afli, Kashif Shah
	Spell Checking Techniques for Replacement of Unknown Words and Data Cleaning for Haitian Creole SMS Translation Sara Stymne
	Joshua 3.0: Syntax-based Machine Translation with the Thrax Grammar Extractor Jonathan Weese, Juri Ganitkevitch, Chris Callison-Burch, Matt Post, Adam Lopez
	DFKI Hybrid Machine Translation System for WMT 2011 - On the Integration of SMT and RBMT Jia Xu, Hans Uszkoreit, Casey Kennington, David Vilar, Xiaojun Zhang
	CEU-UPV English-Spanish system for WMT11 Francisco Zamora-Martinez, Maria Jose Castro-Bleda
	Hierarchical Phrase-Based MT at the Charles University for the WMT 2011 Shared Task Daniel Zeman
12:40	🍴
14:00	Invited Talk Crisis MT: Developing A Cookbook for MT in Crisis Situations William Lewis, Robert Munro, Stephan Vogel
15:40	☕️
16:10	Session 7: Translation Models Chair: Chris Dyer Generative Models of Monolingual and Bilingual Gappy Patterns Kevin Gimpel, Noah A. Smith
16:35	Extraction Programs: A Unified Approach to Translation Rule Extraction Mark Hopkins, Greg Langmead, Tai Vo
17:00	Bayesian Extraction of Minimal SCFG Rules for Hierarchical Phrase-based Translation Baskaran Sankaran, Gholamreza Haffari, Anoop Sarkar
17:25	From n-gram-based to CRF-based Translation Models Thomas Lavergne, Alexandre Allauzen, Josep Maria Crego, François Yvon

Results

General Task

Full results of the shared task: Findings of the 2011 Workshop on Statistical Machine Translation

Tunable metrics task

The tunable metrics shared task was invitation-only.

News translation

The results were determined with a relative ranking, the ≥ others (“greater than or equal to others”) score.

It measures how often a system was judged to be better than or equal to any other system.

→ English

Language pair	System	`≥ others`
Czech →	`UEDIN` `ONLINE-B`	0.69 0.68
Spanish →	`ONLINE-B` `ONLINE-A` `KOC` `SYSTRAN` `ALACANT`	0.72 0.72 0.67 0.66 0.66
French →	`ONLINE-A` `LIMSI` `ONLINE-B` `KIT` `CMU-DENKOWSKI` `RBMT-1`	0.66 0.66 0.66 0.64 0.62 0.58
German →	`ONLINE-B` `CMU-DYER` `ONLINE-A` `RWTH-WUEBKER`	0.72 0.66 0.66 0.59

English →

Language pair	System	`≥ others`
→ Czech	`ONLINE-B` `CU-MARECEK` `UEDIN` `CU-POPEL`	0.65 0.63 0.59 0.58
→ Spanish	`ONLINE-B` `ONLINE-A` `RBMT-3` `PROMT` `CEU-UPV` `UEDIN` `UPPSALA`	0.74 0.72 0.71 0.70 0.65 0.64 0.61
→ French	`LIMSI` `ONLINE-B` `KIT`	0.73 0.70 0.69
→ German	`RBMT-3` `ONLINE-B` `RBMT-1` `DFKI-FEDERMANN` `RBMT-4` `RBMT-2` `ONLINE-A` `LIMSI` `KIT` `COPENHAGEN`	0.73 0.73 0.70 0.68 0.66 0.66 0.65 0.65 0.64 0.56