Structured prediction is typically solved using classifiers which predict an output with structure, often by making several classification decisions which are dependent on one another. Examples of structured prediction problems include language modeling and machine translation (where the words in the output depend on one another) and syntactic problems such as POS-tagging or syntactic parsing (where the classification labels chosen depend on one another).
Domain adaptation is, broadly speaking, the problem of training a classifier using one distribution and testing it on another distribution. There are many examples of this in real-world applications of natural language processing. One example is that a machine translation system may be trained on government proceedings (such as the Proceedings of the European Parliament) but then applied to translating medical documents. Another example is that a syntactic parser may be trained on news articles, but applied to predicting the syntactic structure of sentences in tweets.
Content:
The seminar will discuss these two interesting problems and their intersection, by looking at state-of-the-art research papers. Most of the papers discussed will involve deep learning approaches applied to problems of structured prediction, but familiarity with older classification approaches (e.g., using linear models) will be useful.
Goals:
The goal of the seminar is to understand the basics of structured prediction, domain adaptation, and domain adaptation techniques applied to structured prediction. A secondary goal is to acquire a broad picture of the state-of-the-art in these problems, particularly as applied in natural language processing.
Email Address: SubstituteMyLastName@cis.uni-muenchen.de
Room U139, Tuesdays, 16:00 to 18:00 (c.t.)
| Date | Topic | Reading (DO BEFORE THE MEETING!) | Slides |
| October 17th | Orientation | ||
| October 24th | Introduction to Structured Prediction and Domain Adaptation | pptx pdf | |
| October 31st | HOLIDAY |
Referatsthemen (name: topic)
Sign up by sending an email to Fraser, please specify a ranked list with a number of papers in case your first choice is already assigned.
The initials refer to the person proposing the topic (AF, Dr. Fabienne Braune, Matthias Huck, Dario Stojanovski). Please contact this person for questions about that topic.
Warning: the order and dates here are subject to change, DO NOT COUNT ON THE DATE REMAINING THE SAME, IT MAY BECOME EARLIER OR LATER!
| Date | Topic | Materials | Assigned To | Slides | Hausarbeit Received |
| November 7th | (FB) Y Bengio, R Ducharme, P Vincent (2003). A neural probabilistic language model. Journal of Machine Learning Research 3, 1137-1155 | ||||
| November 7th | (FB) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa (2011). Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research 2011. | paper | Iryna Markovych | slides | yes |
| November 14th | (MH) Sutskever, Ilya, Oriol Vinyals, and Quoc V Le (2014). Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems. | link | Kevin Falkner | slides | yes |
| November 14th | (MH) Bahdanau, Dzmitry, Kyunghyun Cho, Yoshua Bengio (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR. | link | Maximilian Martin | slides | yes |
| November 21st | (AF) Georg Heigold, Günter Neumann and Josef van Genabith (2017). An Extensive Empirical Evaluation of Character-Based Morphological Tagging for 14 Languages. EACL. | paper | Steffen Freisinger | slides | yes |
| November 21st | (AF) Do Kook Choe, Eugene Charniak (2016). Parsing as Language Modeling. EMNLP. | paper | Marina Speranskaya | slides | yes |
| November 28th | (DS) Luheng He, Kenton Lee, Mike Lewis, Luke Zettlemoyer (2017). Deep Semantic Role Labeling: What Works and What's Next. ACL. | paper | Benjamin Plötz | slides | yes |
| November 28th | (FB) Minh-Thang Luong, Ilya Sutskever, Oriol Vinyals, Wojciech Zaremba (2015). Addressing the rare word problem in NMT. ACL | paper | Agata Barcik | slides | yes |
| December 5th | (MH) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, James Glass (2017). What do Neural Machine Translation Models Learn about Morphology? ACL | paper | Daria Glazkova | slides | yes |
| December 5th | (FB) Attention with intention for a neural network conversation model (2015). NIPS workshop on Machine Learning for Spoken Language Understanding and Interaction. | paper | Christoph Papadatos | slides | yes |
| December 12th | (DS) Tong Wang, Ping Chen, John Rochford, Jipeng Qiang (2016) Text Simplification Using Neural Machine Translation. AAAI. | paper | Anastasiya Kryvosheya | slides | yes |
| December 12th | (DS) Xia Cui, Frans Coenen, Danushka Bollegala (2017). Effect of Data Imbalance on Unsupervised Domain Adaptation of Part-of-Speech Tagging and Pivot Selection Strategies. LIDTA | paper | Robin Rojowiec | slides | yes |
| December 19th | (FB) Nanyun Peng and Mark Dredze (2017). Multi-task Domain Adaptation for Sequence Tagging. Proceedings of the 2nd Workshop on Representation Learning for NLP. | paper | Eduard Saller | slides | yes |
| December 19th | (FB) Markus Freitag, Yaser Al-Onaizan (2016). Fast Domain Adaptation for Neural Machine Translation. arXiv preprint arXiv:1612.06897 | paper | Anne Beyer | slides | yes |
| January 9th | (DS) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky (2016). Domain-Adversarial Training of Neural Networks. JMLR | paper | Azada Rustamova | slides | yes |
| January 9th | (DS) Boxing Chen, Colin Cherry, George Foster, Samuel Larkin (2017). Cost Weighting for Neural Machine Translation Domain Adaptation. First Workshop on Neural Machine Translation. | paper | Jannis Vamvas | slides | yes |
| January 16th | (DS) Catherine Kobus, Josep Crego, Jean Senellart (2016). Domain Control for Neural Machine Translation. arXiv. | paper | Xieyidan Abuliezi | slides | yes |
| January 16th | (AF) Yoon Kim, Yacine Jernite, David Sontag, Alexander M. Rush (2016). Character-Aware Neural Language Models. AAAI | paper | |||
| January 23rd | (AF) Jason Lee, Kyunghyun Cho, Thomas Hofmann (2016). Fully Character-Level Neural Machine Translation without Explicit Segmentation. arXiv. | paper | Simon Schäfer | slides | yes |
| January 23rd | (AF) Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, Noah A. Smith (2017). What Do Recurrent Neural Network Grammars Learn About Syntax? EACL | paper | Verena Pongratz | slides | yes |
| January 30th | (MH) Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, Andrew Abel (2017). Memory-augmented Neural Machine Translation. EMNLP | paper | Bernhard Jackl | slides | yes |
| January 30th | (AF) Andrej Karpathy, Li Fei-Fei (2015). Deep Visual-Semantic Alignments for Generating Image Descriptions. CVPR | paper |
Literature:
If you do not have a general machine learning background, I'd suggest:
A Course in Machine Learning
by Hal Daumé III
2017 (0.99 beta version, for now)
http://ciml.info
If you would like to review neural networks and deep learning, this book will be useful:
Neural Network Methods in Natural Language Processing
by Yoav Goldberg
2017
or maybe this (free) review, also by Yoav Goldberg:
http://www.cs.biu.ac.il/~yogo/nnlp.pdf