Complex pairwise relationships—which are ubiquitous in NLP—are best encoded as graph-structured data: social media networks, relations between entities, and syntactic/semantic parses (to name just a few) are all naturally expressed as graphs. However, constructing effective structural priors for neural models over graphs is considerably more difficult than for “traditional” machine learning media such as text and image data—this is to say that graph data requires highly specialized model architectures. In this seminar, we will delve into active research at the intersection of NLP and graph deep learning, with a particular focus on incorporating graph representations into language models, in order to enable them to engage with graph-structured data such as knowledge graphs and dependency parses.
This course assumes a solid background in NLP/ML and current language models: you should be familiar with transformer architectures and common methods/terminology in NLP. You should also (at least) know what a graph is. If you don’t, just read the Wikipedia page, and you should be fine: https://en.wikipedia.org/wiki/Graph_(discrete_mathematics).
You do not, however, need any background in graph deep learning: we will discuss current methodologies for modeling graph-stuctured data (i.e. graph neural networks) in class.
Instructor: Michael Sullivan
Time/Location: Thu 16:15 - 17:45; Building C7.2, Room -1.05
Office Hours: Wed 12:00 - 14:00; Building C7.2, Room 1.04 (or on Zoom/Teams by appointment)
Each week, we will meet to discuss the assigned reading (see Schedule/Reading List below). All students are expected to ask questions and actively engage in the discussion.
At the beginning of the semester, each student will choose a reading for which they will lead the discussion (“Discussion Leader”): if you do not reach out to me to chose a reading, I will randomly assign you a paper. The Discussion Leader will:
While you are not necessarily expected to create slides as Discussion Leader, sometimes visuals can help! Feel free to use a slide deck, if you would like.
The course is divided into topics/themes: if there is another paper that you would like to discuss instead of the assigned reading (that you feel falls under the same topic), come see me during office hours a few weeks before the assigned date. I will update the schedule if I find your paper appropriate for the course and topic.
For students taking the course for 4 credits:
Participation in class discussions: 50%
Discussion Leader: 50%
For students taking the course for 7 credits:
Participation in class discussions: 30%
Discussion Leader: 30%
Term paper: 40%
Useful additional reading: Unifying Large Language Models and Knowledge Graphs: A Roadmap
Students taking the course for 7 credits will be expected to write a survey paper on one of the topics discussed in class (or—with my approval—another topic at the intersection of graph deep learning and NLP). Your task is to take a deep dive into the current literature on your chosen topic, and identify 3-4 major sub-areas.
For each sub-area, identify the main challenges and choose 2-3 papers that are representative of that sub-area: you should include relevant readings that we discussed in class, but they do not count towards the 2-3 paper requirement. For each paper, you will summarize and analyze the authors’ methods (experimental design, model architecture, and training procedure) and results: you are essentially expected to act as a “mini Discussion Leader” (minus the “field questions” part) for each of these papers, but in a written format (obviously).
There is no minimum length for the term paper: if you satisfy all of the requirements described above, your paper will be long enough (I’m expecting these to be somewhere in the neighborhood of eight pages). There is also no strict maximum page count. That being said, please limit your paper to a reasonable length: I would really prefer not to have to read fifteen fifty-page papers at the end of the course!
Here are some examples of survey papers, to give you an idea of the expected structure of your report:
Term papers will be due by March 21, 2025, and should be submitted in ACL format.