Bryan A. Plummer

Assistant Professor
Image and Video Computing Group
Department of Computer Science
Boston University
Office: MCS206
Email: bplum@bu.edu

CV, Google Scholar, Github

My research interests fall within the umbrella of artificial intelligence with a focus visual recognition scene understanding, interpretable machine learning, and understanding the relationship between vision and language.

As of July 2020 I am an Assistant Professor in the Department of Computer Science at Boston University and a core faculty member of the Artificial Intelligence Research (AIR) initiative at the Rafik B. Hariri Institute for Computing and Computational Science & Engineering. I obtained my PhD in the computer vision group at the University of Illinois at Urbana-Champaign advised by Svetlana Lazebnik. After that I worked as a Postdoctoral Associate with Kate Saenko and Stan Sclaroff and later as a Research Assistant Professor before my current position. During my PhD I was fortunate to be named a 3M Foundation Fellow, NSF GRFP honorable mention, and was able to spend my summers working with Hadi Kiapour (eBay), Shuai Zheng (eBay), Robinson Piramuthu (eBay), Matthew Brown (Google), Himanshu Arora (A9), and Stephen Kelley (MIT Lincoln Labs).

PhD Students

Andrea Burns (with Kate Saneko)

Winner of the 2021 Google PhD Fellowship in Machine Perception

Reuben Tan (with Kate Saenko)
Zhongping Zhang
Diala Lteif (with Sarah Bargal)
Piotr Teterwak (with Kate Saenko)
Siqi Wang
Maan Qraitem (with Kate Saenko)
Nannan Li
Arijit Ray (with Kate Saenko)

Other Current Students

Murtadha Al-Barani (masters, with Kate Saenko)
Soren Nelson (masters)
Zora Che (undergrad)
John (Yuanming) Chai (undergrad)

Former Students

Julius Frost (BA/MS, 2021, with Kate Saenko), now at MORSE Corp
Sanjna Agrawal (undergrad, 2021)
Yize Xie (masters, 2020)
Farheen Rahman (undergrad, 2020)
Tammy Qiu (undergrad, 2019, with Kate Saenko), now at MIT Media Lab
Paige Kordas (undergrad, 2018, with David Forsyth), now at Facebook

Teaching

CS585 Image and Video Computing Spring 2022
CS542 Machine Learning Fall 2021 (co-taught with Prof. Kate Saenko)
CS591 Multimodal Machine Learning Spring 2021
CS542 Machine Learning Fall 2020

Publications Grouped by Topic

Go here for publications by year

Visual Grounding


		Maan Qraitem, Bryan A. Plummer. Learning to Reason from General Concepts to Fine-grained Tokens for Discriminative Phrase Detection. arXiv:2112.03237, 2021. [paper][bibtex] @article{qraitemPFPNet2021, author={Maan Qraitem and Bryan A. Plummer}, title={Learning to Reason from General Concepts to Fine-grained Tokens for Discriminative Phrase Detection}, journal={arXiv:2112.03237}, year={2021} }
		Reuben Tan, Bryan A. Plummer, Kate Saenko, Hailin Jin, Bryan Russell. Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos. NeurIPS, 2021. [paper][bibtex] @inproceedings{tanCOMMA2021, author = {Reuben Tan and Bryan A. Plummer and Kate Saenko and Hailin Jin and Bryan Russell}, title = {Look at What I’m Doing: Self-Supervised Spatial Grounding of Narrations in Instructional Videos}, booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, year = {2021} }
		Bryan A. Plummer, Kevin J. Shih, Yichen Li, Ke Xu, Svetlana Lazebnik, Stan Sclaroff, Kate Saenko. Revisiting Image-Language Networks for Open-ended Phrase Detection. TPAMI, 2020. [paper][code][bibtex] @article{plummerPhraseDetection, author={Bryan A. Plummer and Kevin J. Shih and Yichen Li and Ke Xu and Svetlana Lazebnik and Stan Sclaroff and Kate Saenko}, title={Revisiting Image-Language Networks for Open-ended Phrase Detection}, journal={Transactions on Pattern Analysis and Machine Intelligence}, year={2020} }
		Bryan A. Plummer, Paige Kordas, M. Hadi Kiapour, Shuai Zheng, Robinson Piramuthu, Svetlana Lazebnik. Conditional Image-Text Embedding Networks. ECCV, 2018. [paper][code][bibtex] @InProceedings{plummerCITE2018, Author = {Bryan A. Plummer and Paige Kordas and M. Hadi Kiapour and Shuai Zheng and Robinson Piramuthu and Svetlana Lazebnik}, Title = {Conditional Image-Text Embedding Networks}, booktitle = {The European Conference on Computer Vision (ECCV)}, Year = {2018} }
		Bryan A. Plummer, Arun Mallya, Christopher M. Cervantes, Julia Hockenmaier, Svetlana Lazebnik. Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues. ICCV, 2017. [paper][code][bibtex] @inproceedings{plummerPLCLC2017, Author = {Bryan A. Plummer and Arun Mallya and Christopher M. Cervantes and Julia Hockenmaier and Svetlana Lazebnik}, Title = {Phrase Localization and Visual Relationship Detection with Comprehensive Image-Language Cues}, booktitle = {The IEEE International Conference on Computer Vision (ICCV)}, Year = {2017} }
		Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, Svetlana Lazebnik. Flickr30K Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. IJCV, 123(1):74-93, 2017. [paper][project][bibtex] @article{flickrentitiesijcv, author={Bryan A. Plummer and Liwei Wang and Christopher M. Cervantes and Juan C. Caicedo and Julia Hockenmaier and Svetlana Lazebnik}, title={Flickr30K Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models}, journal={International Journal of Computer Vision}, volume={123}, number={1}, pages={74-93}, year={2017} } Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, Svetlana Lazebnik. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. ICCV, 2015. [paper][project][bibtex] @inproceedings{flickrentities, author={Bryan A. Plummer and Liwei Wang and Christopher M. Cervantes and Juan C. Caicedo and Julia Hockenmaier and Svetlana Lazebnik}, title={Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models}, booktitle={The IEEE International Conference on Computer Vision (ICCV)}, year={2015} }

Defending Against Machine Manipulated Media

Reuben Tan, Bryan A. Plummer, Kate Saenko. Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News. EMNLP, 2020. [paper][project][bibtex]

Explainable AI

Olivia Watkins, Sandy Huang, Julius Frost, Kush Bhatia, Eric Weiner, Pieter Abbeel, Trevor Darrell, Bryan A. Plummer, Kate Saenko, Anca Dragan. Explaining robot policies. Applied AI Letters, 2021. [paper][bibtex]

Bryan A. Plummer*, Mariya I. Vasileva*, Vitali Petsiuk, Kate Saenko, David Forsyth. Why do These Match? Explaining the Behavior of Image Similarity Models. ECCV, 2020. [paper][code][supplementary][bibtex]

Language Representation Learning


		Zhongping Zhang, Yiwen Gu, Bryan A. Plummer. Show and Write: Entity-aware News Generation with Image Information. arXiv:2112.05917, 2021. [paper][bibtex] @article{zhangENGIN2021, author={Zhongping Zhang and Yiwen Gu and Bryan A. Plummer}, title={Show and Write: Entity-aware News Generation with Image Information}, journal={arXiv:2112.05917}, year={2021} }
		Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, Bryan A. Plummer. Learning to Scale Multilingual Representations for Vision-Language Tasks. ECCV, 2020. [paper][project][supplementary][bibtex] @InProceedings{burnsSMALR2020, author={Andrea Burns and Donghyun Kim and Derry Wijaya and Kate Saenko and Bryan A. Plummer}, title={Learning to Scale Multilingual Representations for Vision-Language Tasks}, booktitle={The European Conference on Computer Vision (ECCV)}, year={2020} }
		Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, Bryan A. Plummer. MULE: Multimodal Universal Language Embedding. AAAI, 2020. [paper][project][code][bibtex] @inproceedings{kimMULEAAAI2020, author={Donghyun Kim and Kuniaki Saito and Kate Saenko and Stan Sclaroff and Bryan A. Plummer}, title={{MULE: Multimodal Universal Language Embedding}}, booktitle={AAAI Conference on Artificial Intelligence}, year={2020} }
		Andrea Burns, Reuben Tan, Kate Saenko, Stan Sclaroff, Bryan A. Plummer. Language Features Matter: Effective Language Representations for Vision-Language Tasks. ICCV, 2019. [paper][project][bibtex] @InProceedings{burnsLanguage2019, author={Andrea Burns and Reuben Tan and Kate Saenko and Stan Sclaroff and Bryan A. Plummer}, title={Language Features Matter: Effective Language Representations for Vision-Language Tasks}, booktitle={The IEEE International Conference on Computer Vision (ICCV)}, year={2019} }

Vision and Language


		Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A. Plummer. Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands. arXiv:2202.02312, 2022. [paper][code&data][bibtex] @article{burnsMOTIF2022, author={Andrea Burns and Deniz Arsan and Sanjna Agrawal and Ranjitha Kumar and Kate Saenko and Bryan A. Plummer}, title={Interactive Mobile App Navigation with Uncertain or Under-specified Natural Language Commands}, journal={arXiv:2202.02312}, year={2022} }
		Reuben Tan, Huijuan Xu, Kate Saenko, Bryan A. Plummer. LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval. WACV, 2021. [paper][bibtex] @InProceedings{tanLoGAN2021, author={Reuben Tan and Huijuan Xu and Kate Saenko and Bryan A. Plummer}, title={LoGAN: Latent Graph Co-Attention Network for Weakly-Supervised Video Moment Retrieval}, booktitle={IEEE Winter Conference on Applications of Computer Vision (WACV)}, year={2021} }
		Huijuan Xu, Kun He, Bryan A. Plummer, Leonid Sigal, Stan Sclaroff, Kate Saenko. Multilevel Language and Vision Integration for Text-to-Clip Retrieval. AAAI, 2019. [paper][code][bibtex] @InProceedings{xuAAAI2019, author={Huijuan Xu and Kun He and Bryan A. Plummer and Leonid Sigal and Stan Sclaroff and Kate Saenko}, title={Multilevel Language and Vision Integration for Text-to-Clip Retrieval}, booktitle={AAAI Conference on Artificial Intelligence}, year={2019} }
		Tatiana Tommasi, Arun Mallya, Bryan A. Plummer, Svetlana Lazebnik, Alexander C. Berg, Tamara L. Berg. Combining Multiple Cues for Visual Madlibs Question Answering. IJCV, 127(1):38-60, 2019. [paper][bibtex] @article{tommasiIJCV2019, author={Tatiana Tommasi and Arun Mallya and Bryan A. Plummer and Svetlana Lazebnik and Alexander C. Berg and Tamara L. Berg}, title={Combining Multiple Cues for Visual Madlibs Question Answering}, journal={International Journal of Computer Vision}, volume={127}, number={1}, pages={38-60}, year={2019} } Tatiana Tommasi, Arun Mallya, Bryan A. Plummer, Svetlana Lazebnik, Alexander C. Berg, Tamara L. Berg. Solving Visual Madlibs with Multiple Cues. BMVC, 2016. [paper][bibtex] @inproceedings{tommasiBMVC2016, author = {Tatiana Tommasi and Arun Mallya and Bryan A. Plummer and Svetlana Lazebnik and Alexander C. Berg and Tamara L. Berg}, title = {Solving Visual Madlibs with Multiple Cues}, booktitle={British Machine Vision Conference (BMVC)}, year = {2016} }
		Bryan A. Plummer, Matthew Brown, Svetlana Lazebnik. Enhancing Video Summarization via Vision-Language Embedding. CVPR, 2017. [paper][bibtex] @InProceedings{plummerSummarization2017, author={Bryan A. Plummer and Matthew Brown and Svetlana Lazebnik}, title={Enhancing Video Summarization via Vision-Language Embedding}, booktitle={The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, year={2017} }

Efficient Neural Networks

Bryan A. Plummer*, Nikoli Dryden*, Julius Frost, Torsten Hoefler, Kate Saenko. Neural Parameter Allocation Search. ICLR, 2022. [paper][bibtex]

Donghyun Kim, Tian Lan, Chuhang Zou, Ning Xu, Bryan A. Plummer, Stan Sclaroff, Jayan Eledath, Gerard Medioni. Multi-Task Learning from Videos via Efficient Inter-Frame Attention. arXiv:2002.07362, 2020. [paper][bibtex]

Image Similarity and Search


		Siqi Wang, Manyuan Lu, Nikita Moshkov, Juan C. Caicedo, Bryan A. Plummer. Anchoring to Exemplars for Training Mixture-of-Expert Cell Embeddings. arXiv:2112.03208, 2021. [paper][bibtex] @article{wangTEAMS2021, author={Siqi Wang and Manyuan Lu and Nikita Moshkov and Juan C. Caicedo and Bryan A. Plummer}, title={Anchoring to Exemplars for Training Mixture-of-Expert Cell Embeddings}, journal={arXiv:2112.03208}, year={2021} }
		Samarth Mishra, Zhongping Zhang, Yuan Shen, Ranjitha Kumar, Venkatesh Saligrama, Bryan A. Plummer. Effectively Leveraging Attributes for Visual Similarity. ICCV, 2021. [paper][code][bibtex] @InProceedings{mishraPAN2021, author={Samarth Mishra and Zhongping Zhang and Yuan Shen and Ranjitha Kumar and Venkatesh Saligrama and Bryan A. Plummer}, title={Effectively Leveraging Attributes for Visual Similarity}, booktitle={The IEEE International Conference on Computer Vision (ICCV)}, year={2021} }
		Reuben Tan, Mariya I. Vasileva, Kate Saenko, Bryan A. Plummer. Learning Similarity Conditions Without Explicit Supervision. ICCV, 2019. [paper][code][bibtex] @InProceedings{tanSimilarity2019, author={Reuben Tan and Mariya I. Vasileva and Kate Saenko and Bryan A. Plummer}, title={Learning Similarity Conditions Without Explicit Supervision}, booktitle={The IEEE International Conference on Computer Vision (ICCV)}, year={2019} }
		Bryan A. Plummer, M. Hadi Kiapour, Shuai Zheng, Robinson Piramuthu. Give me a hint! Navigating Image Databases using Human-in-the-loop Feedback. WACV, 2019. [paper][bibtex] @InProceedings{plummerHint2019, author={Bryan A. Plummer and M. Hadi Kiapour and Shuai Zheng and Robinson Piramuthu}, title={{Give me a hint! Navigating Image Databases using Human-in-the-loop Feedback}}, booktitle={IEEE Winter Conference on Applications of Computer Vision (WACV)}, year={2019} }
		Mariya I. Vasileva, Bryan A. Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, David Forsyth. Learning Type-Aware Embeddings for Fashion Compatibility. ECCV, 2018. [paper][code][bibtex] @inproceedings{VasilevaECCV18FasionCompatibility, Author = {Mariya I. Vasileva and Bryan A. Plummer and Krishna Dusad and Shreya Rajpal and Ranjitha Kumar and David Forsyth}, Title = {Learning Type-Aware Embeddings for Fashion Compatibility}, booktitle = {The European Conference on Computer Vision (ECCV)}, Year = {2018} }

Domain Adaptation/Transfer Learning

Donghyun Kim, Kuniaki Saito, Tae-Hyun Oh, Bryan A. Plummer, Stan Sclaroff, Kate Saenko. CDS: Cross-Domain Self-supervised Pre-training. ICCV, 2021. [paper][bibtex]

P. Daphne Tsatsoulis, Bryan A. Plummer, David Forsyth. Visual Analogies: A Framework for Defining Aspect Categorization. ECCV TASK-CV Workshop, 2016. Workshop Best Paper Award. [paper][bibtex]