Humans experience and interact with the world through sensory information. This could mean learning correspondences between the sound of bird chirps and the word “robin,” or being able to visually identify and follow your friend “Sarah” in a crowd of people. In many cases, people are able to draw on their latent “multimodal” knowledge — information across multiple data types, from audio to video to text or spoken word — to perform these tasks without many, or sometimes even any, examples.
Multimodal foundation models of artificial intelligence are increasingly gaining capabilities that mimic this. However, their apparent multimodal fluency often belies their actual understanding. An MIT-IBM Computing Research Lab team, co-led by Jim Glass, senior research scientist in the MIT Computer Science and Artificial Intelligence Laboratory, has been investigating critical failures of state-of-the-art multimodal models and developing benchmarks and data-efficient methods to overcome these performance gaps with limited supervision. He speaks about this work in the following interview.
Q: What were your goals for this project, and why would you opt to address AI models’ weaknesses over scaling to larger models?
A: This project has been going on for many years now, with roughly the same MIT-IBM team of speech and vision experts. We've wanted to explore the relationship between perception (things that you see and hear) and language, largely speaking: We have all of this absorbed knowledge and language skills, text-based, in the form of a large language model, so we wanted to know how you can relate them without using a whole bunch of annotated training data. Sometimes the models have seen certain phenomena, a latent representation of something that they have internally, but maybe due to subsequent fine-tuning or something, it disappears, so, developing techniques to try to make this capability re-emerge is a useful thing.
Source: MIT News - AI — Read the full article on the original site ↗
