Ongoing Research Projects
Communicating about Data Visualizations
Data visualizations are a way of presenting dense information in a visually structured format. In this project, we uncover the foundations of how we cognitively represent the information contained in data visualizations, and how they can be effectively described. We work with people who are sighted and BLV (blind or low vision) to investigate how generative AI can be used to aid visual literacy and accessibility of data visualizations. Our research aims to promote equal access, specifically to academic content. This is a collaboration with the cogtoolslab.
An Image is Worth a Thousand Words
— But Context Determines which are Important
Images are omnipresent online but they remain inaccessible to blind and low vision users. Image-based text generation models are a promising resource that have the potential to make images accessible to users on the fly. However, they haven’t been useful in practice since the information they generate is often not relevant or detailed enough. In this work, we investigate (1) how the context an image appears in shapes which information become relevant, and (2) how this context-dependence needs to be integrated in the NLP pipeline: datasets, models, and evaluation metrics. We’re also working with the Wikimedia foundation in an effort to characterize the image accessibility on Wikipedia and investigate strategies for improving editing practices across languages.
What Distinguishes a Banana from a Blue Banana?
When we refer to objects around us, we often mention features that aren’t necessarily required to pick out the object. For instance, when we see a banana that’s blue, we tend to refer to it as a “blue banana” even though “banana” would have been sufficient. We use computational models of language production and comprehension to understand the underlying principles of human reference production, and investigate how different cognitive mechanisms might drive our communication.