Charles Cui

I am a Ph.D. candidate in Computer Science at Northwestern University, advised by Matthew Kay. I am broadly interested in human-AI collaboration, data science, and data visualization. Currently, I use large language models and data science methods to develop adaptive, scalable, and quality-oriented education technology for data visualization. I am passionate about advancing human-AI collaboration to improve our productivity and creativity.

During my Ph.D., I have had the fortune to collaborate with many brilliant minds across disciplines and institutions: I co-direct EAAMO Bridges (formerly MD4SG), which leverages computational methods to improve equity and welfare for marginalized groups. I was a visiting researcher in social data science at the Max Planck Institute for Demographic Research in 2024, where I studied population-level mortality estimation in data-scarce contexts. Previously, I was a graduate fellow at Stanford’s RegLab, where I developed computational methods for racial disparity estimation in public health. I also served as a Data Science for Social Good (DSSG) fellow at Carnegie Mellon University in 2022, where I worked with Vibrant Emotional Health to improve the service of the 988 Suicide & Crisis Lifeline through data-driven call routing.

news

Oct 2, 2024	I completed my thesis prospectus. Thanks to my amazing committee for their great advice!
Aug 30, 2024	I will be presenting our paper Promises and Pitfalls: Using Large Language Models to Generate Visualization Items on the morning of Oct 18 (Fri) at IEEE VIS 2024. Please come to our paper session and chat with me anytime during the conference!

selected publications

2024

IEEE VIS

Promises and Pitfalls: Using Large Language Models to Generate Visualization Items

Yuan Cui, Lily W. Ge, Yiren Ding, Lane Harrison, Fumeng Yang, and Matthew Kay

IEEE VIS, 2024

Abs HTML PDF

Visualization items—factual questions about visualizations that ask viewers to accomplish visualization tasks—are regularly used in the field of information visualization as educational and evaluative materials. For example, researchers of visualization literacy require large, diverse banks of items to conduct studies where the same skill is measured repeatedly on the same participants. Yet, generating a large number of high-quality, diverse items requires significant time and expertise. To address the critical need for a large number of diverse visualization items in education and research, this paper investigates the potential for large language models (LLMs) to automate the generation of multiple-choice visualization items. Through an iterative design process, we develop the VILA (Visualization Items Generated by Large LAnguage Models) pipeline, for efficiently generating visualization items that measure people’s ability to accomplish visualization tasks. We use the VILA pipeline to generate 1,404 candidate items across 12 chart types and 13 visualization tasks. In collaboration with 11 visualization experts, we develop an evaluation rulebook which we then use to rate the quality of all candidate items. The result is the VILA bank of ∼1,100 items. From this evaluation, we also identify and classify current limitations of the VILA pipeline, and discuss the role of human oversight in ensuring quality. In addition, we demonstrate an application of our work by creating a visualization literacy test, VILA-VLAT, which measures people’s ability to complete a diverse set of tasks on various types of visualizations; comparing it to the existing VLAT, VILA-VLAT shows moderate to high convergent validity (R = 0.70). Lastly, we discuss the application areas of the VILA pipeline and the VILA bank and provide practical recommendations for their use.

2023

IEEE VIS

Adaptive Assessment of Visualization Literacy

Yuan Cui, Lily W. Ge, Yiren Ding, Fumeng Yang, Lane Harrison, and Matthew Kay

IEEE VIS, 2023

Abs HTML PDF

Visualization literacy is an essential skill for accurately interpreting data to inform critical decisions. Consequently, it is vital to understand the evolution of this ability and devise targeted interventions to enhance it, requiring concise and repeatable assessments of visualization literacy for individuals. However, current assessments, such as the Visualization Literacy Assessment Test (VLAT), are time-consuming due to their fixed, lengthy format. To address this limitation, we develop two streamlined computerized adaptive tests (CATs) for visualization literacy, A-VLAT and A-CALVI, which measure the same set of skills as their original versions in half the number of questions. Specifically, we (1) employ item response theory (IRT) and non-psychometric constraints to construct adaptive versions of the assessments, (2) finalize the configurations of adaptation through simulation, (3) refine the composition of test items of A-CALVI via a qualitative study, and (4) demonstrate the test-retest reliability (ICC: 0.98 and 0.98) and convergent validity (correlation: 0.81 and 0.66) of both CATs via four online studies. We discuss practical recommendations for using our CATs and opportunities for further customization to leverage the full potential of adaptive assessments.
ACM CHI

CALVI: Critical Thinking Assessment for Literacy in Visualizations

Lily W. Ge, Yuan Cui, and Matthew Kay

ACM CHI, 2023

Abs HTML PDF

Visualization misinformation is a prevalent problem, and combating it requires understanding people’s ability to read, interpret, and reason about erroneous or potentially misleading visualizations, which lacks a reliable measurement: existing visualization literacy tests focus on well-formed visualizations. We systematically develop an assessment for this ability by: (1) developing a precise definition of misleaders (decisions made in the construction of visualizations that can lead to conclusions not supported by the data), (2) constructing initial test items using a design space of misleaders and chart types, (3) trying out the provisional test on 497 participants, and (4) analyzing the test tryout results and refining the items using Item Response Theory, qualitative analysis, a wrong-due-to-misleader score, and the content validity index. Our final bank of 45 items shows high reliability, and we provide item bank usage recommendations for future tests and different use cases.