Charles Cui

I am a Ph.D. candidate in Computer Science at Northwestern University, advised by Matthew Kay. I’m also currently a visiting student at the FIGX Lab of UMD Computer Science, mentored by Fumeng Yang. I am broadly interested in advancing human-AI collaboration to improve our productivity and creativity. In particular, my thesis research includes using data science methods and large language models to build adaptive, scalable, and human-centered technology to measure and promote data visualization literacy and help teachers develop educational assessments. I use a variety of approaches in research, including quantitative and qualitative analysis of human-computer and human-human interactions, as well as designing, developing, and evaluating systems.

I also worked across institutions to explore my interdisciplinary interests: I co-direct EAAMO Bridges, a community that leverages computational methods to improve welfare for marginalized groups. I studied mortality estimation as a social data science researcher at the Max Planck Institute for Demographic Research. I developed statistical methods for disparity estimation as a graduate fellow at Stanford’s RegLab. I also built machine learning solutions for the 988 Lifeline as a Data Science for Social Good fellow at Carnegie Mellon University.

news

Mar 10, 2025	I will be interning at Adobe, San Jose this summer. I’d love to connect if you will be in the area!
Feb 20, 2025	Our paper, AVEC: An Assessment of Visual Encoding Ability in Visualization Construction, is accepted to CHI 2025. Let’s meet up in Yokohama in April!
Dec 1, 2024	I started working as a visiting student researcher at FIGX Lab with Dr. Fumeng Yang at the University of Maryland, College Park.

selected publications

2025

ACM CHI

AVEC: An Assessment of Visual Encoding Ability in Visualization Construction

Lily W. Ge, Yuan Cui, and Matthew Kay

ACM Conference on Human Factors in Computing Systems (CHI), 2025

Abs HTML PDF

Visualization literacy is the ability to both interpret and construct visualizations. Yet existing assessments focus solely on visualization interpretation. A lack of construction-related measurements hinders efforts in understanding and improving literacy in visualizations. We design and develop avec, an assessment of a person’s visual encoding ability—a core component of the larger process of visualization construction—by: (1) creating an initial item bank using a design space of visualization tasks and chart types, (2) designing an assessment tool to support the combinatorial nature of selecting appropriate visual encodings, (3) building an autograder from expert scores of answers to our items, and (4) refining and validating the item bank and autograder through an analysis of test tryout data with 95 participants and feedback from the expert panel. We discuss recommendations for using avec, potential alternative scoring strategies, and the challenges in assessing higher-level visualization skills using constructed-response tests. Supplemental materials are available at: https://osf.io/hg7kx/.

2024

IEEE VIS

Promises and Pitfalls: Using Large Language Models to Generate Visualization Items

Yuan Cui, Lily W. Ge, Yiren Ding, Lane Harrison, Fumeng Yang, and Matthew Kay

IEEE Visualization Conference (VIS), 2024

Abs HTML PDF

Visualization items—factual questions about visualizations that ask viewers to accomplish visualization tasks—are regularly used in the field of information visualization as educational and evaluative materials. For example, researchers of visualization literacy require large, diverse banks of items to conduct studies where the same skill is measured repeatedly on the same participants. Yet, generating a large number of high-quality, diverse items requires significant time and expertise. To address the critical need for a large number of diverse visualization items in education and research, this paper investigates the potential for large language models (LLMs) to automate the generation of multiple-choice visualization items. Through an iterative design process, we develop the VILA (Visualization Items Generated by Large LAnguage Models) pipeline, for efficiently generating visualization items that measure people’s ability to accomplish visualization tasks. We use the VILA pipeline to generate 1,404 candidate items across 12 chart types and 13 visualization tasks. In collaboration with 11 visualization experts, we develop an evaluation rulebook which we then use to rate the quality of all candidate items. The result is the VILA bank of ∼1,100 items. From this evaluation, we also identify and classify current limitations of the VILA pipeline, and discuss the role of human oversight in ensuring quality. In addition, we demonstrate an application of our work by creating a visualization literacy test, VILA-VLAT, which measures people’s ability to complete a diverse set of tasks on various types of visualizations; comparing it to the existing VLAT, VILA-VLAT shows moderate to high convergent validity (R = 0.70). Lastly, we discuss the application areas of the VILA pipeline and the VILA bank and provide practical recommendations for their use.

2023

IEEE VIS

Adaptive Assessment of Visualization Literacy

Yuan Cui, Lily W. Ge, Yiren Ding, Fumeng Yang, Lane Harrison, and Matthew Kay

IEEE Visualization Conference (VIS), 2023

Abs HTML PDF

Visualization literacy is an essential skill for accurately interpreting data to inform critical decisions. Consequently, it is vital to understand the evolution of this ability and devise targeted interventions to enhance it, requiring concise and repeatable assessments of visualization literacy for individuals. However, current assessments, such as the Visualization Literacy Assessment Test (VLAT), are time-consuming due to their fixed, lengthy format. To address this limitation, we develop two streamlined computerized adaptive tests (CATs) for visualization literacy, A-VLAT and A-CALVI, which measure the same set of skills as their original versions in half the number of questions. Specifically, we (1) employ item response theory (IRT) and non-psychometric constraints to construct adaptive versions of the assessments, (2) finalize the configurations of adaptation through simulation, (3) refine the composition of test items of A-CALVI via a qualitative study, and (4) demonstrate the test-retest reliability (ICC: 0.98 and 0.98) and convergent validity (correlation: 0.81 and 0.66) of both CATs via four online studies. We discuss practical recommendations for using our CATs and opportunities for further customization to leverage the full potential of adaptive assessments.
ACM CHI

CALVI: Critical Thinking Assessment for Literacy in Visualizations

Lily W. Ge, Yuan Cui, and Matthew Kay

ACM Conference on Human Factors in Computing Systems (CHI), 2023

Abs HTML PDF

Visualization misinformation is a prevalent problem, and combating it requires understanding people’s ability to read, interpret, and reason about erroneous or potentially misleading visualizations, which lacks a reliable measurement: existing visualization literacy tests focus on well-formed visualizations. We systematically develop an assessment for this ability by: (1) developing a precise definition of misleaders (decisions made in the construction of visualizations that can lead to conclusions not supported by the data), (2) constructing initial test items using a design space of misleaders and chart types, (3) trying out the provisional test on 497 participants, and (4) analyzing the test tryout results and refining the items using Item Response Theory, qualitative analysis, a wrong-due-to-misleader score, and the content validity index. Our final bank of 45 items shows high reliability, and we provide item bank usage recommendations for future tests and different use cases.