Justin Zhao

Research

Methodological research on LLM evaluation, evidence-traceable annotation, comparative AI detection, and causal inference for applied AI systems.

R&R at JCSSLLM AnnotationHPCText-as-Data

Using Messy Text in Future LLM Annotations

An LLM pipeline that uses codebook-guided attention to realign annotations with noisy text, producing clean, evidence-traceable summaries for downstream tasks.

  • Runs on HPC and enables reuse of annotations acquired over years of costly human coding without requiring re-annotation.
  • Reduces weeks of manual labor to hours of parallelized computation while retaining span-level evidence.
LLM EvaluationBenchmarking

The Scope Lines for Social Science Research Tasks Using LLMs

A benchmark framework for measuring how LLM bias affects annotation, extraction, classification, simulation, and other research tasks.

  • Measures bias across task and corpus risk levels rather than relying on generic QA and multiple-choice benchmarks.
  • Produces task-specific scope lines for deciding where unmitigated model bias is acceptable.
Computer VisionVision-Language ModelsRelease-Over-Release EvaluationBenchmarking

Do Newer Vision-Language Models Detect Better? A Release-Over-Release Evaluation on UAV Hyperspectral Landmine and UXO Screening

A release-over-release evaluation of three generations of vision-language models on UAV hyperspectral landmine and UXO screening.

  • Evaluated three generations of vision-language models on hyperspectral anomaly detection, benchmarked the rise and fall of performance across releases, produced an empirical analysis of AI-assisted detection on a safety-critical humanitarian computer vision detection task.
  • Designed a perturbation experiment to diagnose the failure modes of vision-language models on high stakes tasks, and provided guidance for future research and development.
Information ExtractionNamed Entity ExtractionBenchmarking

LLM-Based Rhetoric Extraction from Censorship-Related Legal Texts

This paper extracts rights- and security-related rhetoric from censorship legislation and measures it at cross-national scale.

  • Benchmarks two extraction pipelines against each other, establishing a reproducible, auditable method for measuring censorship rhetoric across national legal corpora.
  • Span-verifiable annotations make every measurement checkable and the cross-national comparison reproducible at scale.
Synthetic ControlCausal Inference

Contextualized Synthetic Control

A methodological extension to synthetic control for settings where the treated unit has no comparable donors in the conventional donor pool.

  • Uses multi-head attention to produce counterfactuals under severe scale disparity.
  • Extends causal inference to cases previously excluded by donor-pool comparability requirements.