UrbanX

[Paper] [Project Page] [Code]

Code for "From Street Views to Urban Science: Discovering Road Safety Factors with Multimodal Large Language Models"

Published in Transportation Research Part C: Emerging Technologies, Volume 188, Article 105692.

If you find this work useful or interesting, please consider starring the repository and citing our paper.

News

• [2026/05/31] We release the open-source UrbanX toolkit with OpenRouter text inference, local vLLM VQA support, real SVI demo fixtures, and an academic project page.

Highlights

TL;DR: UrbanX uses multimodal large language models to discover interpretable, statistically testable road-safety factors from street-view imagery.

• Generates human-readable hypotheses about visually observable urban and transportation features.
• Uses visual question answering over street-view images to convert hypotheses into interpretable categorical embeddings.
• Assesses discovered factors with transparent regression models, p-values, metrics, and reusable run artifacts.
• Supports OpenRouter for text hypothesis generation and local vLLM for the VQA stage.
• Ships with four real SVI examples so the framework can be tested without the full private dataset.

Abstract

Urban and transportation research has long sought to uncover statistically meaningful relationships between key variables and societal outcomes such as road safety, aiming to generate actionable insights that guide the planning, development, and renewal of urban mobility systems. However, traditional workflows face several key challenges: (1) reliance on human experts to propose hypotheses, which can be time-consuming and prone to confirmation bias; (2) limited interpretability, particularly in deep learning approaches; and (3) underutilization of unstructured data that encodes critical urban context. To address these limitations, we propose a Multimodal Large Language Model (MLLM)-based approach for interpretable hypothesis inference, enabling the automated generation, assessment, and refinement of hypotheses concerning urban form and transportation safety. Specifically, we leverage MLLMs to generate road safety-relevant questions and automatically answer them based on street view images (SVIs) through visual question answering (VQA). These responses are used to construct interpretable embeddings for each SVI, which are then incorporated into linear statistical models for transparent and explainable regression analysis. UrbanX supports iterative hypothesis testing and refinement guided by statistical evidence, such as coefficient significance, thereby enabling rigorous, transparent scientific discovery of previously overlooked correlations between urban design and transportation risk. We evaluate our framework on Manhattan street segments and demonstrate that it outperforms pretrained deep learning baselines while offering full interpretability. We demonstrate that UrbanX matches or exceeds the explanatory power of existing expert-curated built environment variables, validating its potential to replace labor-intensive feature engineering with automated, scalable discovery of potential safety-related factors. Beyond road safety, UrbanX can serve as a general-purpose foundation for hypothesis-driven urban mobility analysis, extracting structured insights from unstructured data across diverse socioeconomic and environmental outcomes. This approach establishes a scalable and trustworthy pathway for interpretable, data-driven scientific discovery in urban and transportation systems using foundation models such as MLLMs.

Method

UrbanX contains three iterative modules:

1. Hypothesis Generation: LLMs propose diverse, visually observable road-safety hypotheses.
2. Embedding Construction: MLLMs answer each hypothesis over SVIs through VQA and produce interpretable embeddings.
3. Hypothesis Assessment: transparent regression models evaluate predictive power, statistical significance, and evidence for refinement.

Usage

Below are the instructions for running the open-source version of UrbanX. If you encounter issues during setup, data preparation, inference, or evaluation, please open a GitHub issue.

Repository Structure

.
├── assets/
├── configs/
│   ├── demo_mock.yaml
│   ├── demo_openrouter.yaml
│   └── demo_local_vllm.yaml
├── docs/
│   └── DATA_PREPARATION.md
├── examples/
│   └── data/
│       ├── demo_records.csv
│       ├── example_hypotheses.json
│       └── svi/
├── scripts/
│   └── prepare_manifest.py
├── src/
│   └── urbanx/
│       ├── providers/
│       ├── cli.py
│       ├── embedding.py
│       ├── hypothesis.py
│       ├── modeling.py
│       └── pipeline.py
└── tests/

Environment Setup

UrbanX requires Python 3.10 or newer.

Install the package and development dependencies:

python3 -m pip install -e ".[dev]"

For OpenRouter text inference, set your API key:

cp .env.example .env
export OPENROUTER_API_KEY="..."

For local vLLM VQA inference in a GPU environment:

python3 -m pip install -e ".[vllm]"

Optional local LMDeploy support is available with:

python3 -m pip install -e ".[local]"

Run the Demo

Run the no-cost mock demo:

urbanx run --config configs/demo_mock.yaml
urbanx evaluate --run runs/demo_mock

The mock demo uses examples/data/demo_records.csv, which points to four real SVI sample images under examples/data/svi/. This verifies the full pipeline without network calls, API cost, or GPU requirements.

Run OpenRouter text inference with local vLLM VQA:

export OPENROUTER_API_KEY="..."
urbanx run --config configs/demo_openrouter.yaml

Run local vLLM VQA with fixed example hypotheses:

urbanx run --config configs/demo_local_vllm.yaml

Data Preparation

UrbanX expects a CSV manifest with the following columns:

column	required	description
id	yes	Unique image or road-segment id.
image_uri	yes	Local image path, data URL, or public image URL. Relative paths are resolved from the manifest directory.
target	recommended	Numeric outcome such as crash rate. Required for model fitting.
split	optional	train, val, or test. If absent, UrbanX creates a train/test split.

Use the helper script when you have separate image and target tables:

python scripts/prepare_manifest.py \
  --images-csv path/to/image_index.csv \
  --targets-csv path/to/targets.csv \
  --join-key segment_id \
  --image-column image_path \
  --target-column crash_rate \
  --output data/private/records.csv

For more detail, see docs/DATA_PREPARATION.md.

Output Files

Each run writes a reproducible artifact directory:

runs/<run_id>/
├── config.yaml
├── hypotheses.json
├── embeddings.parquet
├── metrics.json
├── cache.sqlite
├── predictions.csv
└── figures/

The cache is keyed by prompt, model, image URI, and hypothesis, so repeated runs can reuse VQA outputs when the same configuration is used.

Data Policy

The full SVI collection, private target tables, large experiment caches, and unclear-license intermediate artifacts are not redistributed. The repository includes only four real SVI sample images for demonstration and testing. Before publishing a public fork or release, confirm that these example images can be redistributed under the repository's release policy; otherwise replace them with images for which you have explicit redistribution rights. For scientific experiments, prepare a private manifest that points to your own image files and target variables.

The public downloader only fetches public open data samples:

urbanx data fetch-nyc-demo --output data/nyc_collisions_sample.csv --limit 500

Release Disclaimer

This open-source repository was organized and packaged with assistance from OpenAI Codex. The authors reviewed the release structure, source code, documentation, and public-facing assets and remain responsible for maintaining the project. For questions, bug reports, data-preparation issues, or reproducibility concerns, please open an issue on GitHub so the discussion can be tracked and addressed transparently.

License

Code is released under the MIT License. The accompanying project page is hosted at yihongt.github.io/projects/urbanx.

Citation

If you find this work helpful for your research, please consider giving this repo a star and citing our paper:

@article{tang2026street,
  title={From street views to urban science: Discovering road safety factors with multimodal large language models},
  author={Tang, Yihong and Qu, Ao and Yu, Xujing and Deng, Weipeng and Ma, Jun and Zhao, Jinhua and Sun, Lijun},
  journal={Transportation Research Part C: Emerging Technologies},
  volume={188},
  pages={105692},
  year={2026},
  publisher={Elsevier}
}