EVEREST

Endogenous Vision-Language Reinforcement Reasoning Exploration for Urban Socio-Semantic Segmentation

EVEREST recovers the pixel-level extent of socially defined urban entities from a digital map, a spatially aligned satellite image, and a textual target. It actively enumerates candidate instances, renders coarse segmentation feedback, verifies instance boundaries, and produces executable box-and-point prompts for a frozen SAM2 segmenter.

• 📰 News
• 📌 Introduction
• ✨ Key Features
• 🏗️ Architecture
• 📊 Results
• 📦 Installation
• 📂 Data Preparation
• 🚀 Training
• 🔍 Inference
• ⚙️ Configuration
• 📁 Project Structure
• 📝 Citation

⬆ Back to top

• 2026-08-25: Our paper, EVEREST: Endogenous Vision-Language Reinforcement Reasoning Exploration for Urban Socio-Semantic Segmentation, is available on arXiv.

⬆ Back to top

Appearance alone is often insufficient to distinguish socially defined regions such as schools, hospitals, parks, residential areas, and commercial districts. EVEREST treats this task as an interactive multimodal reasoning problem rather than a one-shot mask prediction problem.

The shared vision-language policy first enumerates candidate entities with stable instance identities and bounding boxes. A frozen SAM2 model converts those boxes into a coarse mask, which is rendered back onto the aligned map and satellite image. The policy then verifies every indexed instance, chooses whether to keep, adjust, or drop it, and places positive boundary-refinement points. Because text generation, parsing, rendering, and segmentation are non-differentiable, the shared policy is optimized with group-relative reinforcement learning.

⬆ Back to top

• Pseudocode-guided enumeration discovers candidate entities and anchors each one with a stable instance identity.
• Egocentric verification uses rendered map-satellite feedback to inspect boundaries and perform keep, adjust, or drop decisions.
• Executable visual primitives merge instance-aware boxes and positive points before frozen SAM2 execution.
• Reinforcement reasoning optimization trains the VLM across a non-differentiable parsing, rendering, and segmentation workflow.
• Hierarchical urban semantics supports Socio-name, Socio-class, and Socio-function targets.

⬆ Back to top

⬆ Back to top

⬆ Back to top

• Linux with CUDA
• Python 3.10
• Four high-memory NVIDIA GPUs for the released 4-GPU configuration

Create the environment from the repository root:

conda create -n everest python=3.10 -y
conda activate everest
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0
pip install -r requirements.txt
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install "transformer-engine[pytorch]==2.2.0" deepspeed==0.16.4 vllm==0.8.4 --no-build-isolation

The launchers add the repository root to PYTHONPATH; an editable package installation is not required.

⬆ Back to top

The dataset and model checkpoints are intentionally not included. Set EVEREST_DATASET to the SocioSeg root with the following layout:

SocioSeg/
|-- train/
|   `-- <sample-id>/
|       |-- question.json
|       |-- map.png
|       |-- sat.png
|       `-- mask.png
|-- val/
|   `-- <sample-id>/
|       |-- question.json
|       |-- map.png
|       |-- sat.png
|       `-- mask.png
`-- test/
    `-- <sample-id>/
        |-- question.json
        |-- map.png
        |-- sat.png
        `-- mask.png

Each question.json must contain the problem field consumed by the data loader.

export EVEREST_DATASET=/absolute/path/to/SocioSeg

Optional cache and Ray temporary directories can be configured without editing source files:

export EVEREST_HF_HOME=/absolute/path/to/huggingface-cache
export EVEREST_RAY_TMPDIR=/absolute/path/to/ray-temp

⬆ Back to top

The canonical training configuration starts from Qwen/Qwen2.5-VL-3B-Instruct.

export EVEREST_DATASET=/absolute/path/to/SocioSeg
bash examples/train.sh

On Slurm, pass resource requests at submission time so the launcher remains cluster-independent:

sbatch --gpus=4 --export=ALL,EVEREST_DATASET=/absolute/path/to/SocioSeg examples/train.sh

A timestamped runtime configuration is created under examples/train/. Checkpoints and logs are written below examples/output/train/<MM_DD_HH_MM>/.

⬆ Back to top

Point EVEREST_CHECKPOINT to a trained checkpoint. The path may contain the training timestamp; otherwise a sanitized checkpoint name is used for the inference run directory.

export EVEREST_DATASET=/absolute/path/to/SocioSeg
export EVEREST_CHECKPOINT=/absolute/path/to/checkpoint
bash examples/infer.sh

Inference artifacts are written below examples/output/infer/infer_<run-id>/result/, including stage-one masks, stage-two masks, rendered feedback, model responses, and aggregate metrics.

⬆ Back to top

The canonical files are:

• examples/train/rlvr_megatron.yaml
• examples/infer/rlvr_megatron.yaml
• examples/config/deepspeed_zero.yaml
• examples/config/deepspeed_zero2.yaml
• examples/config/deepspeed_zero3.yaml
• examples/config/deepspeed_zero3_cpuoffload.yaml

Both canonical task configurations map the actor, inference, segmentation, and reference workers across four GPUs. Adjust the device_mapping, tensor-parallel size, batch sizes, and memory utilization for a different cluster.

The default models are:

• Vision-language backbone: Qwen/Qwen2.5-VL-3B-Instruct
• Promptable segmenter: facebook/sam2-hiera-large

⬆ Back to top

EVEREST/
|-- assets/
|   |-- architecture.png
|   |-- everest-banner.png
|   `-- results.png
|-- examples/
|   |-- config/
|   |-- infer/
|   |-- train/
|   |-- infer.sh
|   |-- train.sh
|   |-- start_rlvr_socioseg_pipeline.py
|   `-- start_rlvr_socioseg_pipeline_infer.py
|-- mcore_adapter/
|-- megatron/
|-- roll/
|-- requirements.txt
`-- README.md

⬆ Back to top

If you find this work useful, please cite:

@article{li2026everest,
  title={EVEREST: Endogenous Vision-Language Reinforcement Reasoning Exploration for Urban Socio-Semantic Segmentation},
  author={Li, Qixiu and He, Zhongzhi and Zhu, Xiang and Li, Xiaoyong and Lin, Jiarun and Xu, Weifeng},
  journal={arXiv preprint arXiv:2608.24640},
  year={2026}
}

⬆ Back to top