Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models

We introduce Detection Alignment (DA), a metric that quantifies how well ML models localize vulnerabilities to specific vulnerable lines of code.

The implementation of the DA metric can be found in the file. For a complete explanation, please refer to the paper.

How To Use

• Set up Environment
• Download dataset and models
• Evaluate your vulnerability detector with DA
• Reproduce the experiments
• Cite our work

Set up Environment

# Clone the repository
git clone https://github.com/pralab/vuln-localization-eval.git
cd vuln-localization-eval

# (Optional) Create a conda environment
conda create -n da-evaluation python=3.10 -y
conda activate da-evaluation

# Install dependencies
pip install -r requirements.txt

Download dataset and pretrained models

Work in progress

Run DA Evaluation

Work in progress

Reproduce experiments

Example of CodeBERT usage that calculates line-level scores using attention values from encoder layer 0.

python main_exp.py \
  --test_data_file '/your/path/to/BigVul' \
  --model_path '/your/path/to/pretrained/model' \
  --model_type 'codexglue' \
  --block_size 512 \
  --block_index 0 \
  --device 'cpu' \
  --xai_method 'attention' \
  --seed 42 \
  --vuln_threshold 0.5 \

Contact

We welcome questions, suggestions, and contributions. Please open an issue or pull request to get in touch.

BibTex citation

@article{pintore2025evaluating,
  title={Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models},
  author={Pintore, Marco and Piras, Giorgio and Sotgiu, Angelo and Pintor, Maura and Biggio, Battista},
  journal={arXiv preprint arXiv:2510.11202},
  year={2025}
}

Acknowledgements

This work has been partly supported by the EU-funded Horizon Europe projects ELSA – European Lighthouse on Secure and Safe AI (GA no. 101070617) and Sec4AI4Sec - Cybersecurity for AI-Augmented Systems (GA no. 101120393); and by projects SERICS (PE00000014) and FAIR (PE00000013, CUP: J23C24000090007) under the MUR NRRP funded by the European Union - NextGenerationEU.