SoK: Automated Vulnerability Repair: Methods, Tools, and Assessments

The benchmark dataset Vul4C and experiment results for automated vulnerability repair tools in C/C++.

Table of Contents

1. Benchmark Dataset Vul4C
2. Experimental Tools
3. Repository Structure
4. Usage

1. Benchmark Dataset Vul4C

Our benchmark dataset Vul4C contains 144 vulnerabilities over 19 CWE types and 23 software.

Here is the statistics of Vul4C.

SH-SL = Single-Hunk, Single-Line; SH-ML = Single-Hunk, Multiple-Line;

MH-SF = Multiple-Hunk, Single-File; MH-SF = Multiple-Hunk, Multiple-File.

CWE Type	Total	SH-SL	SH-ML	MH-SF	MH-MF
CWE-119	35	7	11	11	6
CWE-125	29	2	5	12	10
CWE-476	16	1	4	10	1
CWE-369	11	1	8	2	0
CWE-190	9	0	0	6	3
CWE-787	14	4	1	5	4
CWE-20	6	0	1	2	3
CWE-416	4	0	0	4	0
CWE-835	4	0	1	2	1
CWE-189	2	1	0	1	0
CWE-617	2	0	1	1	0
CWE-120	1	0	0	0	1
CWE-415	1	0	1	0	0
CWE-704	1	0	0	1	0
CWE-770	1	0	1	0	0
CWE-191	1	1	0	0	0
CWE-682	1	0	0	0	1
CWE-843	1	0	0	1	0
N/A	5	0	1	3	1
Total	144	17	35	61	31

2. Experimental Tools

2.1 Automated Vulnerability Repair Tools

Tool	Venue	Repository
VulRepair	ESEC/FSE'22	https://github.com/awsm-research/VulRepair
VRepair	TSE'23	https://github.com/ASSERT-KTH/VRepair
VQM	TOSEM'24	https://github.com/awsm-research/VQM
VulMaster	ICSE'24	https://github.com/soarsmu/VulMaster_
ExtractFix	TOSEM'20	https://extractfix.github.io/
VulnFix	ISSTA'22	https://github.com/yuntongzhang/vulnfix
Senx	S&P'19	Not open source, but we asked the artifacts from the authors
Seader	ICPC'22	https://github.com/NiSE-Virginia-Tech/ying-ICPC-2022
SeqTrans	TSE'23	https://github.com/chijianlei/SeqTrans

2.2 Automated Program Repair Tools

Tool	Venue	Repository
CquenceR	ISSRE'21	https://github.com/epicosy/CquenceR
NTR	ICSE'25	https://sites.google.com/view/neuraltemplaterepair
ThinkRepair	ISSTA'24	https://github.com/vinci-grape/ThinkRepair
SRepair	arXiv'24	https://github.com/GhabiX/SRepair

3. Repository Structure

This repository is structured as follow:

|----- Vul4C-Benchmark
    |----- [Software]
        |----- [CVE ID]
            |----- [CVE ID]_[CWE ID]_[filename].diff 
            |----- [CVE ID]_[CWE ID]_[filename]_NEW.c
            |----- [CVE ID]_[CWE ID]_[filename]_OLD.c
            |----- README.txt 
            |----- exploit
            |----- setup.sh
|----- Vul4C_Src: Source code for command line tool.
|----- Framework
|----- Results
    |----- Results.xlsx: All experimental results.
    |----- [Vulnerability Repair Tools]
        |----- [Software]
            |----- [CVE ID]
                |----- 50-Candidates: This folder contains all 50 candidates generated by models. (Only for learning-based methods.)
                |----- Candidate Patches: This folder contains all patches generated by vulnerability repair tools. 
                                          (For learning-based methods, this folder contains all successfully restored patches within original 50 generated candidates.)
                |----- Compilable Patches: This folder contains all successfully compiled patches within all candidate patches.
                |----- Plausible Patches: This folder contains all patches that successfully pass vulnerability exploit test within all compilable patches.
                |----- Correct Patches: This folder contains all correct patches assessed by humans.
|----- test
    |----- [Software]
        |----- test.sh: Script used for compiling and testing the software.
|----- train_valid_data
    |----- train.csv: Train data for learning-based tools.
    |----- valid.csv: Valid data for learning-based tools.
|----- setup.py: setup script for command line tool.
|----- README.md

4. Usage

4.1 Manual

Please ensure that you have properly installed docker and Docker SDK for Python before starting.

You can use our experimental docker images, such as vul4c/vulnfix:1.0.

To use the Vul4C benchmark, you can follow these steps:

1. Run bash setup.sh in the CVE directory. This will generate the source directory and automatically compile the code.

2. Execute the exploit according to the commands in README.txt under the CVE directory. You need to replace /path/to/compile/source with the source path generated in step 1, and replace $FILE with the exploit path.

4.2 Command Line Tool

We have built a command-line tool for Vul4c.Take CVE-2017-9038 as example, its usage is as follows:

1. First, you need to install the command-line tool by running python3 setup.py install. After installation, you can enter vul4c in the command line to check whether the installation was successful.

2. To checkout a vulnerability into the specified directory, use the command:
   vul4c checkout -i CVE-2017-9038 -d /root/test/CVE-2017-9038

3. To compile the checked out vulnerability, use the command:
   vul4c compile -d /root/test/CVE-2017-9038

4. To exploit the checked out vulnerability, use the command:
   vul4c reproduce -d /root/test/CVE-2017-9038

5. To apply a patch to the checked out vulnerability, use the command:
   vul4c apply -d /root/test/CVE-2017-9038 -p patch.diff
   Here, the patch file should conform to the standard diff format. We recommend using the following command to generate the patch file:
   diff -u OLD.c NEW.c > patch.diff
   The format of the patch file is similar to the following:

    --- CVE-2017-9038_CWE-125_readelf.c_OLD.c       2025-06-06 16:26:00.000000000 +0000
    +++ CVE-2017-9038_CWE-125_readelf.c_NEW.c       2025-06-06 16:26:00.000000000 +0000
    @@ -7965,9 +7965,9 @@
        return FALSE;
    
    /* If the offset is invalid then fail.  */
    -  if (word_offset > (sec->sh_size - 4)
    -      /* PR 18879 */
    -      || (sec->sh_size < 5 && word_offset >= sec->sh_size)
    +  if (/* PR 21343 *//* PR 18879 */
    +      sec->sh_size < 4
    +      || word_offset > (sec->sh_size - 4)
        || ((bfd_signed_vma) word_offset) < 0)
        return FALSE;
   

6. Other commands include:

   vul4c info -i <CVE ID>            # Print information about a vulnerability  
   vul4c status                      # List vul4c requirements and their availability  
   vul4c test -d <checkout dir>      # Compile and exploit the checked out vulnerability

To reproduce the experiment, you can use the following command to startup on a selected repair tool and benchmark.

python3 Framework/vul4c.py --tool "VulnFix" --software "jasper" --CVEID "CVE-2016-10248" 

You can find the results in the folder vul4c-result,and the folder where the results will be stored will be named $tool_$CVEID_$timestamp

Other experimental tools could be found at docker repository vul4c.