Content-Length: 8629 | pFad | http://github.com/gitpython-developers/GitPython/pull/1904.patch

thub.com From fdce8375c80abd02b1dc08bd218f09849fea8233 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Sat, 20 Apr 2024 16:48:00 -0400 Subject: [PATCH 1/2] Dockerized "Direct Execution of Fuzz Targets" Adds a Dockerfile to enable easily executing the fuzz targets directly inside a container environment instead of directly on a host machine. This addresses concerns raised in PR #1901 related to how `fuzz_tree.py` writes to the real `/tmp` directory of the file system it is executed on as part of setting up its own test fixtures, but also makes for an easier to use development workflow. See this related comment on PR #1901 for additional context: https://github.com/gitpython-developers/GitPython/pull/1901#issuecomment-2063818998 --- fuzzing/README.md | 43 ++++++++++++++++++++++++---- fuzzing/local-dev-helpers/Dockerfile | 22 ++++++++++++++ 2 files changed, 59 insertions(+), 6 deletions(-) create mode 100644 fuzzing/local-dev-helpers/Dockerfile diff --git a/fuzzing/README.md b/fuzzing/README.md index ab9f6a63f..0a62b4c85 100644 --- a/fuzzing/README.md +++ b/fuzzing/README.md @@ -44,8 +44,7 @@ capabilities, jump into the "Getting Started" section below. ### Setting Up Your Local Environment Before contributing to fuzzing efforts, ensure Python and Docker are installed on your machine. Docker is required for -running fuzzers in containers provided by OSS-Fuzz. [Install Docker](https://docs.docker.com/get-docker/) following the -official guide if you do not already have it. +running fuzzers in containers provided by OSS-Fuzz and for safely executing test files directly. [Install Docker](https://docs.docker.com/get-docker/) following the official guide if you do not already have it. ### Understanding Existing Fuzz Targets @@ -110,19 +109,51 @@ Includes scripts for building and integrating fuzz targets with OSS-Fuzz: - [OSS-Fuzz documentation on the build.sh](https://google.github.io/oss-fuzz/getting-started/new-project-guide/#buildsh) - [See GitPython's build.sh and Dockerfile in the OSS-Fuzz repository](https://github.com/google/oss-fuzz/tree/master/projects/gitpython) +### Local Development Helpers (`local-dev-helpers/`) + +Contains tools to make local development tasks easier. +See [the "Running Fuzzers Locally" section below](#running-fuzzers-locally) for further documentation and use cases related to files found here. + ## Running Fuzzers Locally +> [!WARNING] +> **Some fuzz targets in this repository write to the filesystem** during execution. +> For that reason, it is strongly recommended to **always use Docker when executing fuzz targets**, even when it may be +> possible to do so without it. +> +> Although [I/O operations such as writing to disk are not considered best practice](https://github.com/google/fuzzing/blob/master/docs/good-fuzz-target.md#io), the current implementation of at least one test requires it. +> See [the "Setting Up Your Local Environment" section above](#setting-up-your-local-environment) if you do not already have Docker installed on your machine. +> +> PRs that replace disk I/O with in-memory alternatives are very much welcomed! + ### Direct Execution of Fuzz Targets -For quick testing of changes, [Atheris][atheris-repo] makes it possible to execute a fuzz target directly: +Directly executing fuzz targets allows for quick iteration and testing of changes which can be helpful during early +development of new fuzz targets or for validating changes made to an existing test. +The [Dockerfile](./local-dev-helpers/Dockerfile) located in the `local-dev-helpers/` subdirectory provides a lightweight +container environment preconfigured with [Atheris][atheris-repo] that makes it easy to execute a fuzz target directly. + +**From the root directory of your GitPython repository clone**: -1. Install Atheris following the [installation guide][atheris-repo] for your operating system. -2. Execute a fuzz target, for example: +1. Build the local development helper image: ```shell -python fuzzing/fuzz-targets/fuzz_config.py +docker build -f fuzzing/local-dev-helpers/Dockerfile -t gitpython-fuzzdev . ``` +2. Then execute a fuzz target inside the image, for example: + +```shell + docker run -it -v "$PWD":/src gitpython-fuzzdev python fuzzing/fuzz-targets/fuzz_config.py -atheris_runs=10000 +``` + +The above command executes [`fuzz_config.py`](./fuzz-targets/fuzz_config.py) and exits after `10000` runs, or earlier if +the fuzzer finds an error. + +Docker CLI's `-v` flag specifies a volume mount in Docker that maps the directory in which the command is run (which +should be the root directory of your local GitPython clone) to a directory inside the container, so any modifications +made between invocations will be reflected immediately without the need to rebuild the image each time. + ### Running OSS-Fuzz Locally This approach uses Docker images provided by OSS-Fuzz for building and running fuzz tests locally. It offers diff --git a/fuzzing/local-dev-helpers/Dockerfile b/fuzzing/local-dev-helpers/Dockerfile new file mode 100644 index 000000000..77808ed1d --- /dev/null +++ b/fuzzing/local-dev-helpers/Dockerfile @@ -0,0 +1,22 @@ +# syntax=docker/dockerfile:1 + +# Use the same Python version as OSS-Fuzz to accidental incompatibilities in test code +FROM python:3.8-slim + +LABEL project="GitPython Fuzzing Local Dev Helper" + +WORKDIR /src + +COPY . . + +# Update package managers, install necessary packages, and cleanup unnecessary files in a single RUN to keep the image smaller. +RUN apt-get update && \ + apt-get install --no-install-recommends -y git && \ + python -m pip install --upgrade pip && \ + python -m pip install atheris && \ + python -m pip install -e . && \ + apt-get clean && \ + apt-get autoremove -y && \ + rm -rf /var/lib/apt/lists/* /root/.cache + +CMD ["bash"] From f1451219c5a3a221615d3d38ac251bf5bbe46119 Mon Sep 17 00:00:00 2001 From: DaveLak Date: Sun, 21 Apr 2024 12:19:57 -0400 Subject: [PATCH 2/2] Fix Atheris install in local dev helper Docker image The Atheris package bundles a binary that supplies libFuzzer on some host machines, but in some cases (such as ARM based mac hosts) Atheris seems to require building libFuzzer at install time while pip builds the wheel. In the latter case, clang and related dependencies must be present and available for the build, which itself requires using a non "slim" version of the Python base image and not passing the `--no-install-recommends` flag to `apt-get install` as both prevent the required related libraries from being automatically installed. It is also worth noting that at the time of this commit, the default version of LLVM & Clang installed when `clang` is installed from `apt` is version 14, while the latest stable version is 17 and OSS-Fuzz uses 15. The decision to install the default version (14) available via the debian repos was intentional because a) it appears to work fine for our needs and Atheris version b) specifying a different version requires more complexity depending on install method, but the goal of this Dockerfile is simplicity and low maintenance. If it becomes neccissary to upgrade Clang/LLVM in the future, one option to consider besides installing from source is the apt repository maintained by the LLVM project: https://apt.llvm.org/ See the discussion in this issue for additional context to this change: https://github.com/gitpython-developers/GitPython/pull/1904 --- fuzzing/local-dev-helpers/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/fuzzing/local-dev-helpers/Dockerfile b/fuzzing/local-dev-helpers/Dockerfile index 77808ed1d..426de05dd 100644 --- a/fuzzing/local-dev-helpers/Dockerfile +++ b/fuzzing/local-dev-helpers/Dockerfile @@ -1,7 +1,7 @@ # syntax=docker/dockerfile:1 # Use the same Python version as OSS-Fuzz to accidental incompatibilities in test code -FROM python:3.8-slim +FROM python:3.8-bookworm LABEL project="GitPython Fuzzing Local Dev Helper" @@ -11,12 +11,12 @@ COPY . . # Update package managers, install necessary packages, and cleanup unnecessary files in a single RUN to keep the image smaller. RUN apt-get update && \ - apt-get install --no-install-recommends -y git && \ + apt-get install -y git clang && \ python -m pip install --upgrade pip && \ python -m pip install atheris && \ python -m pip install -e . && \ apt-get clean && \ apt-get autoremove -y && \ - rm -rf /var/lib/apt/lists/* /root/.cache + rm -rf /var/lib/apt/lists/* CMD ["bash"]

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!