# cve-bin-tool tests

You can see all existing tests in [`test/`](https://github.com/ossf/cve-bin-tool/tree/main/test)

- [cve-bin-tool tests](#cve-bin-tool-tests)
  - [Running all tests](#running-all-tests)
  - [Running a single test](#running-a-single-test)
  - [Running tests on different versions of Python](#running-tests-on-different-versions-of-python)
  - [Adding new tests: CVE mapping tests](#adding-new-tests-cve-mapping-tests)
  - [Adding new tests: Signature tests against real files](#adding-new-tests-signature-tests-against-real-files)
  - [Adding new tests: Checker filename mappings](#adding-new-tests-checker-filename-mappings)
  - [Known issues](#known-issues)

## Running all tests

To run all tests for `cve-bin-tool`:

```console
pytest
```

To run scanner and checker tests:

```console
pytest test/test_scanner.py test/test_checkers.py
```

By default, some longer-running tests are turned off. If you want to enable them, you can set the environment variable `LONG_TESTS` to 1. You can do this just in single command as follows:

```console
LONG_TESTS=1 pytest {$TEST_FILE}
```

For scanner tests:

```console
LONG_TESTS=1 pytest test/test_scanner.py test/test_checkers.py
```

By default, tests which rely on external connectivity are turned off. If you want to enable them, you can set the environment variable `EXTERNAL_SYSTEM` to 1. You can do this just in single command as follows:

```console
EXTERNAL_SYSTEM=1 pytest {$TEST_FILE}
```

For NVD tests

```console
EXTERNAL_SYSTEM=1 pytest test/test_source_nvd.py
```

## Running a single test

To run a single test, you can use the unittest framework. For example, here's how to run
the test for `sqlite`:

```console
python -m unittest test.test_scanner.TestScanner.test_sqlite_3_12_2
```

To run a single test in test_scanner you can use the pytest framework. For example, here's how to run the test for `vendor_package_pairs`:

```console
 pytest test/test_scanner.py::TestScanner::test_version_mapping
```

## Running tests on different versions of Python

Our CI runs tests on all currently supported Python versions under Linux.
The [testing configuration file is available on GitHub](https://github.com/ossf/cve-bin-tool/blob/main/.github/workflows/testing.yml). This file will always show the currently used versions on both Linux and Windows.

The recommended way to do this yourself is to use python's `virtualenv`

You can set up virtualenv for all these environments:

```console
virtualenv -p python3.11 venv3.11
virtualenv -p python3.12 venv3.12
```

To activate one of these (the example uses 3.12), run the tests, and deactivate:

```console
source venv3.12/bin/activate
pytest
deactivate

```

## Adding new tests: CVE mapping tests

- You can see the code for scanner tests in ['test/test_scanner.py'](https://github.com/ossf/cve-bin-tool/blob/main/test/test_scanner.py)
- You can see checker wise test data in ['test/test_data'](https://github.com/ossf/cve-bin-tool/blob/main/test/test_data)
- If you just want to add a new mapping test for a checker, add a dictionary of _product_, _version_ and _version_strings_ in the mapping_test_data list. Here, _version_strings_ are the list of strings that contain version signature or strings that commonly can be found in the module. For example: this is how the current mapping_test_data for gnutls look like. You should add the details of the new test case data at the end of `mapping_test_data` list:

```python
mapping_test_data = [
    {
        "product": "gnutls",
        "version": "2.1.6",
        "version_strings": ["gnutls-cli 2.1.6"]
    },
    {
        "product": "gnutls",
        "version": "2.3.11",
        "version_strings": ["gnutls-serv 2.3.11"],
    },
]
```

> Note: sometimes the database we're using doesn't have perfect mapping between CVEs and product versions -- if you try to write a test that doesn't work because of that mapping but the description in the CVE says that version should be vulnerable, don't discard it! Instead, please make a note of it in a GitHub issue so it can be investigated and possibly reported upstream.

## Adding new tests: Signature tests against real files

To make the basic test suite run quickly, we create "faked" binary files to test the **CVE mappings**. However, we want to be able to test real files to test that the **signatures** work on real-world data.

You can see test data for package tests in _package_test_data_ variable of the test data file you are writing test for.

We have a `test_version_in_package` function in `test_scanner` that takes in package details (_url_, _package name_, _module name_, _version_), downloads the package, runs the scanner against it, and confirms it is the package that you've specified. But we need more tests!

- To add a new test, find an appropriate publicly available file (linux distribution packages and public releases of the packages itself are ideal). You should add the details of the new test case in the `package_test_data` variable of the file for which you are writing test for. For example: this is how the current `package_test_data` for binutils look like. You should add the details of the new test case data at the end of `package_test_data` list:

```python
package_test_data = [
    {
        "url": "http://security.ubuntu.com/ubuntu/pool/main/b/binutils/",
        "package_name": "binutils_2.26.1-1ubuntu1~16.04.8_amd64.deb",
        "product": "binutils",
        "version": "2.26.1",
         "other_products": [],
    },
    {
        "url": "http://mirror.centos.org/centos/7/os/x86_64/Packages/",
        "package_name": "binutils-2.27-43.base.el7.x86_64.rpm",
        "product": "binutils",
        "version": "2.27",
         "other_products": ["zlib"],
    },
]

```
The ```other_products``` attribute might match any binaries provided, so we can check that only the expected products are found in a given binary. (e.g. if an imaginary package called `CryptographyExtensions` included OpenSSL, we'd expect to detect both in `CryptographyExtensions-1.2.rpm`).

Ideally, we should have at least one such test for each checker, and it would be nice to have some different sources for each as well. For example, for packages available in common Linux distributions, we might want to have one from Fedora, one from Debian, and one direct from upstream to show that we detect all of those versions.

> Note: we're getting the `LONG_TESTS()` from tests.util in the top of the files where it's being used. If you're adding a long test to a test file that previously didn't have any, you'll need to add that at the top of the file as well.

## Adding new tests: Checker filename mappings

To test the filename mappings, rather than making a bunch of empty files, we're calling the checkers directly in `test/test_checkers.py`. You can add a new test by specifying the name of the checker you want to test, the file name, and the expected result that the scanner should say it "is".

```python
    @pytest.mark.parametrize(
        "checker_name, file_name, expected_result",
        [
            ("python", "python", ["python"]),
            ("python", "python3.8", ["python"]),
        ],
    )
```

The function `test_filename_is` will then load the checker you have specified (and fail spectacularly if you specify a checker that does not exist), try to run `get_versions()` with an empty file content and the filename you specified, then check that it "is" something (as opposed to "contains") and that the modulename that `get_versions` returns is in fact the `expected_result` you specified.

For ease of maintenance, please keep the parametrize list in alphabetical order when you add a new tests.

You can then run your new test using pytest:

```console
pytest test/test_checkers.py
```

You can also use all the pytest functionality to run groups of tests. For example, this will run the python-related tests (but not the bluetooth one):

```console
pytest -v test/test_checkers.py -k python
```

## Known issues

If you're using Windows and plan to run PDF tests, we **strongly** recommend also `pdftotext`. We experienced problems running tests without this. The best approach to do this is through [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html) (click [here](https://anaconda.org/conda-forge/pdftotext) to find out how to install this package with conda).