diff-cover

Run coverage and linting reports on diffs

Description

diff-cover |pypi-version| |conda-version| |build-status|
========================================================================================

Automatically find diff lines that need test coverage.
Also finds diff lines that have violations (according to tools such
as pycodestyle, pyflakes, flake8, or pylint).
This is used as a code quality metric during code reviews.

Overview
--------

Diff coverage is the percentage of new or modified
lines that are covered by tests.  This provides a clear
and achievable standard for code review: If you touch a line
of code, that line should be covered.  Code coverage
is *every* developer's responsibility!

The ``diff-cover`` command line tool compares an XML coverage report
with the output of ``git diff``.  It then reports coverage information
for lines in the diff.

Currently, ``diff-cover`` requires that:

- You are using ``git`` for version control.
- Your test runner generates coverage reports in Cobertura, Clover
  or JaCoCo XML format, or LCov format.

Supported XML or LCov coverage reports can be generated with many coverage tools,
including:

- Cobertura__ (Java)
- Clover__ (Java)
- JaCoCo__ (Java)
- coverage.py__ (Python)
- JSCover__ (JavaScript)
- lcov__ (C/C++)

__ http://cobertura.sourceforge.net/
__ http://openclover.org/
__ https://www.jacoco.org/
__ http://nedbatchelder.com/code/coverage/
__ http://tntim96.github.io/JSCover/
__ https://ltp.sourceforge.net/coverage/lcov.php


``diff-cover`` is designed to be extended.  If you are interested
in adding support for other version control systems or coverage
report formats, see below for information on how to contribute!


Installation
------------

To install the latest release:

.. code:: bash

    pip install diff-cover


To install the development version:

.. code:: bash

    git clone https://github.com/Bachmann1234/diff-cover.git
    cd diff-cover
    poetry install
    poetry shell


Getting Started
---------------

1. Set the current working directory to a ``git`` repository.

2. Run your test suite under coverage and generate a [Cobertura, Clover or JaCoCo] XML report.
   For example, using `pytest-cov`__:

.. code:: bash

    pytest --cov --cov-report=xml

__ https://pypi.org/project/pytest-cov

This will create a ``coverage.xml`` file in the current working directory.

**NOTE**: If you are using a different coverage generator, you will
need to use different commands to generate the coverage XML report.


3. Run ``diff-cover``:

.. code:: bash

    diff-cover coverage.xml

This will compare the current ``git`` branch to ``origin/main`` and print
the diff coverage report to the console.

You can also generate an HTML, JSON or Markdown version of the report:

.. code:: bash

    diff-cover coverage.xml --format html:report.html
    diff-cover coverage.xml --format json:report.json
    diff-cover coverage.xml --format markdown:report.md
    diff-cover coverage.xml --format html:report.html,markdown:report.md

Multiple XML Coverage Reports
-------------------------------

In the case that one has multiple xml reports form multiple test suites, you
can get a combined coverage report (a line is counted  as covered if it is
covered in ANY of the xml reports) by running ``diff-cover`` with multiple
coverage reports as arguments. You may specify any arbitrary number of coverage
reports:

.. code:: bash

    diff-cover coverage1.xml coverage2.xml

Quality Coverage
-----------------
You can use diff-cover to see quality reports on the diff as well by running
``diff-quality``.

.. code :: bash

    diff-quality --violations=<tool>

Where ``tool`` is the quality checker to use. Currently ``pycodestyle``, ``pyflakes``,
``flake8``, ``pylint``, ``checkstyle``, ``checkstylexml``, ``ruff.check``, ``clang`` are supported, but more
checkers can (and should!) be supported. See the section "Adding `diff-quality``
Support for a New Quality Checker".

NOTE: There's no way to run ``findbugs`` from ``diff-quality`` as it operating
over the generated java bytecode and should be integrated into the build
framework.

Like ``diff-cover``, HTML, JSON or Markdown reports can be generated with

.. code:: bash

    diff-quality --violations=<tool> --format html:report.html
    diff-quality --violations=<tool> --format json:report.json
    diff-quality --violations=<tool> --format markdown:report.md
    diff-quality --violations=<tool> --format html:report.html,markdown:report.md

If you have already generated a report using ``pycodestyle``, ``pyflakes``, ``flake8``,
``pylint``, ``checkstyle``, ``checkstylexml``, or ``findbugs`` you can pass the report
to ``diff-quality``.  This is more efficient than letting ``diff-quality`` re-run
``pycodestyle``, ``pyflakes``, ``flake8``, ``pylint``, ``checkstyle``, or ``checkstylexml``.

.. code:: bash

    # For pylint < 1.0
    pylint -f parseable > pylint_report.txt

    # For pylint >= 1.0
    pylint --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" > pylint_report.txt

    # Use the generated pylint report when running diff-quality
    diff-quality --violations=pylint pylint_report.txt

    # Use a generated pycodestyle report when running diff-quality.
    pycodestyle > pycodestyle_report.txt
    diff-quality --violations=pycodestyle pycodestyle_report.txt

Note that you must use the ``-f parseable`` option to generate
the ``pylint`` report for pylint versions less than 1.0 and the
``--msg-template`` option for versions >= 1.0.

``diff-quality`` will also accept multiple ``pycodestyle``, ``pyflakes``, ``flake8``,
or ``pylint`` reports:

.. code:: bash

    diff-quality --violations=pylint report_1.txt report_2.txt

If you need to pass in additional options you can with the ``options`` flag

.. code:: bash

    diff-quality --violations=pycodestyle --options="--exclude='*/migrations*' --statistics" pycodestyle_report.txt

Compare Branch
--------------

By default, ``diff-cover`` compares the current branch to ``origin/main``.  To specify a different compare branch:

.. code:: bash

    diff-cover coverage.xml --compare-branch=origin/release

Diff File
--------------

You may provide a file containing the output of ``git diff`` to ``diff-cover`` instead of using a branch name.

For example, Say you have 2 branches ``main`` and ``feature``. Lets say after creating and checking out the feature branch,
you make commits ``A``, ``B``, and ``C`` in that order.


If you want to see all changes between the ``feature`` and ``main`` branch, you can generate a diff file like this:

.. code:: bash

    git diff main..feature > diff.txt

If you want to see the changes between the ``feature`` branch and the commit ``A``, you can generate a diff file using the following command:

.. code:: bash

    git diff A..feature > diff.txt

You can then run ``diff-cover`` with the diff file as an argument:

.. code:: bash

    diff-cover coverage.xml --diff-file=diff.txt

Total Percent Formatting
------------------------

By default, ``diff-cover`` and ``diff-quality`` round the total coverage/quality
percentage to an integer. To keep up to two decimal places, use
``--total-percent-float``:

.. code:: bash

    diff-cover coverage.xml --total-percent-float
    diff-quality --violations=pycodestyle --total-percent-float

Fail Under
----------

To have ``diff-cover`` and ``diff-quality`` return a non zero status code if the report quality/coverage percentage is
below a certain threshold specify the fail-under parameter

.. code:: bash

    diff-cover coverage.xml --fail-under=80
    diff-quality --violations=pycodestyle --fail-under=80

The above will return a non zero status if the coverage or quality score was below 80%.

Exclude/Include paths
---------------------

Explicit exclusion of paths is possible for both ``diff-cover`` and ``diff-quality``, while inclusion is
only supported for ``diff-quality`` (since 5.1.0).

The exclude option works with ``fnmatch``, include with ``glob``. Both options can consume multiple values.
Include options should be wrapped in double quotes to prevent shell globbing. Also they should be relative to
the current git directory.

.. code:: bash

    diff-cover coverage.xml --exclude setup.py
    diff-quality --violations=pycodestyle --exclude setup.py

    diff-quality --violations=pycodestyle --include project/foo/**

The following is executed for every changed file:

#. check if any include pattern was specified
#. if yes, check if the changed file is part of at least one include pattern
#. check if the file is part of any exclude pattern

Ignore/Include based on file status in git
------------------------------------------
Both ``diff-cover`` and ``diff-quality`` allow users to ignore and include files based on the git
status: staged, unstaged, untracked:

* ``--ignore-staged``: ignore all staged files (by default include them)
* ``--ignore-unstaged``: ignore all unstaged files (by default include them)
* ``--include-untracked``: include all untracked files (by default ignore them)

Quiet mode
----------
Both ``diff-cover`` and ``diff-quality`` support a quiet mode which is disable by default.
It can be enabled by using the ``-q``/``--quiet`` flag:

.. code:: bash

    diff-cover coverage.xml -q
    diff-quality --violations=pycodestyle -q

If enabled, the tool will only print errors and failures but no information or warning messages.

Compatibility with multi-line statements
----------------------------------------
``diff-cover`` relies on the comparison of diff reports and coverage reports, and does not report
lines that appear in one and not in the other. While diff reports list all lines that changed,
coverage reports usually list code statements. As a result, a change in a multi-line statement may not be analyzed by ``diff-cover``.

As a workaround, you can use the argument ``--expand-coverage-report``: lines not appearing in the coverage reports will be added to them with the same number of hits as the previously reported line. ``diff-cover`` will then perfor