flake8/docs/source/internal/option_handling.rst

Option and Configuration Handling
=================================

Option Management
-----------------

Command-line options are often also set in configuration files for |Flake8|.
While not all options are meant to be parsed from configuration files, many
default options are also parsed from configuration files as well as
most plugin options.

In |Flake8| 2, plugins received a :class:`optparse.OptionParser` instance and
called :meth:`optparse.OptionParser.add_option` to register options. If the
plugin author also wanted to have that option parsed from config files they
also had to do something like:

.. code-block:: python

    parser.config_options.append('my_config_option')
    parser.config_options.extend(['config_opt1', 'config_opt2'])

This was previously undocumented and led to a lot of confusion about why
registered options were not automatically parsed from configuration files.

Since |Flake8| 3 was rewritten from scratch, we decided to take a different
approach to configuration file parsing. Instead of needing to know about an
undocumented attribute that pep8 looks for, |Flake8| 3 now accepts a parameter
to ``add_option``, specifically ``parse_from_config`` which is a boolean
value.

|Flake8| does this by creating its own abstractions on top of :mod:`optparse`.
The first abstraction is the :class:`flake8.options.manager.Option` class. The
second is the :class:`flake8.options.manager.OptionManager`. In fact, we add
three new parameters:

- ``parse_from_config``

- ``comma_separated_list``

- ``normalize_paths``

The last two are not specifically for configuration file handling, but they
do improve that dramatically. We found that there were options that, when
specified in a configuration file, often necessitated being spit
multiple lines and those options were almost always comma-separated. For
example, let's consider a user's list of ignored error codes for a project:

.. code-block:: ini

    [flake8]
    ignore =
        # Reasoning
        E111,
        # Reasoning
        E711,
        # Reasoning
        E712,
        # Reasoning
        E121,
        # Reasoning
        E122,
        # Reasoning
        E123,
        # Reasoning
        E131,
        # Reasoning
        E251

It makes sense here to allow users to specify the value this way, but, the
standard libary's :class:`configparser.RawConfigParser` class does returns a
string that looks like

.. code-block:: python

    "\nE111,  \nE711,  \nE712,  \nE121,  \nE122,  \nE123,  \nE131,  \nE251  "

This means that a typical call to :meth:`str.split` with ``','`` will not be
sufficient here. Telling |Flake8| that something is a comma-separated list
(e.g., ``comma_separated_list=True``) will handle this for you. |Flake8| will
return:

.. code-block:: python

    ["E111", "E711", "E712", "E121", "E122", "E123", "E131", "E251"]

Next let's look at how users might like to specify their ``exclude`` list.
Presently OpenStack's Nova project has this line in their `tox.ini`_:

.. code-block:: ini

    exclude = .venv,.git,.tox,dist,doc,*openstack/common/*,*lib/python*,*egg,build,tools/xenserver*,releasenotes

We think we can all agree that this would be easier to read like this:

.. code-block:: ini

    exclude =
        .venv,
        .git,
        .tox,
        dist,
        doc,
        *openstack/common/*,
        *lib/python*,
        *egg,
        build,
        tools/xenserver*,
        releasenotes

In this case, since these are actually intended to be paths, we would specify
both ``comma_separated_list=True`` and ``normalize_paths=True`` because we
want the paths to be provided to us with some consistency (either all absolute
paths or not).

Now let's look at how this will actually be used. Most plugin developers
will receive an instance of :class:`~flake8.options.manager.OptionManager` so
to ease the transition we kept the same API as the
:class:`optparse.OptionParser` object. The only difference is that
:meth:`~flake8.options.manager.OptionManager.add_option` accepts the three
extra arguments we highlighted above.

.. _tox.ini:
    https://github.com/openstack/nova/blob/3eb190c4cfc0eefddac6c2cc1b94a699fb1687f8/tox.ini#L155

Configuration File Management
-----------------------------

In |Flake8| 2, configuration file discovery and management was handled by
pep8.  In pep8's 1.6 release series, it drastically broke how discovery and
merging worked (as a result of trying to improve it). To avoid a dependency
breaking |Flake8| again in the future, we have created our own discovery and
management.
As part of managing this ourselves, we decided to change management/discovery
for 3.0.0. We have done the following:

- User files (files stored in a user's home directory or in the XDG directory
  inside their home directory) are the first files read. For example, if the
  user has a ``~/.flake8`` file, we will read that first.

- Project files (files stored in the current directory) are read next and
  merged on top of the user file. In other words, configuration in project
  files takes precedence over configuration in user files.

- **New in 3.0.0** The user can specify ``--append-config <path-to-file>``
  repeatedly to include extra configuration files that should be read and
  take precedence over user and project files.

- **New in 3.0.0** The user can specify ``--config <path-to-file>`` to so this
  file is the only configuration file used. This is a change from |Flake8| 2
  where pep8 would simply merge this configuration file into the configuration
  generated by user and project files (where this takes precedence).

- **New in 3.0.0** The user can specify ``--isolated`` to disable
  configuration via discovered configuration files.

To facilitate the configuration file management, we've taken a different
approach to discovery and management of files than pep8. In pep8 1.5, 1.6, and
1.7 configuration discovery and management was centralized in `66 lines of
very terse python`_ which was confusing and not very explicit. The terseness
of this function (|Flake8|'s authors believe) caused the confusion and
problems with pep8's 1.6 series. As such, |Flake8| has separated out
discovery, management, and merging into a module to make reasoning about each
of these pieces easier and more explicit (as well as easier to test).

Configuration file discovery is managed by the
:class:`~flake8.options.config.ConfigFileFinder` object. This object needs to
know information about the program's name, any extra arguments passed to it,
and any configuration files that should be appended to the list of discovered
files. It provides methods for finding the files and similiar methods for
parsing those fles. For example, it provides
:meth:`~flake8.options.config.ConfigFileFinder.local_config_files` to find
known local config files (and append the extra configuration files) and it
also provides :meth:`~flake8.options.config.ConfigFileFinder.local_configs`
to parse those configuration files.

.. note:: ``local_config_files`` also filters out non-existent files.

Configuration file merging and managemnt is controlled by the
:class:`~flake8.options.config.MergedConfigParser`. This requires the instance
of :class:`~flake8.options.manager.OptionManager` that the program is using,
the list of appended config files, and the list of extra arguments. This
object is currently the sole user of the
:class:`~flake8.options.config.ConfigFileFinder` object. It appropriately
initializes the object and uses it in each of

- :meth:`~flake8.options.config.MergedConfigParser.parse_cli_config`
- :meth:`~flake8.options.config.MergedConfigParser.parse_local_config`
- :meth:`~flake8.options.config.MergedConfigParser.parse_user_config`

Finally,
:meth:`~flake8.options.config.MergedConfigParser.merge_user_and_local_config`
takes the user and local configuration files that are parsed by
:meth:`~flake8.options.config.MergedConfigParser.parse_local_config` and
:meth:`~flake8.options.config.MergedConfigParser.parse_user_config`. The
main usage of the ``MergedConfigParser`` is in
:func:`~flake8.options.aggregator.aggregate_options`.

Aggregating Configuration File and Command Line Arguments
---------------------------------------------------------

:func:`~flake8.options.aggregator.aggregate_options` accepts an instance of
:class:`~flake8.options.maanger.OptionManager` and does the work to parse the
command-line arguments passed by the user necessary for creating an instance
of :class:`~flake8.options.config.MergedConfigParser`.

After parsing the configuration file, we determine the default ignore list. We
use the defaults from the OptionManager and update those with the parsed
configuration files. Finally we parse the user-provided options one last time
using the option defaults and configuration file values as defaults. The
parser merges on the command-line specified arguments for us so we have our
final, definitive, aggregated options.

.. _66 lines of very terse python:
    https://github.com/PyCQA/pep8/blob/b8088a2b6bc5b76bece174efad877f764529bc74/pep8.py#L1981..L2047

API Documentation
-----------------

.. autofunction:: flake8.options.aggregator.aggregate_options

.. autoclass:: flake8.options.manager.Option
    :members: __init__, normalize, to_optparse

.. autoclass:: flake8.options.manager.OptionManager
    :members:
    :special-members:

.. autoclass:: flake8.options.config.ConfigFileFinder
    :members:
    :special-members:

.. autoclass:: flake8.options.config.MergedConfigParser
    :members:
    :special-members: