From 2d72fc23c804be2697446509e55234f8d96c00fe Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 6 Dec 2015 12:23:45 -0600 Subject: [PATCH 001/364] Add skeleton of DESIGN documentation --- DESIGN.rst | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 DESIGN.rst diff --git a/DESIGN.rst b/DESIGN.rst new file mode 100644 index 0000000..6f40e51 --- /dev/null +++ b/DESIGN.rst @@ -0,0 +1,77 @@ +============== + Design Goals +============== + +Outline +------- + +#. :ref:`plugins` + + #. :ref:`checking` + + #. :ref:`autofixing` + + #. :ref:`options-passing` + +#. :ref:`options` + + #. :ref:`better-select-ignore` + +#. :ref:`standard-in` + +#. :ref:`multiprocessing` + +#. :ref:`formatting` + +#. :ref:`report-generation` + +.. _plugins: + +Better Plugins Support +---------------------- + +.. _checking: + +Support for Plugins that Only Run Checks +++++++++++++++++++++++++++++++++++++++++ + +.. _autofixing: + +Support for Plugins that Autofix Errors ++++++++++++++++++++++++++++++++++++++++ + +.. _options-passing: + +Support for Plugins Require Parsed Options +++++++++++++++++++++++++++++++++++++++++++ + +.. _options: + +Better Options Support +---------------------- + +.. _better-select-ignore: + +Support for Better Select/Ignore Handling ++++++++++++++++++++++++++++++++++++++++++ + +.. _standard-in: + +Better stdin support +-------------------- + +.. _multiprocessing: + +Multiprocessing Support +----------------------- + +.. _formatting: + +Better (Flake8-centralized) Formatting Support +---------------------------------------------- + +.. _report-generation: + + +Better (Flake8-centralized) Report Generation Support +----------------------------------------------------- From eb29f117386641fdc0bca2c3833ec14fea0a5751 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 6 Dec 2015 13:36:01 -0600 Subject: [PATCH 002/364] Flesh out some portions of the design goals --- DESIGN.rst | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/DESIGN.rst b/DESIGN.rst index 6f40e51..dee98ee 100644 --- a/DESIGN.rst +++ b/DESIGN.rst @@ -11,8 +11,12 @@ Outline #. :ref:`autofixing` + #. :ref:`reporter-plugins` + #. :ref:`options-passing` + #. :ref:`plugin-default-ignore` + #. :ref:`options` #. :ref:`better-select-ignore` @@ -30,31 +34,106 @@ Outline Better Plugins Support ---------------------- +Currently, Flake8 has some rather excellent support for plugins. It currently +allows for the following: + +- Third-party packages to register checks + +- Checks to be disabled by default + +- Checks to accept an AST compiled tree, physical lines, or logical lines. + +- Flake8 handles running those checks in separate subprocesses as necessary + +That said, plugins cannot access the options passed on the command-line, or +options parsed from config files (without parsing them, themselves) and all +reporting is handled by pep8 instead of flake8 which reduces the flexibility +users have in aggregating reports. + .. _checking: Support for Plugins that Only Run Checks ++++++++++++++++++++++++++++++++++++++++ +Flake8 currently already supports plugins that only run checks. This support +needs to continue and should be trivial to continue. + .. _autofixing: Support for Plugins that Autofix Errors +++++++++++++++++++++++++++++++++++++++ +Flake8 should enable people writing plugins for both core Flake8 checkers and +third-party checkers that allow the code to be automatically fixed. The trick +is in how to do this. + +Once Flake8 has control over running plugins and treats pep8, flake8, and +mccabe as "plugins", it will aggregate the errors returned by all of the +plugins and be able to "notify" other plugins that have chosen to listen for +errors so those plugins can auto-fix the problems in the file. + +See https://gitlab.com/pycqa/flake8/issues/84 + +.. _reporter-plugins: + +Support for Plugins that Format Output +++++++++++++++++++++++++++++++++++++++ + +Flake8 currently supports formatting output via pep8's ``--format`` option. +This works but is fundamentally a bit limiting. Allowing users to replace or +compose formatters would allow for certain formatters to highlight more +important information over less important information as the user deems +necessary. + +See https://gitlab.com/pycqa/flake8/issues/66 + .. _options-passing: Support for Plugins Require Parsed Options ++++++++++++++++++++++++++++++++++++++++++ +Plugins currently are able to use ``add_options`` and ``parse_options`` +classmethods to register and retrieve options information. This is admittedly +a little awkward and could be improved, but should at least be preserved in +this rewrite. + +See potential improvements as a result of +https://gitlab.com/pycqa/flake8/issues/88 + +.. _plugin-default-ignore: + +Support for Plugins Specifying Default Ignore list +++++++++++++++++++++++++++++++++++++++++++++++++++ + +Plugins currently have no way of extending the default ignore list. This means +they have to hard-code checks to auto-ignore errors. + .. _options: Better Options Support ---------------------- +Currently there are some options handled by pep8 that are handled poorly. +Further, the way the options work is confusing to some, e.g., when specifying +``--ignore``, users do not expect it to override the ``DEFAULT_IGNORE`` list. +Users also don't expect ``--ignore`` and ``--select`` to step on each other's +toes. + .. _better-select-ignore: Support for Better Select/Ignore Handling +++++++++++++++++++++++++++++++++++++++++ +Currently ``--select`` and ``--ignore`` cause one or the other to be ignored. +Users presently cannot specify both for granularity. This should be +significantly improved. + +Further, new tools have developed ``--add-select`` and ``--add-ignore`` which +allows an add-only interface. This seems to be a good direction to follow. +Flake8 should support this. + +See https://github.com/PyCQA/pep8/issues/390 + .. _standard-in: Better stdin support From 00dfe8698bf4ecdb0f9814eae782b15f0b16694a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 6 Dec 2015 15:11:23 -0600 Subject: [PATCH 003/364] Wrap up design goals --- DESIGN.rst | 37 +++++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/DESIGN.rst b/DESIGN.rst index dee98ee..c5b5e82 100644 --- a/DESIGN.rst +++ b/DESIGN.rst @@ -17,6 +17,8 @@ Outline #. :ref:`plugin-default-ignore` + #. :ref:`report-generation` + #. :ref:`options` #. :ref:`better-select-ignore` @@ -25,10 +27,6 @@ Outline #. :ref:`multiprocessing` -#. :ref:`formatting` - -#. :ref:`report-generation` - .. _plugins: Better Plugins Support @@ -87,6 +85,14 @@ necessary. See https://gitlab.com/pycqa/flake8/issues/66 +.. _report-generation: + +Support for Report Generation ++++++++++++++++++++++++++++++ + +Flake8 should support pluggable report formats. See also pluggable report +formats for https://github.com/openstack/bandit + .. _options-passing: Support for Plugins Require Parsed Options @@ -139,18 +145,25 @@ See https://github.com/PyCQA/pep8/issues/390 Better stdin support -------------------- +Currently, flake8 accepts input from standard-in to check. It also currently +monkey-patches pep8 to cache that value. It would be better if there was one +way to retrieve the stdin input for plugins. Flake8 should provide this +directly instead of pep8 providing it. + +See +https://gitlab.com/pycqa/flake8/commit/41393c9b6de513ea169b61c175b71018e8a12336 + .. _multiprocessing: Multiprocessing Support ----------------------- -.. _formatting: +Flake8's existing multiprocessing support (and handling for different error +cases needs to persist through this redesign). -Better (Flake8-centralized) Formatting Support ----------------------------------------------- +See: -.. _report-generation: - - -Better (Flake8-centralized) Report Generation Support ------------------------------------------------------ +- https://gitlab.com/pycqa/flake8/issues/8 +- https://gitlab.com/pycqa/flake8/issues/17 +- https://gitlab.com/pycqa/flake8/issues/44 +- https://gitlab.com/pycqa/flake8/issues/74 From ae1ad0e2bf15a3043326c008a7a7b048050dec49 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 8 Dec 2015 15:41:49 -0600 Subject: [PATCH 004/364] Add design note that has been roaming my brain --- DESIGN.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/DESIGN.rst b/DESIGN.rst index c5b5e82..7cdada3 100644 --- a/DESIGN.rst +++ b/DESIGN.rst @@ -72,6 +72,8 @@ errors so those plugins can auto-fix the problems in the file. See https://gitlab.com/pycqa/flake8/issues/84 +.. note:: Will probably need a Trie implementation for this + .. _reporter-plugins: Support for Plugins that Format Output From f013698072ccba9c2c188184840ce6bb13f9723f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 18 Dec 2015 13:49:59 -0600 Subject: [PATCH 005/364] Update with extra info about plugins --- DESIGN.rst | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/DESIGN.rst b/DESIGN.rst index 7cdada3..573e093 100644 --- a/DESIGN.rst +++ b/DESIGN.rst @@ -70,6 +70,9 @@ mccabe as "plugins", it will aggregate the errors returned by all of the plugins and be able to "notify" other plugins that have chosen to listen for errors so those plugins can auto-fix the problems in the file. +We should also be considerate of allowing these plugins to be composable. Each +plugin should have a way of defining its capabilities. + See https://gitlab.com/pycqa/flake8/issues/84 .. note:: Will probably need a Trie implementation for this @@ -95,6 +98,10 @@ Support for Report Generation Flake8 should support pluggable report formats. See also pluggable report formats for https://github.com/openstack/bandit +Report generation plugins may also choose to implement a way to store previous +runs of flake8. As such these plugins should be designed to be composable as +well. + .. _options-passing: Support for Plugins Require Parsed Options From 7b2a1c157b947150c19ba559d74b0904a5daa8f8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 28 Dec 2015 23:32:42 -0600 Subject: [PATCH 006/364] Bare bones of a notification system --- .gitignore | 5 +++ example.py | 10 ++++++ flake8/__init__.py | 1 + flake8/_trie.py | 74 ++++++++++++++++++++++++++++++++++++++++++ flake8/notifier.py | 22 +++++++++++++ flake8/style_guide.py | 0 setup.py | 75 +++++++++++++++++++++++++++++++++++++++++++ tests/test_trie.py | 14 ++++++++ tox.ini | 8 +++++ 9 files changed, 209 insertions(+) create mode 100644 .gitignore create mode 100644 example.py create mode 100644 flake8/__init__.py create mode 100644 flake8/_trie.py create mode 100644 flake8/notifier.py create mode 100644 flake8/style_guide.py create mode 100644 setup.py create mode 100644 tests/test_trie.py create mode 100644 tox.ini diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..a41da07 --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +.cache/* +.tox/* +*.pyc +*.sw* +*.egg-info diff --git a/example.py b/example.py new file mode 100644 index 0000000..a201782 --- /dev/null +++ b/example.py @@ -0,0 +1,10 @@ +from flake8 import _trie as trie + + +tree = trie.Trie() +for i in range(5): + tree.add('E103', 'E103-listener-{0}'.format(i)) + j = i + 1 + tree.add('E1{0}3'.format(j), 'E1{0}3-listener'.format(j)) +for i in range(10): + tree.add('W1{0:02d}'.format(i), 'W1{0:02d}-listener'.format(i)) diff --git a/flake8/__init__.py b/flake8/__init__.py new file mode 100644 index 0000000..6e648d3 --- /dev/null +++ b/flake8/__init__.py @@ -0,0 +1 @@ +__version__ = '3.0.0a1' diff --git a/flake8/_trie.py b/flake8/_trie.py new file mode 100644 index 0000000..ddd3c19 --- /dev/null +++ b/flake8/_trie.py @@ -0,0 +1,74 @@ +"""Independent implementation of a Trie tree.""" + +__all__ = ('Trie', 'TrieNode') + + +def _iterate_stringlike_objects(string): + for i in range(len(string)): + yield string[i:i+1] + + +class Trie(object): + """The object that manages the trie nodes.""" + + def __init__(self): + """Initialize an empty trie.""" + self.root = TrieNode(None, None) + + def add(self, path, node_data): + """Add the node data to the path described.""" + node = self.root + for prefix in _iterate_stringlike_objects(path): + child = node.find_prefix(prefix) + if child is None: + child = node.add_child(prefix, []) + node = child + node.data.append(node_data) + + def find(self, path): + """Find a node based on the path provided.""" + node = self.root + for prefix in _iterate_stringlike_objects(path): + child = node.find_prefix(prefix) + if child is None: + return None + node = child + return node + + +class TrieNode(object): + """The majority of the implementation details of a Trie.""" + + def __init__(self, prefix, data, children=None): + """Initialize a TrieNode with data and children.""" + self.children = children or {} + self.data = data + self.prefix = prefix + + def __repr__(self): + """Generate an easy to read representation of the node.""" + return 'TrieNode(prefix={0}, data={1}, children={2})'.format( + self.prefix, self.data, dict(self.children) + ) + + def find_prefix(self, prefix): + """Find the prefix in the children of this node. + + :returns: A child matching the prefix or None. + :rtype: :class:`~TrieNode` or None + """ + return self.children.get(prefix, None) + + def add_child(self, prefix, data, children=None): + """Create and add a new child node. + + :returns: The newly created node + :rtype: :class:`~TrieNode` + """ + new_node = TrieNode(prefix, data, children) + self.children[prefix] = new_node + return new_node + + def traverse(self): + """Traverse children of this node in order.""" + raise NotImplementedError() diff --git a/flake8/notifier.py b/flake8/notifier.py new file mode 100644 index 0000000..7b24b2f --- /dev/null +++ b/flake8/notifier.py @@ -0,0 +1,22 @@ +from flake8 import _trie + + +class Notifier(object): + def __init__(self): + self.listeners = _trie.Trie() + + def listeners_for(self, error_code): + node = self.listeners.find(error_code) + for listener in node.data: + yield listener + if node.children: + for child in node.traverse(): + for listener in child.data: + yield listener + + def notify(self, error_code, *args, **kwargs): + for listener in self.listeners_for(error_code): + listener.notify(*args, **kwargs) + + def register_listener(self, error_code, listener): + self.listeners.add(error_code, listener) diff --git a/flake8/style_guide.py b/flake8/style_guide.py new file mode 100644 index 0000000..e69de29 diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..fb0585e --- /dev/null +++ b/setup.py @@ -0,0 +1,75 @@ +# -*- coding: utf-8 -*- +from __future__ import with_statement +from setuptools import setup +try: + # Work around a traceback with Nose on Python 2.6 + # http://bugs.python.org/issue15881#msg170215 + __import__('multiprocessing') +except ImportError: + pass + +try: + # Use https://docs.python.org/3/library/unittest.mock.html + from unittest import mock +except ImportError: + # < Python 3.3 + mock = None + + +tests_require = ['nose'] +if mock is None: + tests_require += ['mock'] + + +def get_version(fname='flake8/__init__.py'): + with open(fname) as f: + for line in f: + if line.startswith('__version__'): + return eval(line.split('=')[-1]) + + +def get_long_description(): + descr = [] + for fname in ('README.rst', 'CHANGES.rst'): + with open(fname) as f: + descr.append(f.read()) + return '\n\n'.join(descr) + + +setup( + name="flake8", + license="MIT", + version=get_version(), + description="the modular source code checker: pep8, pyflakes and co", + # long_description=get_long_description(), + author="Tarek Ziade", + author_email="tarek@ziade.org", + maintainer="Ian Cordasco", + maintainer_email="graffatcolmingov@gmail.com", + url="https://gitlab.com/pycqa/flake8", + packages=["flake8"], + install_requires=[ + "pyflakes >= 0.8.1, < 1.1", + "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", + "mccabe >= 0.2.1, < 0.4", + ], + entry_points={ + 'distutils.commands': ['flake8 = flake8.main:Flake8Command'], + 'console_scripts': ['flake8 = flake8.main:main'], + 'flake8.extension': [ + 'F = flake8._pyflakes:FlakesChecker', + ], + }, + classifiers=[ + "Environment :: Console", + "Intended Audience :: Developers", + "License :: OSI Approved :: MIT License", + "Programming Language :: Python", + "Programming Language :: Python :: 2", + "Programming Language :: Python :: 3", + "Topic :: Software Development :: Libraries :: Python Modules", + "Topic :: Software Development :: Quality Assurance", + ], + tests_require=tests_require, + test_suite='nose.collector', +) diff --git a/tests/test_trie.py b/tests/test_trie.py new file mode 100644 index 0000000..10a10aa --- /dev/null +++ b/tests/test_trie.py @@ -0,0 +1,14 @@ +from flake8 import _trie as trie + + +def test_build_tree(): + tree = trie.Trie() + for i in range(5): + tree.add('E103', 'E103-listener-{0}'.format(i)) + j = i + 1 + tree.add('E1{0}3'.format(j), 'E1{0}3-listener'.format(j)) + for i in range(10): + tree.add('W1{0:02d}'.format(i), 'W1{0:02d}-listener'.format(i)) + + assert tree.find('E103') is not None + assert tree.find('E200') is None diff --git a/tox.ini b/tox.ini new file mode 100644 index 0000000..de47800 --- /dev/null +++ b/tox.ini @@ -0,0 +1,8 @@ +[tox] +envlist = py26,py27,py32,py33,py34,py35 + +[testenv] +deps = + pytest +commands = + py.test From 75f15340ac3b57bacab06a0dd47177c4eaa02d9b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 29 Dec 2015 08:00:37 -0600 Subject: [PATCH 007/364] Remove toy file from git --- example.py | 10 ---------- 1 file changed, 10 deletions(-) delete mode 100644 example.py diff --git a/example.py b/example.py deleted file mode 100644 index a201782..0000000 --- a/example.py +++ /dev/null @@ -1,10 +0,0 @@ -from flake8 import _trie as trie - - -tree = trie.Trie() -for i in range(5): - tree.add('E103', 'E103-listener-{0}'.format(i)) - j = i + 1 - tree.add('E1{0}3'.format(j), 'E1{0}3-listener'.format(j)) -for i in range(10): - tree.add('W1{0:02d}'.format(i), 'W1{0:02d}-listener'.format(i)) From 57e8583e2958c888d1d3c1df4bfb871e7956e308 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 29 Dec 2015 08:07:47 -0600 Subject: [PATCH 008/364] Add tree/sub-tree traversal to Trie Accurately retrieve listeners for a specific error code --- flake8/_trie.py | 31 +++++++++++++++++++++++++++---- flake8/notifier.py | 9 +++++---- 2 files changed, 32 insertions(+), 8 deletions(-) diff --git a/flake8/_trie.py b/flake8/_trie.py index ddd3c19..858c067 100644 --- a/flake8/_trie.py +++ b/flake8/_trie.py @@ -35,6 +35,16 @@ class Trie(object): node = child return node + def traverse(self): + """Traverse this tree. + + This performs a depth-first pre-order traversal of children in this + tree. It returns the results consistently by first sorting the + children based on their prefix and then traversing them in + alphabetical order. + """ + return self.root.traverse() + class TrieNode(object): """The majority of the implementation details of a Trie.""" @@ -47,8 +57,8 @@ class TrieNode(object): def __repr__(self): """Generate an easy to read representation of the node.""" - return 'TrieNode(prefix={0}, data={1}, children={2})'.format( - self.prefix, self.data, dict(self.children) + return 'TrieNode(prefix={0}, data={1})'.format( + self.prefix, self.data ) def find_prefix(self, prefix): @@ -70,5 +80,18 @@ class TrieNode(object): return new_node def traverse(self): - """Traverse children of this node in order.""" - raise NotImplementedError() + """Traverse children of this node. + + This performs a depth-first pre-order traversal of the remaining + children in this sub-tree. It returns the results consistently by + first sorting the children based on their prefix and then traversing + them in alphabetical order. + """ + if not self.children: + return + + for prefix in sorted(self.children.keys()): + child = self.children[prefix] + yield child + for child in child.traverse(): + yield child diff --git a/flake8/notifier.py b/flake8/notifier.py index 7b24b2f..625acd7 100644 --- a/flake8/notifier.py +++ b/flake8/notifier.py @@ -7,12 +7,13 @@ class Notifier(object): def listeners_for(self, error_code): node = self.listeners.find(error_code) + if node is None: + return for listener in node.data: yield listener - if node.children: - for child in node.traverse(): - for listener in child.data: - yield listener + for child in node.traverse(): + for listener in child.data: + yield listener def notify(self, error_code, *args, **kwargs): for listener in self.listeners_for(error_code): From d3a9f7d58c47a1fd22cb415d987ed1212db8b66f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 29 Dec 2015 08:12:54 -0600 Subject: [PATCH 009/364] Add docstrings to Notifier class --- flake8/notifier.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/flake8/notifier.py b/flake8/notifier.py index 625acd7..d366fbf 100644 --- a/flake8/notifier.py +++ b/flake8/notifier.py @@ -3,9 +3,31 @@ from flake8 import _trie class Notifier(object): def __init__(self): + """Initialize an empty notifier object.""" self.listeners = _trie.Trie() def listeners_for(self, error_code): + """Retrieve listeners for an error_code. + + The error code does not need to be a specific error code. For example, + There may be listeners registered for E100, E101, E110, E112, and + E126. If you wanted to get all listeners starting with 'E1' then you + would pass 'E1' as the error code here. + + Example usage + + .. code-block:: python + + from flake8 import notifier + + n = notifier.Notifier() + # register listeners + for listener in n.listeners_for('E1'): + listener.notify(...) + + for listener in n.listeners_for('W102'): + listener.notify(...) + """ node = self.listeners.find(error_code) if node is None: return @@ -16,8 +38,10 @@ class Notifier(object): yield listener def notify(self, error_code, *args, **kwargs): + """Notify all listeners for the specified error code.""" for listener in self.listeners_for(error_code): listener.notify(*args, **kwargs) def register_listener(self, error_code, listener): + """Register a listener for a specific error_code.""" self.listeners.add(error_code, listener) From d1d1d6003249c6f857ca34941728c65ba7d63359 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 29 Dec 2015 22:49:05 -0600 Subject: [PATCH 010/364] Add tests for the _trie module --- tests/test_trie.py | 127 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 117 insertions(+), 10 deletions(-) diff --git a/tests/test_trie.py b/tests/test_trie.py index 10a10aa..0a5a0a9 100644 --- a/tests/test_trie.py +++ b/tests/test_trie.py @@ -1,14 +1,121 @@ from flake8 import _trie as trie -def test_build_tree(): - tree = trie.Trie() - for i in range(5): - tree.add('E103', 'E103-listener-{0}'.format(i)) - j = i + 1 - tree.add('E1{0}3'.format(j), 'E1{0}3-listener'.format(j)) - for i in range(10): - tree.add('W1{0:02d}'.format(i), 'W1{0:02d}-listener'.format(i)) +class TestTrie(object): + """Collection of tests for the Trie class.""" - assert tree.find('E103') is not None - assert tree.find('E200') is None + def test_traverse_without_data(self): + """Verify the behaviour when traversing an empty Trie.""" + tree = trie.Trie() + assert list(tree.traverse()) == [] + + def test_traverse_with_data(self): + """Verify that traversal of our Trie is depth-first and pre-order.""" + tree = trie.Trie() + tree.add('A', 'A') + tree.add('a', 'a') + tree.add('AB', 'B') + tree.add('Ab', 'b') + tree.add('AbC', 'C') + tree.add('Abc', 'c') + # The trie tree here should look something like + # + # + # / \ + # A a + # / | + # B b + # / \ + # C c + # + # And the traversal should look like: + # + # A B b C c a + expected_order = ['A', 'B', 'b', 'C', 'c', 'a'] + for expected, actual_node in zip(expected_order, tree.traverse()): + assert actual_node.prefix == expected + + def test_find(self): + """Exercise the Trie.find method.""" + tree = trie.Trie() + tree.add('A', 'A') + tree.add('a', 'a') + tree.add('AB', 'AB') + tree.add('Ab', 'Ab') + tree.add('AbC', 'AbC') + tree.add('Abc', 'Abc') + + assert tree.find('AB').data == ['AB'] + assert tree.find('AbC').data == ['AbC'] + assert tree.find('A').data == ['A'] + assert tree.find('X') is None + + +class TestTrieNode(object): + """Collection of tests for the TrieNode class.""" + + def test_add_child(self): + """Verify we add children appropriately.""" + node = trie.TrieNode('E', 'E is for Eat') + assert node.find_prefix('a') is None + added = node.add_child('a', 'a is for Apple') + assert node.find_prefix('a') is added + + def test_add_child_overrides_previous_child(self): + """Verify adding a child will replace the previous child.""" + node = trie.TrieNode('E', 'E is for Eat', children={ + 'a': trie.TrieNode('a', 'a is for Apple') + }) + previous = node.find_prefix('a') + assert previous is not None + + added = node.add_child('a', 'a is for Ascertain') + assert node.find_prefix('a') is added + + def test_find_prefix(self): + """Verify we can find a child with the specified prefix.""" + node = trie.TrieNode('E', 'E is for Eat', children={ + 'a': trie.TrieNode('a', 'a is for Apple') + }) + child = node.find_prefix('a') + assert child is not None + assert child.prefix == 'a' + assert child.data == 'a is for Apple' + + def test_find_prefix_returns_None_when_no_children_have_the_prefix(self): + """Verify we receive None from find_prefix for missing children.""" + node = trie.TrieNode('E', 'E is for Eat', children={ + 'a': trie.TrieNode('a', 'a is for Apple') + }) + assert node.find_prefix('b') is None + + def test_traverse_does_nothing_when_a_node_has_no_children(self): + """Verify traversing a node with no children does nothing.""" + node = trie.TrieNode('E', 'E is for Eat') + assert list(node.traverse()) == [] + + def test_traverse(self): + """Verify traversal is depth-first and pre-order.""" + root = trie.TrieNode(None, None) + node = root.add_child('A', 'A') + root.add_child('a', 'a') + node.add_child('B', 'B') + node = node.add_child('b', 'b') + node.add_child('C', 'C') + node.add_child('c', 'c') + # The sub-tree here should look something like + # + # + # / \ + # A a + # / | + # B b + # / \ + # C c + # + # And the traversal should look like: + # + # A B b C c a + expected_order = ['A', 'B', 'b', 'C', 'c', 'a'] + for expected, actual_node in zip(expected_order, root.traverse()): + assert actual_node.prefix == expected From 222be9ac4918d2ea85e9ac3d20832106c10d30a1 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 29 Dec 2015 23:04:36 -0600 Subject: [PATCH 011/364] Start adding tests for Notifier class --- flake8/notifier.py | 2 +- tests/test_notifier.py | 31 +++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) create mode 100644 tests/test_notifier.py diff --git a/flake8/notifier.py b/flake8/notifier.py index d366fbf..5a75f87 100644 --- a/flake8/notifier.py +++ b/flake8/notifier.py @@ -40,7 +40,7 @@ class Notifier(object): def notify(self, error_code, *args, **kwargs): """Notify all listeners for the specified error code.""" for listener in self.listeners_for(error_code): - listener.notify(*args, **kwargs) + listener.notify(error_code, *args, **kwargs) def register_listener(self, error_code, listener): """Register a listener for a specific error_code.""" diff --git a/tests/test_notifier.py b/tests/test_notifier.py new file mode 100644 index 0000000..44f2dc9 --- /dev/null +++ b/tests/test_notifier.py @@ -0,0 +1,31 @@ +import pytest + +from flake8 import notifier + +class _Listener(object): + def __init__(self, error_code): + self.error_code = error_code + self.was_notified = False + + def notify(self, error_code, *args, **kwargs): + assert self.error_code == error_code + self.was_notified = True + + +class TestNotifier(object): + @pytest.fixture(autouse=True) + def setup(self): + self.notifier = notifier.Notifier() + self.listener_map = {} + + for i in range(10): + for j in range(30): + error_code = 'E{0}{1:02d}'.format(i, j) + listener = _Listener(error_code) + self.listener_map[error_code] = listener + self.notifier.register_listener(error_code, listener) + + def test_notify_a_single_error_code(self): + """Show that we notify a specific error code.""" + self.notifier.notify('E111', 'extra', 'args') + assert self.listener_map['E111'].was_notified is True From 37b92cd4b47723115f9e27760307b10508808bc6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 29 Dec 2015 23:28:20 -0600 Subject: [PATCH 012/364] Fix logic for Notifier.listeners_for Add tests for proper logic around notifier --- DESIGN.rst | 12 ++++++++++++ flake8/notifier.py | 25 ++++++++++--------------- tests/test_notifier.py | 20 ++++++++++++-------- 3 files changed, 34 insertions(+), 23 deletions(-) diff --git a/DESIGN.rst b/DESIGN.rst index 573e093..e472671 100644 --- a/DESIGN.rst +++ b/DESIGN.rst @@ -77,6 +77,18 @@ See https://gitlab.com/pycqa/flake8/issues/84 .. note:: Will probably need a Trie implementation for this +What we *might* want is for a autofix plugin to register something like + +:: + + 'flake8.autofix_extension': [ + 'E1 = my_fixer.E1Listener', + 'E2 = my_fixer.E2Listener', + ] + +This means that the notifer would need to take an error code like ``E111`` and +then notify anything listening for ``E111``, ``E11``, ``E1``, and ``E``. + .. _reporter-plugins: Support for Plugins that Format Output diff --git a/flake8/notifier.py b/flake8/notifier.py index 5a75f87..6bf51ef 100644 --- a/flake8/notifier.py +++ b/flake8/notifier.py @@ -9,12 +9,12 @@ class Notifier(object): def listeners_for(self, error_code): """Retrieve listeners for an error_code. - The error code does not need to be a specific error code. For example, - There may be listeners registered for E100, E101, E110, E112, and - E126. If you wanted to get all listeners starting with 'E1' then you - would pass 'E1' as the error code here. + There may be listeners registered for E1, E100, E101, E110, E112, and + E126. To get all the listeners for one of E100, E101, E110, E112, or + E126 you would also need to incorporate the listeners for E1 (since + they're all in the same class). - Example usage + Example usage: .. code-block:: python @@ -22,20 +22,15 @@ class Notifier(object): n = notifier.Notifier() # register listeners - for listener in n.listeners_for('E1'): - listener.notify(...) - for listener in n.listeners_for('W102'): listener.notify(...) """ - node = self.listeners.find(error_code) - if node is None: - return - for listener in node.data: - yield listener - for child in node.traverse(): - for listener in child.data: + path = error_code + while path: + node = self.listeners.find(path) + for listener in node.data: yield listener + path = path[:-1] def notify(self, error_code, *args, **kwargs): """Notify all listeners for the specified error code.""" diff --git a/tests/test_notifier.py b/tests/test_notifier.py index 44f2dc9..b0365f3 100644 --- a/tests/test_notifier.py +++ b/tests/test_notifier.py @@ -8,7 +8,7 @@ class _Listener(object): self.was_notified = False def notify(self, error_code, *args, **kwargs): - assert self.error_code == error_code + assert error_code.startswith(self.error_code) self.was_notified = True @@ -18,14 +18,18 @@ class TestNotifier(object): self.notifier = notifier.Notifier() self.listener_map = {} - for i in range(10): - for j in range(30): - error_code = 'E{0}{1:02d}'.format(i, j) - listener = _Listener(error_code) - self.listener_map[error_code] = listener - self.notifier.register_listener(error_code, listener) + def add_listener(error_code): + listener = _Listener(error_code) + self.listener_map[error_code] = listener + self.notifier.register_listener(error_code, listener) - def test_notify_a_single_error_code(self): + for i in range(10): + add_listener('E{0}'.format(i)) + for j in range(30): + add_listener('E{0}{1:02d}'.format(i, j)) + + def test_notify(self): """Show that we notify a specific error code.""" self.notifier.notify('E111', 'extra', 'args') assert self.listener_map['E111'].was_notified is True + assert self.listener_map['E1'].was_notified is True From f539464368bd06b16a624b785ab97bb055a573aa Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 30 Dec 2015 17:26:28 -0600 Subject: [PATCH 013/364] Start working on our option handling story --- flake8/option_parser.py | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 flake8/option_parser.py diff --git a/flake8/option_parser.py b/flake8/option_parser.py new file mode 100644 index 0000000..9d006ba --- /dev/null +++ b/flake8/option_parser.py @@ -0,0 +1,32 @@ +import optparse + + +class Option(object): + def __init__(self, short_option_name=None, long_option_name=None, + # Options below here are taken from the optparse.Option class + action=None, default=None, type=None, dest=None, + nargs=None, const=None, choices=None, callback=None, + callback_args=None, callback_kwargs=None, help=None, + metavar=None, + # Options below here are specific to Flake8 + parse_from_config=False + ): + self.short_option_name = short_option_name + self.long_option_name = long_option_name + self.option_args = filter(None, (short_option_name, long_option_name)) + self.option_kwargs = { + 'action': action, + 'default': default, + 'type': type, + 'dest': dest, + 'callback': callback, + 'callback_args': callback_args, + 'callback_kwargs': callback_kwargs, + 'help': help, + 'metavar': metavar, + } + for key, value in self.option_kwargs.items(): + setattr(self, key, value) + self.parse_from_config = parse_from_config + self._option = optparse.Option(*self.option_args, + **self.option_kwargs) From a6d790cc10ff5bce7d3d506b52e4d4df0b73009b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 4 Jan 2016 19:34:10 -0600 Subject: [PATCH 014/364] Start adding logging for --verbose --- flake8/__init__.py | 11 +++++++++++ flake8/option_parser.py | 12 ++++++++++++ 2 files changed, 23 insertions(+) diff --git a/flake8/__init__.py b/flake8/__init__.py index 6e648d3..49cbabd 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -1 +1,12 @@ +import logging + +try: + from logging import NullHandler +except ImportError: + class NullHandler(logging.Handler): + def emit(self, record): + pass + +logging.getLogger(__name__).addHandler(NullHandler()) + __version__ = '3.0.0a1' diff --git a/flake8/option_parser.py b/flake8/option_parser.py index 9d006ba..0343158 100644 --- a/flake8/option_parser.py +++ b/flake8/option_parser.py @@ -1,4 +1,16 @@ +import logging import optparse +import os.path +import sys + +if sys.version_info < (3, 0): + import ConfigParser as configparser +else: + import configparser + +from . import utils + +LOG = logging.getLogger(__name__) class Option(object): From 7ec5a2ada3df656bb319f88ebce0c7574567c07c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 4 Jan 2016 19:34:41 -0600 Subject: [PATCH 015/364] Sketch out config parsing and option managing --- flake8/option_parser.py | 208 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 205 insertions(+), 3 deletions(-) diff --git a/flake8/option_parser.py b/flake8/option_parser.py index 0343158..5b369c0 100644 --- a/flake8/option_parser.py +++ b/flake8/option_parser.py @@ -21,8 +21,30 @@ class Option(object): callback_args=None, callback_kwargs=None, help=None, metavar=None, # Options below here are specific to Flake8 - parse_from_config=False + parse_from_config=False, comma_separated_list=False, ): + """Initialize an Option instance wrapping optparse.Option. + + :param str short_option_name: + The short name of the option (e.g., ``-x``). This will be the + first argument passed to :class:`~optparse.Option`. + :param str long_option_name: + The long name of the option (e.g., ``--xtra-long-option``). This + will be the second argument passed to :class:`~optparse.Option`. + :param str action: + Any action allowed by :mod:`optparse`. + :param default: + Default value of the option. + :param type: + Any type allowed by :mod:`optparse`. + :param dest: + Attribute name to store parsed option value as. + :param nargs: + Number of arguments to parse for this option. + :param const: + Constant value to store on a common destination. + + """ self.short_option_name = short_option_name self.long_option_name = long_option_name self.option_args = filter(None, (short_option_name, long_option_name)) @@ -40,5 +62,185 @@ class Option(object): for key, value in self.option_kwargs.items(): setattr(self, key, value) self.parse_from_config = parse_from_config - self._option = optparse.Option(*self.option_args, - **self.option_kwargs) + self.comma_separated_list = comma_separated_list + + if parse_from_config: + if not long_option_name: + raise ValueError('When specifying parse_from_config=True, ' + 'a long_option_name must also be specified.') + self.config_name = long_option_name.strip('-').replace('-', '_') + + def to_optparse(self): + """Convert a Flake8 Option to an optparse Option.""" + return optparse.Option(*self.option_args, + **self.option_kwargs) + + +class OptionManager(object): + def __init__(self, prog=None, version=None, + usage='%prog [options] input'): + self.parser = optparse.OptionParser(prog=prog, version=version, + usage=usage) + self.config_options_dict = {} + self.options = [] + self.program_name = prog + self.version = version + + def add_option(self, *args, **kwargs): + option = Option(*args, **kwargs) + self.parser.add_option(option.to_optparse()) + self.options.append(option) + if option.parse_from_config: + self.config_options_dict[option.config_name] = option + + def parse_args(self, args=None, values=None): + return self.parser.parse_args(args, values) + + +class ConfigFileFinder(object): + PROJECT_FILENAMES = ('setup.cfg', 'tox.ini') + + def __init__(self, program_name, args): + # Platform specific settings + self.is_windows = sys.platform == 'win32' + self.xdg_home = os.environ.get('XDG_CONFIG_HOME', + os.path.expanduser('~/.config')) + + # Look for '.' files + self.program_config = '.' + program_name + self.program_name = program_name + + # List of filenames to find in the local/project directory + self.project_filenames = ('setup.cfg', 'tox.ini', self.program_config) + # List of filenames to find "globally" + self.global_filenames = (self.program_config,) + + self.local_directory = os.curdir + + if not args: + args = ['.'] + self.parent = self.tail = os.path.abspath(os.path.commonprefix(args)) + + @staticmethod + def _read_config(files): + config = configparser.RawConfigParser() + found_files = config.read(files) + return (config, found_files) + + def generate_possible_local_config_files(self): + tail = self.tail + parent = self.parent + while tail: + for project_filename in self.project_filenames: + filename = os.path.abspath(os.path.join(parent, + project_filename)) + yield filename + (parent, tail) = os.path.split(parent) + + def local_config_files(self): + return [ + filename + for filename in self.generate_possible_local_config_files() + if os.path.exists(filename) + ] + + def local_configs(self): + config, found_files = self._read_config(self.local_config_files()) + if found_files: + LOG.debug('Found local configuration files: %s', found_files) + return config + + def user_config_file(self): + if self.is_windows: + return os.path.expanduser('~\\' + self.program_config) + return os.path.join(self.xdg_home, self.program_name) + + def user_config(self): + config, found_files = self._read_config(self.user_config_file()) + if found_files: + LOG.debug('Found user configuration files: %s', found_files) + return config + + +class MergedConfigParser(object): + GETINT_METHODS = set(['int', 'count']) + GETBOOL_METHODS = set(['store_true', 'store_false']) + + def __init__(self, option_manager, args=None): + self.option_manager = option_manager + self.program_name = option_manager.program_name + self.args = args + self.config_finder = ConfigFileFinder(self.program_name, self.args) + self.config_options = option_manager.config_options_dict + + def _parse_config(self, config_parser): + config = self.config_finder.local_configs() + if not config.has_section(self.program_name): + LOG.debug('Local configuration files have no %s section', + self.program_name) + return {} + + config_dict = {} + for option_name in config_parser.options(self.program_name): + if option_name not in self.config_options: + LOG.debug('Option "%s" is not registered. Ignoring.', + option_name) + continue + option = self.config_options[option_name] + + # Use the appropriate method to parse the config value + method = config.get + if option.type in self.GETINT_METHODS: + method = config.getint + elif option.action in self.GETBOOL_METHODS: + method = config.getboolean + + value = method(self.program_name, option_name) + LOG.debug('Option "%s" returned value: %r', option_name, value) + + final_value = value + if option.comma_separated_list: + final_value = utils.parse_comma_separated_list(value) + + config_dict[option_name] = final_value + + def is_configured_by(self, config): + """Check if the specified config parser has an appropriate section.""" + return config.has_section(self.program_name) + + def parse_local_config(self): + """Parse and return the local configuration files.""" + config = self.config_finder.local_configs() + if not self.is_configured_by(config): + LOG.debug('Local configuration files have no %s section', + self.program_name) + return + + LOG.debug('Parsing local configuration files.') + return self._parse_config(config) + + def parse_user_config(self): + """Parse and return the user configuration files.""" + config = self.config_finder.user_config() + if not self.is_configured_by(config): + LOG.debug('User configuration files have no %s section', + self.program_name) + return + + LOG.debug('Parsing user configuration files.') + return self._parse_config(config) + + def parse(self): + """Parse and return the local and user config files. + + First this copies over the parsed local configuration and then + iterates over the options in the user configuration and sets them if + they were not set by the local configuration file. + """ + user_config = self.parse_user_config() + config = self.parse_local_config() + + for option, value in user_config.items(): + config.setdefault(option, value) + + return config From 07a29e45db86ed4b6eb95134363b88acd93f6384 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 4 Jan 2016 23:24:11 -0600 Subject: [PATCH 016/364] Split flake8.option_parser into submodule --- flake8/options/__init__.py | 0 .../{option_parser.py => options/config.py} | 85 ------------- flake8/options/manager.py | 119 ++++++++++++++++++ 3 files changed, 119 insertions(+), 85 deletions(-) create mode 100644 flake8/options/__init__.py rename flake8/{option_parser.py => options/config.py} (62%) create mode 100644 flake8/options/manager.py diff --git a/flake8/options/__init__.py b/flake8/options/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/flake8/option_parser.py b/flake8/options/config.py similarity index 62% rename from flake8/option_parser.py rename to flake8/options/config.py index 5b369c0..708cd90 100644 --- a/flake8/option_parser.py +++ b/flake8/options/config.py @@ -1,5 +1,4 @@ import logging -import optparse import os.path import sys @@ -13,90 +12,6 @@ from . import utils LOG = logging.getLogger(__name__) -class Option(object): - def __init__(self, short_option_name=None, long_option_name=None, - # Options below here are taken from the optparse.Option class - action=None, default=None, type=None, dest=None, - nargs=None, const=None, choices=None, callback=None, - callback_args=None, callback_kwargs=None, help=None, - metavar=None, - # Options below here are specific to Flake8 - parse_from_config=False, comma_separated_list=False, - ): - """Initialize an Option instance wrapping optparse.Option. - - :param str short_option_name: - The short name of the option (e.g., ``-x``). This will be the - first argument passed to :class:`~optparse.Option`. - :param str long_option_name: - The long name of the option (e.g., ``--xtra-long-option``). This - will be the second argument passed to :class:`~optparse.Option`. - :param str action: - Any action allowed by :mod:`optparse`. - :param default: - Default value of the option. - :param type: - Any type allowed by :mod:`optparse`. - :param dest: - Attribute name to store parsed option value as. - :param nargs: - Number of arguments to parse for this option. - :param const: - Constant value to store on a common destination. - - """ - self.short_option_name = short_option_name - self.long_option_name = long_option_name - self.option_args = filter(None, (short_option_name, long_option_name)) - self.option_kwargs = { - 'action': action, - 'default': default, - 'type': type, - 'dest': dest, - 'callback': callback, - 'callback_args': callback_args, - 'callback_kwargs': callback_kwargs, - 'help': help, - 'metavar': metavar, - } - for key, value in self.option_kwargs.items(): - setattr(self, key, value) - self.parse_from_config = parse_from_config - self.comma_separated_list = comma_separated_list - - if parse_from_config: - if not long_option_name: - raise ValueError('When specifying parse_from_config=True, ' - 'a long_option_name must also be specified.') - self.config_name = long_option_name.strip('-').replace('-', '_') - - def to_optparse(self): - """Convert a Flake8 Option to an optparse Option.""" - return optparse.Option(*self.option_args, - **self.option_kwargs) - - -class OptionManager(object): - def __init__(self, prog=None, version=None, - usage='%prog [options] input'): - self.parser = optparse.OptionParser(prog=prog, version=version, - usage=usage) - self.config_options_dict = {} - self.options = [] - self.program_name = prog - self.version = version - - def add_option(self, *args, **kwargs): - option = Option(*args, **kwargs) - self.parser.add_option(option.to_optparse()) - self.options.append(option) - if option.parse_from_config: - self.config_options_dict[option.config_name] = option - - def parse_args(self, args=None, values=None): - return self.parser.parse_args(args, values) - - class ConfigFileFinder(object): PROJECT_FILENAMES = ('setup.cfg', 'tox.ini') diff --git a/flake8/options/manager.py b/flake8/options/manager.py new file mode 100644 index 0000000..4b32234 --- /dev/null +++ b/flake8/options/manager.py @@ -0,0 +1,119 @@ +import logging +import optparse + +LOG = logging.getLogger(__name__) + + +class Option(object): + def __init__(self, short_option_name=None, long_option_name=None, + # Options below here are taken from the optparse.Option class + action=None, default=None, type=None, dest=None, + nargs=None, const=None, choices=None, callback=None, + callback_args=None, callback_kwargs=None, help=None, + metavar=None, + # Options below here are specific to Flake8 + parse_from_config=False, comma_separated_list=False, + ): + """Initialize an Option instance wrapping optparse.Option. + + The following are all passed directly through to optparse. + + :param str short_option_name: + The short name of the option (e.g., ``-x``). This will be the + first argument passed to :class:`~optparse.Option`. + :param str long_option_name: + The long name of the option (e.g., ``--xtra-long-option``). This + will be the second argument passed to :class:`~optparse.Option`. + :param str action: + Any action allowed by :mod:`optparse`. + :param default: + Default value of the option. + :param type: + Any type allowed by :mod:`optparse`. + :param dest: + Attribute name to store parsed option value as. + :param nargs: + Number of arguments to parse for this option. + :param const: + Constant value to store on a common destination. Usually used in + conjuntion with ``action="store_const"``. + :param iterable choices: + Possible values for the option. + :param callable callback: + Callback used if the action is ``"callback"``. + :param iterable callback_args: + Additional positional arguments to the callback callable. + :param dictionary callback_kwargs: + Keyword arguments to the callback callable. + :param str help: + Help text displayed in the usage information. + :param str metavar: + Name to use instead of the long option name for help text. + + The following parameters are for Flake8's option handling alone. + + :param bool parse_from_config: + Whether or not this option should be parsed out of config files. + :param bool comma_separated_list: + Whether the option is a comma separated list when parsing from a + config file. + """ + self.short_option_name = short_option_name + self.long_option_name = long_option_name + self.option_args = filter(None, (short_option_name, long_option_name)) + self.option_kwargs = { + 'action': action, + 'default': default, + 'type': type, + 'dest': dest, + 'callback': callback, + 'callback_args': callback_args, + 'callback_kwargs': callback_kwargs, + 'help': help, + 'metavar': metavar, + } + for key, value in self.option_kwargs.items(): + setattr(self, key, value) + self.parse_from_config = parse_from_config + self.comma_separated_list = comma_separated_list + + if parse_from_config: + if not long_option_name: + raise ValueError('When specifying parse_from_config=True, ' + 'a long_option_name must also be specified.') + self.config_name = long_option_name.strip('-').replace('-', '_') + + def __repr__(self): + return ( + 'Option({0}, {1}, action={action}, default={default}, ' + 'dest={dest}, type={type}, callback={callback}, help={help},' + ' callback={callback}, callback_args={callback_args}, ' + 'callback_kwargs={callback_kwargs}, metavar={metavar})' + ).format(*self.option_args, **self.option_kwargs) + + def to_optparse(self): + """Convert a Flake8 Option to an optparse Option.""" + return optparse.Option(*self.option_args, + **self.option_kwargs) + + +class OptionManager(object): + def __init__(self, prog=None, version=None, + usage='%prog [options] input'): + self.parser = optparse.OptionParser(prog=prog, version=version, + usage=usage) + self.config_options_dict = {} + self.options = [] + self.program_name = prog + self.version = version + + def add_option(self, *args, **kwargs): + option = Option(*args, **kwargs) + self.parser.add_option(option.to_optparse()) + self.options.append(option) + if option.parse_from_config: + self.config_options_dict[option.config_name] = option + LOG.debug('Registered option "%s".', option) + + def parse_args(self, args=None, values=None): + return self.parser.parse_args(args, values) From 67aff6d2ec0dbb558bcbee98e0c4a932d87f739c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 7 Jan 2016 09:37:34 -0600 Subject: [PATCH 017/364] Updates to config/options aggregation - Adds another flag to the Option class - Adds normalization to the config parsing - Removes dead code from _parse_config - Handles user specifying --append-config, --config, and --isolated but does not handle their mutual exclusion --- flake8/options/aggregator.py | 51 +++++++++++++++++++++ flake8/options/config.py | 88 ++++++++++++++++++++++++++++-------- flake8/options/manager.py | 27 +++++++++++ 3 files changed, 148 insertions(+), 18 deletions(-) create mode 100644 flake8/options/aggregator.py diff --git a/flake8/options/aggregator.py b/flake8/options/aggregator.py new file mode 100644 index 0000000..863fa05 --- /dev/null +++ b/flake8/options/aggregator.py @@ -0,0 +1,51 @@ +"""Aggregation function for CLI specified options and config file options. + +This holds the logic that uses the collected and merged config files and +applies the user-specified command-line configuration on top of it. +""" +import logging + +from flake8.options import config +from flake8 import utils + +LOG = logging.getLogger(__name__) + + +def aggregate_options(manager, arglist=None, values=None): + """Function that aggregates the CLI and Config options.""" + # Get defaults from the option parser + default_values, _ = manager.parse_args([], values=values) + # Get original CLI values so we can find additional config file paths and + # see if --config was specified. + original_values, original_args = manager.parse_args() + extra_config_files = utils.normalize_paths(original_values.append_config) + + # Make our new configuration file mergerator + config_parser = config.MergedConfigParser( + option_manager=manager, + extra_config_files=extra_config_files, + args=original_args, + ) + + # Get the parsed config + parsed_config = config_parser.parse(original_values.config, + original_values.isolated) + + # Merge values parsed from config onto the default values returned + for config_name, value in parsed_config.items(): + dest_name = config_name + # If the config name is somehow different from the destination name, + # fetch the destination name from our Option + if not hasattr(default_values, config_name): + dest_name = config_parser.config_options[config_name].dest + + LOG.debug('Overriding default value of (%s) for "%s" with (%s)', + getattr(default_values, dest_name, None), + dest_name, + value) + # Override the default values with the config values + setattr(default_values, dest_name, value) + + # Finally parse the command-line options + final_values, args = manager.parse_args(arglist, default_values) + return final_values, args diff --git a/flake8/options/config.py b/flake8/options/config.py index 708cd90..9317e7a 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -11,11 +11,16 @@ from . import utils LOG = logging.getLogger(__name__) +__all__('ConfigFileFinder', 'MergedConfigParser') + class ConfigFileFinder(object): PROJECT_FILENAMES = ('setup.cfg', 'tox.ini') - def __init__(self, program_name, args): + def __init__(self, program_name, args, extra_config_files): + # The values of --append-config from the CLI + self.extra_config_files = extra_config_files + # Platform specific settings self.is_windows = sys.platform == 'win32' self.xdg_home = os.environ.get('XDG_CONFIG_HOME', @@ -27,8 +32,6 @@ class ConfigFileFinder(object): # List of filenames to find in the local/project directory self.project_filenames = ('setup.cfg', 'tox.ini', self.program_config) - # List of filenames to find "globally" - self.global_filenames = (self.program_config,) self.local_directory = os.curdir @@ -42,6 +45,12 @@ class ConfigFileFinder(object): found_files = config.read(files) return (config, found_files) + def cli_config(self, files): + config, found_files = self._read_config(files) + if found_files: + LOG.debug('Found cli configuration files: %s', found_files) + return config + def generate_possible_local_config_files(self): tail = self.tail parent = self.parent @@ -57,7 +66,7 @@ class ConfigFileFinder(object): filename for filename in self.generate_possible_local_config_files() if os.path.exists(filename) - ] + ] + self.extra_config_files def local_configs(self): config, found_files = self._read_config(self.local_config_files()) @@ -81,20 +90,33 @@ class MergedConfigParser(object): GETINT_METHODS = set(['int', 'count']) GETBOOL_METHODS = set(['store_true', 'store_false']) - def __init__(self, option_manager, args=None): + def __init__(self, option_manager, extra_config_files=None, args=None): + # Our instance of flake8.options.manager.OptionManager self.option_manager = option_manager + # The prog value for the cli parser self.program_name = option_manager.program_name + # Parsed extra arguments self.args = args + # Our instance of our ConfigFileFinder self.config_finder = ConfigFileFinder(self.program_name, self.args) + # Mapping of configuration option names to + # flake8.options.manager.Option instances self.config_options = option_manager.config_options_dict + self.extra_config_files = extra_config_files or [] + + @staticmethod + def _normalize_value(option, value): + if option.normalize_paths: + final_value = utils.normalize_paths(value) + elif option.comma_separated_list: + final_value = utils.parse_comma_separated_list(value) + else: + final_value = value + LOG.debug('%r has been normalized to %r for option "%s"', + value, final_value, option.config_name) + return final_value def _parse_config(self, config_parser): - config = self.config_finder.local_configs() - if not config.has_section(self.program_name): - LOG.debug('Local configuration files have no %s section', - self.program_name) - return {} - config_dict = {} for option_name in config_parser.options(self.program_name): if option_name not in self.config_options: @@ -113,10 +135,7 @@ class MergedConfigParser(object): value = method(self.program_name, option_name) LOG.debug('Option "%s" returned value: %r', option_name, value) - final_value = value - if option.comma_separated_list: - final_value = utils.parse_comma_separated_list(value) - + final_value = self._normalize_value(value) config_dict[option_name] = final_value def is_configured_by(self, config): @@ -129,7 +148,7 @@ class MergedConfigParser(object): if not self.is_configured_by(config): LOG.debug('Local configuration files have no %s section', self.program_name) - return + return {} LOG.debug('Parsing local configuration files.') return self._parse_config(config) @@ -140,18 +159,51 @@ class MergedConfigParser(object): if not self.is_configured_by(config): LOG.debug('User configuration files have no %s section', self.program_name) - return + return {} LOG.debug('Parsing user configuration files.') return self._parse_config(config) - def parse(self): + def parse_cli_config(self, config_path): + """Parse and return the file specified by --config.""" + config = self.config_finder.cli_config(config_path) + if not self.is_configured_by(config): + LOG.debug('CLI configuration files have no %s section', + self.program_name) + return {} + + LOG.debug('Parsing CLI configuration files.') + return self._parse_config(config) + + def parse(self, cli_config=None, isolated=False): """Parse and return the local and user config files. First this copies over the parsed local configuration and then iterates over the options in the user configuration and sets them if they were not set by the local configuration file. + + :param str cli_config: + Value of --config when specified at the command-line. Overrides + all other config files. + :param bool isolated: + Determines if we should parse configuration files at all or not. + If running in isolated mode, we ignore all configuration files + :returns: + Dictionary of parsed configuration options + :rtype: + dict """ + if isolated: + LOG.debug('Refusing to parse configuration files due to user-' + 'requested isolation') + return {} + + if cli_config: + LOG.debug('Ignoring user and locally found configuration files. ' + 'Reading only configuration from "%s" specified via ' + '--config by the user', cli_config) + return self.parse_cli_config(cli_config) + user_config = self.parse_user_config() config = self.parse_local_config() diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 4b32234..034444b 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -5,6 +5,7 @@ LOG = logging.getLogger(__name__) class Option(object): + """Our wrapper around an optparse.Option object to add features.""" def __init__(self, short_option_name=None, long_option_name=None, # Options below here are taken from the optparse.Option class action=None, default=None, type=None, dest=None, @@ -13,6 +14,7 @@ class Option(object): metavar=None, # Options below here are specific to Flake8 parse_from_config=False, comma_separated_list=False, + normalize_paths=False, ): """Initialize an Option instance wrapping optparse.Option. @@ -57,6 +59,9 @@ class Option(object): :param bool comma_separated_list: Whether the option is a comma separated list when parsing from a config file. + :param bool normalize_paths: + Whether the option is expecting a path or list of paths and should + attempt to normalize the paths to absolute paths. """ self.short_option_name = short_option_name self.long_option_name = long_option_name @@ -72,11 +77,16 @@ class Option(object): 'help': help, 'metavar': metavar, } + # Set attributes for our option arguments for key, value in self.option_kwargs.items(): setattr(self, key, value) + + # Set our custom attributes self.parse_from_config = parse_from_config self.comma_separated_list = comma_separated_list + self.normalize_paths = normalize_paths + self.config_name = None if parse_from_config: if not long_option_name: raise ValueError('When specifying parse_from_config=True, ' @@ -108,6 +118,16 @@ class OptionManager(object): self.version = version def add_option(self, *args, **kwargs): + """Create and register a new option. + + See parameters for :class:`~flake8.options.manager.Option` for + acceptable arguments to this method. + + .. note:: + + ``short_option_name`` and ``long_option_name`` may be specified + positionally as they are with optparse normally. + """ option = Option(*args, **kwargs) self.parser.add_option(option.to_optparse()) self.options.append(option) @@ -116,4 +136,11 @@ class OptionManager(object): LOG.debug('Registered option "%s".', option) def parse_args(self, args=None, values=None): + """Simple proxy to calling the OptionParser's parse_args method. + + .. todo:: + + Normalize values based on our extra attributes from + :class:`~flake8.options.manager.OptionManager`. + """ return self.parser.parse_args(args, values) From 93369e112f290e233c030ba9d0b36360a65649db Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 12:40:26 -0600 Subject: [PATCH 018/364] Add empty module to start testing options submodule --- flake8/utils.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 flake8/utils.py diff --git a/flake8/utils.py b/flake8/utils.py new file mode 100644 index 0000000..e69de29 From 1e9878611a93b3af583e222a15c0b4575fc4f027 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 12:55:31 -0600 Subject: [PATCH 019/364] Move unit tests into tests/unit --- tests/unit/__init__.py | 0 tests/{ => unit}/test_notifier.py | 0 tests/unit/test_option.py | 52 +++++++++++++++++++++++++++++++ tests/{ => unit}/test_trie.py | 0 4 files changed, 52 insertions(+) create mode 100644 tests/unit/__init__.py rename tests/{ => unit}/test_notifier.py (100%) create mode 100644 tests/unit/test_option.py rename tests/{ => unit}/test_trie.py (100%) diff --git a/tests/unit/__init__.py b/tests/unit/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/test_notifier.py b/tests/unit/test_notifier.py similarity index 100% rename from tests/test_notifier.py rename to tests/unit/test_notifier.py diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py new file mode 100644 index 0000000..d1491eb --- /dev/null +++ b/tests/unit/test_option.py @@ -0,0 +1,52 @@ +"""Unit tests for flake8.options.manager.Option.""" +import mock + +from flake8.options import manager + + +def test_to_optparse(): + """Test conversion to an optparse.Option class.""" + opt = manager.Option( + short_option_name='-t', + long_option_name='--test', + action='count', + parse_from_config=True, + normalize_paths=True, + ) + assert opt.normalize_paths is True + assert opt.parse_from_config is True + + optparse_opt = opt.to_optparse() + assert not hasattr(optparse_opt, 'parse_from_config') + assert not hasattr(optparse_opt, 'normalize_paths') + assert optparse_opt.action == 'count' + + +@mock.patch('optparse.Option') +def test_to_optparse_creates_an_option_as_we_expect(Option): + """Show that we pass all keyword args to optparse.Option.""" + opt = manager.Option('-t', '--test', action='count') + opt.to_optparse() + option_kwargs = { + 'action': 'count', + 'default': None, + 'type': None, + 'dest': None, + 'callback': None, + 'callback_args': None, + 'callback_kwargs': None, + 'help': None, + 'metavar': None, + } + + Option.assert_called_once_with( + '-t', '--test', **option_kwargs + ) + + +def test_config_name_generation(): + """Show that we generate the config name deterministically.""" + opt = manager.Option(long_option_name='--some-very-long-option-name', + parse_from_config=True) + + assert opt.config_name == 'some_very_long_option_name' diff --git a/tests/test_trie.py b/tests/unit/test_trie.py similarity index 100% rename from tests/test_trie.py rename to tests/unit/test_trie.py From 19e5ea767d8a087fb63e0236252c9c316ff66904 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 12:55:43 -0600 Subject: [PATCH 020/364] Add the options submodule to the package --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index fb0585e..d0650f4 100644 --- a/setup.py +++ b/setup.py @@ -47,7 +47,7 @@ setup( maintainer="Ian Cordasco", maintainer_email="graffatcolmingov@gmail.com", url="https://gitlab.com/pycqa/flake8", - packages=["flake8"], + packages=["flake8", "flake8.options"], install_requires=[ "pyflakes >= 0.8.1, < 1.1", "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", From 3a8f99b4580fea358517e749bbba6599c40ba9e4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 12:56:03 -0600 Subject: [PATCH 021/364] Add mock to list of test dependencies --- tox.ini | 1 + 1 file changed, 1 insertion(+) diff --git a/tox.ini b/tox.ini index de47800..ecaae55 100644 --- a/tox.ini +++ b/tox.ini @@ -3,6 +3,7 @@ envlist = py26,py27,py32,py33,py34,py35 [testenv] deps = + mock pytest commands = py.test From 7361603d7a5b41b4e878ba1c2dad0890fb888d68 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 12:56:16 -0600 Subject: [PATCH 022/364] Flesh out OptionManager#parse_args --- flake8/options/manager.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 034444b..baccd95 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -1,6 +1,8 @@ import logging import optparse +from flake8 import utils + LOG = logging.getLogger(__name__) @@ -136,11 +138,16 @@ class OptionManager(object): LOG.debug('Registered option "%s".', option) def parse_args(self, args=None, values=None): - """Simple proxy to calling the OptionParser's parse_args method. + """Simple proxy to calling the OptionParser's parse_args method.""" + options, xargs = self.parser.parse_args(args, values) + for config_name, option in self.config_options_dict.items(): + dest = option.dest or config_name + if self.normalize_paths: + old_value = getattr(options, dest) + setattr(options, dest, utils.normalize_paths(old_value)) + elif self.comma_separated_list: + old_value = getattr(options, dest) + setattr(options, dest, + utils.parse_comma_separated_list(old_value)) - .. todo:: - - Normalize values based on our extra attributes from - :class:`~flake8.options.manager.OptionManager`. - """ - return self.parser.parse_args(args, values) + return options, xargs From a7ce4f00ec1f002780184c724fdce3b6fdc74ce3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 13:53:59 -0600 Subject: [PATCH 023/364] Start testing the OptionManager --- flake8/options/manager.py | 8 ++++-- tests/unit/test_option_manager.py | 46 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 2 deletions(-) create mode 100644 tests/unit/test_option_manager.py diff --git a/flake8/options/manager.py b/flake8/options/manager.py index baccd95..a471960 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -105,8 +105,10 @@ class Option(object): def to_optparse(self): """Convert a Flake8 Option to an optparse Option.""" - return optparse.Option(*self.option_args, - **self.option_kwargs) + if not hasattr(self, '_opt'): + self._opt = optparse.Option(*self.option_args, + **self.option_kwargs) + return self._opt class OptionManager(object): @@ -130,6 +132,8 @@ class OptionManager(object): ``short_option_name`` and ``long_option_name`` may be specified positionally as they are with optparse normally. """ + if len(args) == 1 and args[0].startswith('--'): + args = (None, args[0]) option = Option(*args, **kwargs) self.parser.add_option(option.to_optparse()) self.options.append(option) diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py new file mode 100644 index 0000000..27c47d6 --- /dev/null +++ b/tests/unit/test_option_manager.py @@ -0,0 +1,46 @@ +"""Unit tests for flake.options.manager.OptionManager.""" +import optparse + +import pytest + +from flake8.options import manager + + +@pytest.fixture +def optmanager(): + return manager.OptionManager(prog='flake8', version='3.0.0b1') + + +def test_option_manager_creates_option_parser(optmanager): + """Verify that a new manager creates a new parser.""" + assert optmanager.parser is not None + assert isinstance(optmanager.parser, optparse.OptionParser) is True + + +def test_add_option_short_option_only(optmanager): + """Verify the behaviour of adding a short-option only.""" + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + + optmanager.add_option('-s', help='Test short opt') + assert optmanager.options[0].short_option_name == '-s' + + +def test_add_option_long_option_only(optmanager): + """Verify the behaviour of adding a long-option only.""" + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + + optmanager.add_option('--long', help='Test long opt') + assert optmanager.options[0].short_option_name is None + assert optmanager.options[0].long_option_name == '--long' + + +def test_add_short_and_long_option_names(optmanager): + """Verify the behaviour of using both short and long option names.""" + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + + optmanager.add_option('-b', '--both', help='Test both opts') + assert optmanager.options[0].short_option_name == '-b' + assert optmanager.options[0].long_option_name == '--both' From 5a6656d78de3a2f9051fc053bb6294971abfcdd1 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 15:22:03 -0600 Subject: [PATCH 024/364] Test parse_from_config behaviour of Options --- tests/unit/test_option.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py index d1491eb..51b5dcf 100644 --- a/tests/unit/test_option.py +++ b/tests/unit/test_option.py @@ -1,5 +1,6 @@ """Unit tests for flake8.options.manager.Option.""" import mock +import pytest from flake8.options import manager @@ -50,3 +51,9 @@ def test_config_name_generation(): parse_from_config=True) assert opt.config_name == 'some_very_long_option_name' + + +def test_config_name_needs_long_option_name(): + """Show that we error out if the Option should be parsed from config.""" + with pytest.raises(ValueError): + manager.Option('-s', parse_from_config=True) From 09bc9e2720546fb5bbbc0015148e8e108ec88d24 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 15:22:22 -0600 Subject: [PATCH 025/364] Add test for custom arguments to add_option --- tests/unit/test_option_manager.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 27c47d6..4983c1d 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -44,3 +44,16 @@ def test_add_short_and_long_option_names(optmanager): optmanager.add_option('-b', '--both', help='Test both opts') assert optmanager.options[0].short_option_name == '-b' assert optmanager.options[0].long_option_name == '--both' + + +def test_add_option_with_custom_args(optmanager): + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + + optmanager.add_option('--parse', parse_from_config=True) + optmanager.add_option('--commas', comma_separated_list=True) + optmanager.add_option('--files', normalize_paths=True) + + attrs = ['parse_from_config', 'comma_separated_list', 'normalize_paths'] + for option, attr in zip(optmanager.options, attrs): + assert getattr(option, attr) is True From 36cc2545426d788c93443c24265b7b51cb8494fd Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 15:32:44 -0600 Subject: [PATCH 026/364] Generate dest for Options always --- flake8/options/manager.py | 9 +++++++-- tests/unit/test_option.py | 2 +- 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index a471960..f4bd1d8 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -72,7 +72,7 @@ class Option(object): 'action': action, 'default': default, 'type': type, - 'dest': dest, + 'dest': self._make_dest(dest), 'callback': callback, 'callback_args': callback_args, 'callback_kwargs': callback_kwargs, @@ -93,7 +93,7 @@ class Option(object): if not long_option_name: raise ValueError('When specifying parse_from_config=True, ' 'a long_option_name must also be specified.') - self.config_name = long_option_name.strip('-').replace('-', '_') + self.config_name = self.dest def __repr__(self): return ( @@ -103,6 +103,11 @@ class Option(object): 'callback_kwargs={callback_kwargs}, metavar={metavar})' ).format(*self.option_args, **self.option_kwargs) + def _make_dest(self, dest): + if self.long_option_name: + return self.long_option_name[2:].replace('-', '_') + return self.short_option_name[1] + def to_optparse(self): """Convert a Flake8 Option to an optparse Option.""" if not hasattr(self, '_opt'): diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py index 51b5dcf..c6fc98c 100644 --- a/tests/unit/test_option.py +++ b/tests/unit/test_option.py @@ -32,7 +32,7 @@ def test_to_optparse_creates_an_option_as_we_expect(Option): 'action': 'count', 'default': None, 'type': None, - 'dest': None, + 'dest': 'test', 'callback': None, 'callback_args': None, 'callback_kwargs': None, From ad0200e792361d2909955135bca49117ab7ad4e0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 15:33:01 -0600 Subject: [PATCH 027/364] Add failing test for OptionManager.parse_args --- flake8/options/manager.py | 8 ++++---- tests/unit/test_option_manager.py | 17 +++++++++++++++++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index f4bd1d8..6c0d036 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -149,12 +149,12 @@ class OptionManager(object): def parse_args(self, args=None, values=None): """Simple proxy to calling the OptionParser's parse_args method.""" options, xargs = self.parser.parse_args(args, values) - for config_name, option in self.config_options_dict.items(): - dest = option.dest or config_name - if self.normalize_paths: + for option in self.options: + dest = option.dest + if option.normalize_paths: old_value = getattr(options, dest) setattr(options, dest, utils.normalize_paths(old_value)) - elif self.comma_separated_list: + elif option.comma_separated_list: old_value = getattr(options, dest) setattr(options, dest, utils.parse_comma_separated_list(old_value)) diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 4983c1d..37c9570 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -1,5 +1,6 @@ """Unit tests for flake.options.manager.OptionManager.""" import optparse +import os import pytest @@ -57,3 +58,19 @@ def test_add_option_with_custom_args(optmanager): attrs = ['parse_from_config', 'comma_separated_list', 'normalize_paths'] for option, attr in zip(optmanager.options, attrs): assert getattr(option, attr) is True + + +def test_parse_args(optmanager): + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + + optmanager.add_option('-v', '--verbose', action='count') + optmanager.add_option('--config', normalize_paths=True) + optmanager.add_option('--exclude', default='E123,W234', + comma_separated_list=True) + + options, args = optmanager.parse_args( + ['-v', '-v', '-v', '--config', '../config.ini'] + ) + assert options.verbose == 3 + assert options.config == os.path.abspath('../config.ini') From a4042d6d692354f486d9f3e3f810d08ff3d7339e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 15:58:26 -0600 Subject: [PATCH 028/364] Start fleshing out flake8.utils Add parse_comma_separated_list, normalize_path, and normalize_paths and add logic in OptionManager.parse_args to use the right normalize_path* function based on comma_separated_list value of the option. --- flake8/options/manager.py | 25 +++++++++++++------- flake8/utils.py | 50 +++++++++++++++++++++++++++++++++++++++ tests/unit/test_utils.py | 14 +++++++++++ 3 files changed, 81 insertions(+), 8 deletions(-) create mode 100644 tests/unit/test_utils.py diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 6c0d036..3e4c128 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -150,13 +150,22 @@ class OptionManager(object): """Simple proxy to calling the OptionParser's parse_args method.""" options, xargs = self.parser.parse_args(args, values) for option in self.options: - dest = option.dest - if option.normalize_paths: - old_value = getattr(options, dest) - setattr(options, dest, utils.normalize_paths(old_value)) - elif option.comma_separated_list: - old_value = getattr(options, dest) - setattr(options, dest, - utils.parse_comma_separated_list(old_value)) + _normalize_option(options, option) return options, xargs + + +def _normalize_option(options, option): + dest = option.dest + if option.normalize_paths: + old_value = getattr(options, dest) + # Decide whether to parse a list of paths or a single path + normalize = utils.normalize_path + if option.comma_separated_list: + normalize = utils.normalize_paths + setattr(options, dest, normalize(old_value)) + + elif option.comma_separated_list: + old_value = getattr(options, dest) + setattr(options, dest, + utils.parse_comma_separated_list(old_value)) diff --git a/flake8/utils.py b/flake8/utils.py index e69de29..72c7852 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -0,0 +1,50 @@ +"""Utility methods for flake8.""" +import os + + +def parse_comma_separated_list(value): + """Parse a comma-separated list. + + :param value: + String or list of strings to be parsed and normalized. + :returns: + List of values with whitespace stripped. + :rtype: + list + """ + if not value: + return [] + + if not isinstance(value, (list, tuple)): + value = value.split(',') + + return [item.strip() for item in value] + + +def normalize_paths(paths, parent=os.curdir): + """Parse a comma-separated list of paths. + + :returns: + The normalized paths. + :rtype: + [str] + """ + paths = [] + for path in parse_comma_separated_list(paths): + if '/' in path: + path = os.path.abspath(os.path.join(parent, path)) + paths.append(path.rstrip('/')) + return paths + + +def normalize_path(path, parent=os.curdir): + """Normalize a single-path. + + :returns: + The normalized path. + :rtype: + str + """ + if '/' in path: + path = os.path.abspath(os.path.join(parent, path)) + return path diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py new file mode 100644 index 0000000..28c47c8 --- /dev/null +++ b/tests/unit/test_utils.py @@ -0,0 +1,14 @@ +"""Tests for flake8's utils module.""" +import pytest + +from flake8 import utils + + +@pytest.mark.parametrize("value,expected", [ + ("E123,\n\tW234,\n E206", ["E123", "W234", "E206"]), + ("E123,W234,E206", ["E123", "W234", "E206"]), + (["E123", "W234", "E206"], ["E123", "W234", "E206"]), + (["E123", "\n\tW234", "\n E206"], ["E123", "W234", "E206"]), +]) +def test_parse_comma_separated_list(value, expected): + assert utils.parse_comma_separated_list(value) == expected From a7f7bbbeb2002ba0f581d6f77bca664c2057a4a6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 20:29:09 -0600 Subject: [PATCH 029/364] Handle when dest is provided to Option --- flake8/options/manager.py | 5 ++++- tests/unit/test_option.py | 5 +++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 3e4c128..38eb605 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -93,7 +93,7 @@ class Option(object): if not long_option_name: raise ValueError('When specifying parse_from_config=True, ' 'a long_option_name must also be specified.') - self.config_name = self.dest + self.config_name = long_option_name[2:].replace('-', '_') def __repr__(self): return ( @@ -104,6 +104,9 @@ class Option(object): ).format(*self.option_args, **self.option_kwargs) def _make_dest(self, dest): + if dest: + return dest + if self.long_option_name: return self.long_option_name[2:].replace('-', '_') return self.short_option_name[1] diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py index c6fc98c..55cae5c 100644 --- a/tests/unit/test_option.py +++ b/tests/unit/test_option.py @@ -57,3 +57,8 @@ def test_config_name_needs_long_option_name(): """Show that we error out if the Option should be parsed from config.""" with pytest.raises(ValueError): manager.Option('-s', parse_from_config=True) + + +def test_dest_is_not_overridden(): + opt = manager.Option('-s', '--short', dest='something_not_short') + assert opt.dest == 'something_not_short' From 13ca3dfe37885cb3cf384a9fcc10fe16866cc223 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 20:30:28 -0600 Subject: [PATCH 030/364] Show that even defaults are properly normalized --- tests/unit/test_option_manager.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 37c9570..cc23669 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -60,7 +60,7 @@ def test_add_option_with_custom_args(optmanager): assert getattr(option, attr) is True -def test_parse_args(optmanager): +def test_parse_args_normalize_path(optmanager): assert optmanager.options == [] assert optmanager.config_options_dict == {} @@ -74,3 +74,4 @@ def test_parse_args(optmanager): ) assert options.verbose == 3 assert options.config == os.path.abspath('../config.ini') + assert options.exclude == ['E123', 'W234'] From ec26d46dffac5ace86222e99b465a8d1d73dbc8e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 21:04:11 -0600 Subject: [PATCH 031/364] Add tests for normalize_path and normalize_paths --- tests/unit/test_utils.py | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 28c47c8..b0e243c 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -1,9 +1,14 @@ """Tests for flake8's utils module.""" +import os + import pytest from flake8 import utils +RELATIVE_PATHS = ["flake8", "pep8", "pyflakes", "mccabe"] + + @pytest.mark.parametrize("value,expected", [ ("E123,\n\tW234,\n E206", ["E123", "W234", "E206"]), ("E123,W234,E206", ["E123", "W234", "E206"]), @@ -11,4 +16,26 @@ from flake8 import utils (["E123", "\n\tW234", "\n E206"], ["E123", "W234", "E206"]), ]) def test_parse_comma_separated_list(value, expected): + """Verify that similar inputs produce identical outputs.""" assert utils.parse_comma_separated_list(value) == expected + + +@pytest.mark.parametrize("value,expected", [ + ("flake8", "flake8"), + ("../flake8", os.path.abspath("../flake8")), + ("flake8/", os.path.abspath("flake8")), +]) +def test_normalize_path(value, expected): + """Verify that we normalize paths provided to the tool.""" + assert utils.normalize_path(value) == expected + + +@pytest.mark.parametrize("value,expected", [ + ("flake8,pep8,pyflakes,mccabe", ["flake8", "pep8", "pyflakes", "mccabe"]), + ("flake8,\n\tpep8,\n pyflakes,\n\n mccabe", + ["flake8", "pep8", "pyflakes", "mccabe"]), + ("../flake8,../pep8,../pyflakes,../mccabe", + [os.path.abspath("../" + p) for p in RELATIVE_PATHS]), +]) +def test_normalize_paths(value, expected): + assert utils.normalize_paths(value) == expected From 237a35fe8841475b944374b45039f040d75ade09 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 21:14:36 -0600 Subject: [PATCH 032/364] Simplify normalize_paths --- flake8/options/manager.py | 1 - flake8/utils.py | 9 ++------- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 38eb605..723a700 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -167,7 +167,6 @@ def _normalize_option(options, option): if option.comma_separated_list: normalize = utils.normalize_paths setattr(options, dest, normalize(old_value)) - elif option.comma_separated_list: old_value = getattr(options, dest) setattr(options, dest, diff --git a/flake8/utils.py b/flake8/utils.py index 72c7852..0dc16e2 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -29,12 +29,7 @@ def normalize_paths(paths, parent=os.curdir): :rtype: [str] """ - paths = [] - for path in parse_comma_separated_list(paths): - if '/' in path: - path = os.path.abspath(os.path.join(parent, path)) - paths.append(path.rstrip('/')) - return paths + return [normalize_path(p) for p in parse_comma_separated_list(paths)] def normalize_path(path, parent=os.curdir): @@ -47,4 +42,4 @@ def normalize_path(path, parent=os.curdir): """ if '/' in path: path = os.path.abspath(os.path.join(parent, path)) - return path + return path.rstrip('/') From cc3a9210fd558340d3d50192682bc3df1cbd40a7 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 21:15:03 -0600 Subject: [PATCH 033/364] Add more tests around parse_args --- tests/unit/test_option_manager.py | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index cc23669..9b1dbba 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -48,6 +48,7 @@ def test_add_short_and_long_option_names(optmanager): def test_add_option_with_custom_args(optmanager): + """Verify that add_option handles custom Flake8 parameters.""" assert optmanager.options == [] assert optmanager.config_options_dict == {} @@ -61,17 +62,35 @@ def test_add_option_with_custom_args(optmanager): def test_parse_args_normalize_path(optmanager): + """Show that parse_args handles path normalization.""" assert optmanager.options == [] assert optmanager.config_options_dict == {} - optmanager.add_option('-v', '--verbose', action='count') optmanager.add_option('--config', normalize_paths=True) + + options, args = optmanager.parse_args(['--config', '../config.ini']) + assert options.config == os.path.abspath('../config.ini') + + +def test_parse_args_handles_comma_separated_defaults(optmanager): + """Show that parse_args handles defaults that are comma separated.""" + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + optmanager.add_option('--exclude', default='E123,W234', comma_separated_list=True) - options, args = optmanager.parse_args( - ['-v', '-v', '-v', '--config', '../config.ini'] - ) - assert options.verbose == 3 - assert options.config == os.path.abspath('../config.ini') + options, args = optmanager.parse_args([]) assert options.exclude == ['E123', 'W234'] + + +def test_parse_args_handles_comma_separated_lists(optmanager): + """Show that parse_args handles user-specified comma separated lists.""" + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + + optmanager.add_option('--exclude', default='E123,W234', + comma_separated_list=True) + + options, args = optmanager.parse_args(['--exclude', 'E201,W111,F280']) + assert options.exclude == ['E201', 'W111', 'F280'] From 684f19f5e3f7f62e58fe87adfaf6c44dc50ff43f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 21:17:22 -0600 Subject: [PATCH 034/364] Re-factor normalization in parse_args --- flake8/options/manager.py | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 723a700..353ab55 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -111,6 +111,20 @@ class Option(object): return self.long_option_name[2:].replace('-', '_') return self.short_option_name[1] + def normalize(self, options): + dest = self.dest + if self.normalize_paths: + old_value = getattr(options, dest) + # Decide whether to parse a list of paths or a single path + normalize = utils.normalize_path + if self.comma_separated_list: + normalize = utils.normalize_paths + setattr(options, dest, normalize(old_value)) + elif self.comma_separated_list: + old_value = getattr(options, dest) + setattr(options, dest, + utils.parse_comma_separated_list(old_value)) + def to_optparse(self): """Convert a Flake8 Option to an optparse Option.""" if not hasattr(self, '_opt'): @@ -153,21 +167,6 @@ class OptionManager(object): """Simple proxy to calling the OptionParser's parse_args method.""" options, xargs = self.parser.parse_args(args, values) for option in self.options: - _normalize_option(options, option) + option.normalize(options) return options, xargs - - -def _normalize_option(options, option): - dest = option.dest - if option.normalize_paths: - old_value = getattr(options, dest) - # Decide whether to parse a list of paths or a single path - normalize = utils.normalize_path - if option.comma_separated_list: - normalize = utils.normalize_paths - setattr(options, dest, normalize(old_value)) - elif option.comma_separated_list: - old_value = getattr(options, dest) - setattr(options, dest, - utils.parse_comma_separated_list(old_value)) From 9fe9ef06f7d00a9082aa5a43e80b97e8a5f07362 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 21:20:20 -0600 Subject: [PATCH 035/364] Add test for comma-separated paths normalization --- tests/unit/test_option_manager.py | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 9b1dbba..a9e641d 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -73,7 +73,7 @@ def test_parse_args_normalize_path(optmanager): def test_parse_args_handles_comma_separated_defaults(optmanager): - """Show that parse_args handles defaults that are comma separated.""" + """Show that parse_args handles defaults that are comma-separated.""" assert optmanager.options == [] assert optmanager.config_options_dict == {} @@ -85,7 +85,7 @@ def test_parse_args_handles_comma_separated_defaults(optmanager): def test_parse_args_handles_comma_separated_lists(optmanager): - """Show that parse_args handles user-specified comma separated lists.""" + """Show that parse_args handles user-specified comma-separated lists.""" assert optmanager.options == [] assert optmanager.config_options_dict == {} @@ -94,3 +94,21 @@ def test_parse_args_handles_comma_separated_lists(optmanager): options, args = optmanager.parse_args(['--exclude', 'E201,W111,F280']) assert options.exclude == ['E201', 'W111', 'F280'] + + +def test_parse_args_normalize_paths(optmanager): + """Verify parse_args normalizes a comma-separated list of paths.""" + assert optmanager.options == [] + assert optmanager.config_options_dict == {} + + optmanager.add_option('--extra-config', normalize_paths=True, + comma_separated_list=True) + + options, args = optmanager.parse_args([ + '--extra-config', '../config.ini,tox.ini,flake8/some-other.cfg' + ]) + assert options.extra_config == [ + os.path.abspath('../config.ini'), + 'tox.ini', + os.path.abspath('flake8/some-other.cfg'), + ] From e076fec666b087c86d392545e949ec9aea62fe4c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 21:28:49 -0600 Subject: [PATCH 036/364] Add some docstrings to manager module --- flake8/options/manager.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 353ab55..e249ffb 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -1,3 +1,4 @@ +"""Option handling and Option management logic.""" import logging import optparse @@ -8,6 +9,7 @@ LOG = logging.getLogger(__name__) class Option(object): """Our wrapper around an optparse.Option object to add features.""" + def __init__(self, short_option_name=None, long_option_name=None, # Options below here are taken from the optparse.Option class action=None, default=None, type=None, dest=None, @@ -96,6 +98,7 @@ class Option(object): self.config_name = long_option_name[2:].replace('-', '_') def __repr__(self): + """Simple representation of an Option class.""" return ( 'Option({0}, {1}, action={action}, default={default}, ' 'dest={dest}, type={type}, callback={callback}, help={help},' @@ -134,8 +137,19 @@ class Option(object): class OptionManager(object): + """Manage Options and OptionParser while adding post-processing.""" + def __init__(self, prog=None, version=None, usage='%prog [options] input'): + """Initialize an instance of an OptionManager. + + :param str prog: + Name of the actual program (e.g., flake8). + :param str version: + Version string for the program. + :param str usage: + Basic usage string used by the OptionParser. + """ self.parser = optparse.OptionParser(prog=prog, version=version, usage=usage) self.config_options_dict = {} From b4b59a96279da897d2a0fc545682051d46c0dc43 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 21:29:08 -0600 Subject: [PATCH 037/364] Make Option#normalize more reusable --- flake8/options/manager.py | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index e249ffb..fafc1e9 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -114,19 +114,17 @@ class Option(object): return self.long_option_name[2:].replace('-', '_') return self.short_option_name[1] - def normalize(self, options): - dest = self.dest + def normalize(self, value): + """Normalize the value based on the option configuration.""" if self.normalize_paths: - old_value = getattr(options, dest) # Decide whether to parse a list of paths or a single path normalize = utils.normalize_path if self.comma_separated_list: normalize = utils.normalize_paths - setattr(options, dest, normalize(old_value)) + return normalize(value) elif self.comma_separated_list: - old_value = getattr(options, dest) - setattr(options, dest, - utils.parse_comma_separated_list(old_value)) + return utils.parse_comma_separated_list(value) + return value def to_optparse(self): """Convert a Flake8 Option to an optparse Option.""" @@ -181,6 +179,7 @@ class OptionManager(object): """Simple proxy to calling the OptionParser's parse_args method.""" options, xargs = self.parser.parse_args(args, values) for option in self.options: - option.normalize(options) + old_value = getattr(options, option.dest) + setattr(options, option.dest, option.normalize(old_value)) return options, xargs From 30f4ccda0e6d82a526fee883cc86aaefb5dd2b10 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 22:34:13 -0600 Subject: [PATCH 038/364] Add docstrings to config module --- flake8/options/config.py | 51 ++++++++++++++++++++++++++++++---------- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/flake8/options/config.py b/flake8/options/config.py index 9317e7a..666f9d9 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -1,3 +1,4 @@ +"""Config handling logic for Flake8.""" import logging import os.path import sys @@ -7,17 +8,26 @@ if sys.version_info < (3, 0): else: import configparser -from . import utils - LOG = logging.getLogger(__name__) -__all__('ConfigFileFinder', 'MergedConfigParser') +__all__ = ('ConfigFileFinder', 'MergedConfigParser') class ConfigFileFinder(object): + """Encapsulate the logic for finding and reading config files.""" + PROJECT_FILENAMES = ('setup.cfg', 'tox.ini') def __init__(self, program_name, args, extra_config_files): + """Initialize object to find config files. + + :param str program_name: + Name of the current program (e.g., flake8). + :param list args: + The extra arguments passed on the command-line. + :param list extra_config_files: + Extra configuration files specified by the user to read. + """ # The values of --append-config from the CLI self.extra_config_files = extra_config_files @@ -46,12 +56,14 @@ class ConfigFileFinder(object): return (config, found_files) def cli_config(self, files): + """Read and parse the config file specified on the command-line.""" config, found_files = self._read_config(files) if found_files: LOG.debug('Found cli configuration files: %s', found_files) return config def generate_possible_local_config_files(self): + """Find and generate all local config files.""" tail = self.tail parent = self.parent while tail: @@ -62,6 +74,7 @@ class ConfigFileFinder(object): (parent, tail) = os.path.split(parent) def local_config_files(self): + """Find all local config files.""" return [ filename for filename in self.generate_possible_local_config_files() @@ -69,17 +82,20 @@ class ConfigFileFinder(object): ] + self.extra_config_files def local_configs(self): + """Parse all local config files into one config object.""" config, found_files = self._read_config(self.local_config_files()) if found_files: LOG.debug('Found local configuration files: %s', found_files) return config def user_config_file(self): + """Find the user-level config file.""" if self.is_windows: return os.path.expanduser('~\\' + self.program_config) return os.path.join(self.xdg_home, self.program_name) def user_config(self): + """Parse the user config file into a config object.""" config, found_files = self._read_config(self.user_config_file()) if found_files: LOG.debug('Found user configuration files: %s', found_files) @@ -87,10 +103,26 @@ class ConfigFileFinder(object): class MergedConfigParser(object): + """Encapsulate merging different types of configuration files. + + This parses out the options registered that were specified in the + configuration files, handles extra configuration files, and returns + dictionaries with the parsed values. + """ + GETINT_METHODS = set(['int', 'count']) GETBOOL_METHODS = set(['store_true', 'store_false']) def __init__(self, option_manager, extra_config_files=None, args=None): + """Initialize the MergedConfigParser instance. + + :param flake8.option.manager.OptionManager option_manager: + Initialized OptionManager. + :param list extra_config_files: + List of extra config files to parse. + :params list args: + The extra parsed arguments from the command-line. + """ # Our instance of flake8.options.manager.OptionManager self.option_manager = option_manager # The prog value for the cli parser @@ -106,12 +138,7 @@ class MergedConfigParser(object): @staticmethod def _normalize_value(option, value): - if option.normalize_paths: - final_value = utils.normalize_paths(value) - elif option.comma_separated_list: - final_value = utils.parse_comma_separated_list(value) - else: - final_value = value + final_value = option.normalize(value) LOG.debug('%r has been normalized to %r for option "%s"', value, final_value, option.config_name) return final_value @@ -126,11 +153,11 @@ class MergedConfigParser(object): option = self.config_options[option_name] # Use the appropriate method to parse the config value - method = config.get + method = config_parser.get if option.type in self.GETINT_METHODS: - method = config.getint + method = config_parser.getint elif option.action in self.GETBOOL_METHODS: - method = config.getboolean + method = config_parser.getboolean value = method(self.program_name, option_name) LOG.debug('Option "%s" returned value: %r', option_name, value) From 8dd160c98efc8fcfef74514fabc506042f6881b2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 22:34:39 -0600 Subject: [PATCH 039/364] Start writing tests for the config finder --- tests/unit/test_config_file_finder.py | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 tests/unit/test_config_file_finder.py diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py new file mode 100644 index 0000000..2df7694 --- /dev/null +++ b/tests/unit/test_config_file_finder.py @@ -0,0 +1,11 @@ +"""Tests for the ConfigFileFinder.""" +import os + +from flake8.options import config + + +def test_uses_default_args(): + """Show that we default the args value.""" + finder = config.ConfigFileFinder('flake8', None, []) + assert finder.args == ['.'] + assert finder.parent == os.path.abspath('.') From dd46f02b5863fb0850d3421b8cd698d1d3ecdfba Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 9 Jan 2016 23:02:20 -0600 Subject: [PATCH 040/364] Add config file fixtures Add more ConfigFileFinder tests --- tests/fixtures/config_files/cli-specified.ini | 9 +++++++++ tests/unit/test_config_file_finder.py | 19 ++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) create mode 100644 tests/fixtures/config_files/cli-specified.ini diff --git a/tests/fixtures/config_files/cli-specified.ini b/tests/fixtures/config_files/cli-specified.ini new file mode 100644 index 0000000..753604a --- /dev/null +++ b/tests/fixtures/config_files/cli-specified.ini @@ -0,0 +1,9 @@ +[flake8] +ignore = + E123, + W234, + E111 +exclude = + foo/, + bar/, + bogus/ diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 2df7694..8ed940d 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -1,5 +1,8 @@ """Tests for the ConfigFileFinder.""" import os +import sys + +import mock from flake8.options import config @@ -7,5 +10,19 @@ from flake8.options import config def test_uses_default_args(): """Show that we default the args value.""" finder = config.ConfigFileFinder('flake8', None, []) - assert finder.args == ['.'] assert finder.parent == os.path.abspath('.') + + +@mock.patch.object(sys, 'platform', 'win32') +def test_windows_detection(): + """Verify we detect Windows to the best of our knowledge.""" + finder = config.ConfigFileFinder('flake8', None, []) + assert finder.is_windows is True + + +def test_cli_config(): + cli_filepath = 'tests/fixtures/config_files/cli-specified.ini' + finder = config.ConfigFileFinder('flake8', None, []) + + parsed_config = finder.cli_config(cli_filepath) + assert parsed_config.has_section('flake8') From 949d3e48fe094d8c5629f7e9f15e1b6503ee9d49 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 08:40:35 -0600 Subject: [PATCH 041/364] Add tests for generate_possible_local_config_files Fix a bug found via testing where we traverse all the way to the root directory looking for config files. --- flake8/options/config.py | 5 ++++- tests/unit/test_config_file_finder.py | 23 +++++++++++++++++++++++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/flake8/options/config.py b/flake8/options/config.py index 666f9d9..e82cb62 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -43,7 +43,7 @@ class ConfigFileFinder(object): # List of filenames to find in the local/project directory self.project_filenames = ('setup.cfg', 'tox.ini', self.program_config) - self.local_directory = os.curdir + self.local_directory = os.path.abspath(os.curdir) if not args: args = ['.'] @@ -66,11 +66,14 @@ class ConfigFileFinder(object): """Find and generate all local config files.""" tail = self.tail parent = self.parent + local_dir = self.local_directory while tail: for project_filename in self.project_filenames: filename = os.path.abspath(os.path.join(parent, project_filename)) yield filename + if parent == local_dir: + break (parent, tail) = os.path.split(parent) def local_config_files(self): diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 8ed940d..8904316 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -3,6 +3,7 @@ import os import sys import mock +import pytest from flake8.options import config @@ -21,8 +22,30 @@ def test_windows_detection(): def test_cli_config(): + """Verify opening and reading the file specified via the cli.""" cli_filepath = 'tests/fixtures/config_files/cli-specified.ini' finder = config.ConfigFileFinder('flake8', None, []) parsed_config = finder.cli_config(cli_filepath) assert parsed_config.has_section('flake8') + + +@pytest.mark.parametrize('args,expected', [ + ([], # No arguments + [os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini'), + os.path.abspath('.flake8')]), + (['flake8/options', 'flake8/'], # Common prefix of "flake8/" + [os.path.abspath('flake8/setup.cfg'), + os.path.abspath('flake8/tox.ini'), + os.path.abspath('flake8/.flake8'), + os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini'), + os.path.abspath('.flake8')]), +]) +def test_generate_possible_local_config_files(args, expected): + """Verify generation of all possible config paths.""" + finder = config.ConfigFileFinder('flake8', args, []) + + assert (list(finder.generate_possible_local_config_files()) == + expected) From cb276d63e37b16a07c6f5e4897bd6c3bc8add445 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:04:39 -0600 Subject: [PATCH 042/364] Parametrize windows detection unit test --- tests/unit/test_config_file_finder.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 8904316..3a8f8f2 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -14,11 +14,16 @@ def test_uses_default_args(): assert finder.parent == os.path.abspath('.') -@mock.patch.object(sys, 'platform', 'win32') -def test_windows_detection(): +@pytest.mark.parametrize('platform,is_windows', [ + ('win32', True), + ('linux', False), + ('darwin', False), +]) +def test_windows_detection(platform, is_windows): """Verify we detect Windows to the best of our knowledge.""" - finder = config.ConfigFileFinder('flake8', None, []) - assert finder.is_windows is True + with mock.patch.object(sys, 'platform', platform): + finder = config.ConfigFileFinder('flake8', None, []) + assert finder.is_windows is is_windows def test_cli_config(): From 204a367095a6d63bcf6dcc690cd51d2db5fe8ff9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:05:09 -0600 Subject: [PATCH 043/364] Add more parameters to file location generation test --- tests/unit/test_config_file_finder.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 3a8f8f2..e723397 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -36,17 +36,30 @@ def test_cli_config(): @pytest.mark.parametrize('args,expected', [ - ([], # No arguments + # No arguments, common prefix of abspath('.') + ([], [os.path.abspath('setup.cfg'), os.path.abspath('tox.ini'), os.path.abspath('.flake8')]), - (['flake8/options', 'flake8/'], # Common prefix of "flake8/" + # Common prefix of "flake8/" + (['flake8/options', 'flake8/'], [os.path.abspath('flake8/setup.cfg'), os.path.abspath('flake8/tox.ini'), os.path.abspath('flake8/.flake8'), os.path.abspath('setup.cfg'), os.path.abspath('tox.ini'), os.path.abspath('.flake8')]), + # Common prefix of "flake8/options" + (['flake8/options', 'flake8/options/sub'], + [os.path.abspath('flake8/options/setup.cfg'), + os.path.abspath('flake8/options/tox.ini'), + os.path.abspath('flake8/options/.flake8'), + os.path.abspath('flake8/setup.cfg'), + os.path.abspath('flake8/tox.ini'), + os.path.abspath('flake8/.flake8'), + os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini'), + os.path.abspath('.flake8')]), ]) def test_generate_possible_local_config_files(args, expected): """Verify generation of all possible config paths.""" From fdff0f705e2dd2dbdb9427d277b181068644d2f6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:05:21 -0600 Subject: [PATCH 044/364] Allow users to pass args to py.test via tox --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index ecaae55..650c16e 100644 --- a/tox.ini +++ b/tox.ini @@ -6,4 +6,4 @@ deps = mock pytest commands = - py.test + py.test {posargs} From b8dfc9a8e2aef52a52ac753d617f2e44c76a49d5 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:39:37 -0600 Subject: [PATCH 045/364] Start testing ConfigFileFinder.local_config_files Make sure extra_config_files are all absolute paths and filter them in local_config_files to ensure they exist. --- flake8/options/config.py | 21 ++++++++++++++++++--- tests/unit/test_config_file_finder.py | 24 ++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/flake8/options/config.py b/flake8/options/config.py index e82cb62..d4f64d8 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -29,7 +29,11 @@ class ConfigFileFinder(object): Extra configuration files specified by the user to read. """ # The values of --append-config from the CLI - self.extra_config_files = extra_config_files + extra_config_files = extra_config_files or [] + self.extra_config_files = [ + # Ensure the paths are absolute paths for local_config_files + os.path.abspath(f) for f in extra_config_files + ] # Platform specific settings self.is_windows = sys.platform == 'win32' @@ -77,12 +81,23 @@ class ConfigFileFinder(object): (parent, tail) = os.path.split(parent) def local_config_files(self): - """Find all local config files.""" + """Find all local config files which actually exist. + + Filter results from + :meth:`~ConfigFileFinder.generate_possible_local_config_files` based + on whether the filename exists or not. + + :returns: + List of files that exist that are local project config files with + extra config files appended to that list (which also exist). + :rtype: + [str] + """ return [ filename for filename in self.generate_possible_local_config_files() if os.path.exists(filename) - ] + self.extra_config_files + ] + list(filter(os.path.exists, self.extra_config_files)) def local_configs(self): """Parse all local config files into one config object.""" diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index e723397..4b73979 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -67,3 +67,27 @@ def test_generate_possible_local_config_files(args, expected): assert (list(finder.generate_possible_local_config_files()) == expected) + + +@pytest.mark.parametrize('args,extra_config_files,expected', [ + # No arguments, common prefix of abspath('.') + ([], + [], + [os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini')]), + # Common prefix of "flake8/" + (['flake8/options', 'flake8/'], + [], + [os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini')]), + # Common prefix of "flake8/options" + (['flake8/options', 'flake8/options/sub'], + [], + [os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini')]), +]) +def test_local_config_files(args, extra_config_files, expected): + """Verify discovery of local config files.""" + finder = config.ConfigFileFinder('flake8', args, extra_config_files) + + assert list(finder.local_config_files()) == expected From 91d20207fbb44993b5baf763e1fb9202ea9a1a46 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:40:09 -0600 Subject: [PATCH 046/364] Add setup.cfg for testing purposes --- setup.cfg | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 setup.cfg diff --git a/setup.cfg b/setup.cfg new file mode 100644 index 0000000..e69de29 From 7269d84845f9aa4896a418d6d594682ab0b6d074 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:45:37 -0600 Subject: [PATCH 047/364] Add test parameters with extra_config_files --- tests/unit/test_config_file_finder.py | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 4b73979..35fa4a6 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -7,6 +7,8 @@ import pytest from flake8.options import config +CLI_SPECIFIED_FILEPATH = 'tests/fixtures/config_files/cli-specified.ini' + def test_uses_default_args(): """Show that we default the args value.""" @@ -28,7 +30,7 @@ def test_windows_detection(platform, is_windows): def test_cli_config(): """Verify opening and reading the file specified via the cli.""" - cli_filepath = 'tests/fixtures/config_files/cli-specified.ini' + cli_filepath = CLI_SPECIFIED_FILEPATH finder = config.ConfigFileFinder('flake8', None, []) parsed_config = finder.cli_config(cli_filepath) @@ -85,6 +87,12 @@ def test_generate_possible_local_config_files(args, expected): [], [os.path.abspath('setup.cfg'), os.path.abspath('tox.ini')]), + # Common prefix of "flake8/" with extra config files specified + (['flake8/'], + [CLI_SPECIFIED_FILEPATH], + [os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini'), + os.path.abspath(CLI_SPECIFIED_FILEPATH)]), ]) def test_local_config_files(args, extra_config_files, expected): """Verify discovery of local config files.""" From 9231aae1d6ecfca856e97fa4279cf06e83f42e57 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:50:16 -0600 Subject: [PATCH 048/364] Add README to fixtures directory --- tests/fixtures/config_files/README.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 tests/fixtures/config_files/README.rst diff --git a/tests/fixtures/config_files/README.rst b/tests/fixtures/config_files/README.rst new file mode 100644 index 0000000..b69e080 --- /dev/null +++ b/tests/fixtures/config_files/README.rst @@ -0,0 +1,22 @@ +About this directory +==================== + +The files in this directory are test fixtures for unit and integration tests. +Their purpose is described below. Please note the list of file names that can +not be created as they are already used by tests. + +New fixtures are preferred over updating existing features unless existing +tests will fail. + +Files that should not be created +-------------------------------- + +- ``tests/fixtures/config_files/missing.ini`` + +Purposes of existing fixtures +----------------------------- + +``tests/fixtures/config_files/cli-specified.ini`` + + This should only be used when providing config file(s) specified by the + user on the command-line. From de96b24bad8a1b099749a8b8473de9e885b4718e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 09:50:32 -0600 Subject: [PATCH 049/364] Add parameter for missing extra config file --- tests/unit/test_config_file_finder.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 35fa4a6..948e741 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -93,6 +93,13 @@ def test_generate_possible_local_config_files(args, expected): [os.path.abspath('setup.cfg'), os.path.abspath('tox.ini'), os.path.abspath(CLI_SPECIFIED_FILEPATH)]), + # Common prefix of "flake8/" with missing extra config files specified + (['flake8/'], + [CLI_SPECIFIED_FILEPATH, + 'tests/fixtures/config_files/missing.ini'], + [os.path.abspath('setup.cfg'), + os.path.abspath('tox.ini'), + os.path.abspath(CLI_SPECIFIED_FILEPATH)]), ]) def test_local_config_files(args, extra_config_files, expected): """Verify discovery of local config files.""" From 6553198074593841d720c9978103fb0d5f6e6915 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 12:01:22 -0600 Subject: [PATCH 050/364] Update setup.py --- setup.cfg | 5 +++++ setup.py | 20 ++++++++------------ 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/setup.cfg b/setup.cfg index e69de29..f05f933 100644 --- a/setup.cfg +++ b/setup.cfg @@ -0,0 +1,5 @@ +[aliases] +test=pytest + +[bdist_wheel] +universal=1 diff --git a/setup.py b/setup.py index d0650f4..92ea08f 100644 --- a/setup.py +++ b/setup.py @@ -1,6 +1,9 @@ # -*- coding: utf-8 -*- from __future__ import with_statement -from setuptools import setup +import setuptools + +import flake8 + try: # Work around a traceback with Nose on Python 2.6 # http://bugs.python.org/issue15881#msg170215 @@ -16,18 +19,11 @@ except ImportError: mock = None -tests_require = ['nose'] +tests_require = ['pytest'] if mock is None: tests_require += ['mock'] -def get_version(fname='flake8/__init__.py'): - with open(fname) as f: - for line in f: - if line.startswith('__version__'): - return eval(line.split('=')[-1]) - - def get_long_description(): descr = [] for fname in ('README.rst', 'CHANGES.rst'): @@ -36,10 +32,10 @@ def get_long_description(): return '\n\n'.join(descr) -setup( +setuptools.setup( name="flake8", license="MIT", - version=get_version(), + version=flake8.__version__, description="the modular source code checker: pep8, pyflakes and co", # long_description=get_long_description(), author="Tarek Ziade", @@ -71,5 +67,5 @@ setup( "Topic :: Software Development :: Quality Assurance", ], tests_require=tests_require, - test_suite='nose.collector', + setup_requires=['pytest-runner'], ) From 8b85e93e21fc44fcc1501b799141d16c4201381f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 12:52:29 -0600 Subject: [PATCH 051/364] Add test for local_configs method --- tests/unit/test_config_file_finder.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 948e741..200f4e5 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -1,4 +1,9 @@ """Tests for the ConfigFileFinder.""" +try: + import ConfigParser as configparser +except ImportError: + import configparser + import os import sys @@ -106,3 +111,10 @@ def test_local_config_files(args, extra_config_files, expected): finder = config.ConfigFileFinder('flake8', args, extra_config_files) assert list(finder.local_config_files()) == expected + + +def test_local_configs(): + """Verify we return a ConfigParser.""" + finder = config.ConfigFileFinder('flake8', None, []) + + assert isinstance(finder.local_configs(), configparser.RawConfigParser) From 66da9160b4f0ff730a6146f9504c4d947be3fab2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 13:05:35 -0600 Subject: [PATCH 052/364] Start testing MergedConfigParser Rename the GETINT/GETBOOL sets on the MergedConfigParser to use better names. --- flake8/options/config.py | 28 ++++++++++------- tests/unit/test_merged_config_parser.py | 41 +++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 11 deletions(-) create mode 100644 tests/unit/test_merged_config_parser.py diff --git a/flake8/options/config.py b/flake8/options/config.py index d4f64d8..558ec93 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -128,8 +128,12 @@ class MergedConfigParser(object): dictionaries with the parsed values. """ - GETINT_METHODS = set(['int', 'count']) - GETBOOL_METHODS = set(['store_true', 'store_false']) + #: Set of types that should use the + #: :meth:`~configparser.RawConfigParser.getint` method. + GETINT_TYPES = set(['int', 'count']) + #: Set of actions that should use the + #: :meth:`~configparser.RawConfigParser.getbool` method. + GETBOOL_ACTIONS = set(['store_true', 'store_false']) def __init__(self, option_manager, extra_config_files=None, args=None): """Initialize the MergedConfigParser instance. @@ -141,18 +145,20 @@ class MergedConfigParser(object): :params list args: The extra parsed arguments from the command-line. """ - # Our instance of flake8.options.manager.OptionManager + #: Our instance of flake8.options.manager.OptionManager self.option_manager = option_manager - # The prog value for the cli parser + #: The prog value for the cli parser self.program_name = option_manager.program_name - # Parsed extra arguments + #: Parsed extra arguments self.args = args - # Our instance of our ConfigFileFinder - self.config_finder = ConfigFileFinder(self.program_name, self.args) - # Mapping of configuration option names to - # flake8.options.manager.Option instances + #: Mapping of configuration option names to + #: :class:`~flake8.options.manager.Option` instances self.config_options = option_manager.config_options_dict + #: List of extra config files self.extra_config_files = extra_config_files or [] + #: Our instance of our :class:`~ConfigFileFinder` + self.config_finder = ConfigFileFinder(self.program_name, self.args, + self.extra_config_files) @staticmethod def _normalize_value(option, value): @@ -172,9 +178,9 @@ class MergedConfigParser(object): # Use the appropriate method to parse the config value method = config_parser.get - if option.type in self.GETINT_METHODS: + if option.type in self.GETINT_TYPES: method = config_parser.getint - elif option.action in self.GETBOOL_METHODS: + elif option.action in self.GETBOOL_ACTIONS: method = config_parser.getboolean value = method(self.program_name, option_name) diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py new file mode 100644 index 0000000..2e79b4a --- /dev/null +++ b/tests/unit/test_merged_config_parser.py @@ -0,0 +1,41 @@ +"""Unit tests for flake8.options.config.MergedConfigParser.""" +import mock +import pytest + +from flake8.options import config +from flake8.options import manager + + +@pytest.fixture +def optmanager(): + return manager.OptionManager(prog='flake8', version='3.0.0a1') + + +@pytest.mark.parametrize('args,extra_config_files', [ + (None, None), + (None, []), + (None, ['foo.ini']), + ('flake8/', []), + ('flake8/', ['foo.ini']), +]) +def test_creates_its_own_config_file_finder(args, extra_config_files, + optmanager): + """Verify we create a ConfigFileFinder correctly.""" + class_path = 'flake8.options.config.ConfigFileFinder' + with mock.patch(class_path) as ConfigFileFinder: + parser = config.MergedConfigParser( + option_manager=optmanager, + extra_config_files=extra_config_files, + args=args, + ) + + assert parser.program_name == 'flake8' + ConfigFileFinder.assert_called_once_with( + 'flake8', + args, + extra_config_files or [], + ) + + +def test_parse_cli_config(optmanager): + pass From c33b9d24b102b1dd845886e3434bd079b1d58cb4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 13:13:08 -0600 Subject: [PATCH 053/364] Add top-level function to configure logging --- flake8/__init__.py | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index 49cbabd..6f33c82 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -1,4 +1,5 @@ import logging +import sys try: from logging import NullHandler @@ -7,6 +8,48 @@ except ImportError: def emit(self, record): pass -logging.getLogger(__name__).addHandler(NullHandler()) +LOG = logging.getLogger(__name__) +LOG.addHandler(NullHandler()) + +# Clean up after LOG config +del NullHandler __version__ = '3.0.0a1' + + +_VERBOSITY_TO_LOG_LEVEL = { + # output more than warnings but not debugging info + 1: logging.INFO, + # output debugging information and everything else + 2: logging.DEBUG, +} + + +def configure_logging(verbosity, filename=None): + """Configure logging for flake8. + + :param int verbosity: + How verbose to be in logging information. + :param str filename: + Name of the file to append log information to. + If ``None`` this will log to ``sys.stderr``. + If the name is "stdout" or "stderr" this will log to the appropriate + stream. + """ + global LOG + if verbosity <= 0: + return + if verbosity > 2: + verbosity = 2 + + log_level = _VERBOSITY_TO_LOG_LEVEL[verbosity] + + if not filename or filename in ('stderr', 'stdout'): + handler = logging.StreamHandler(getattr(sys, filename)) + else: + handler = logging.FileHandler(filename) + + LOG.addHandler(handler) + LOG.setLevel(log_level) + LOG.debug('Added a %s logging handler to logger root at %s', + filename, __name__) From 16669a6c4567d870121217407e20ae9b7d5d326b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 13:13:33 -0600 Subject: [PATCH 054/364] Add test configuration --- tests/conftest.py | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 tests/conftest.py diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..7fa72ff --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,5 @@ +import sys + +import flake8 + +flake8.configure_logging(2, 'test-logs-%s.%s.log' % sys.version_info[0:2]) From 8b35e29f4ff327b42ea990540e4b4ff75fc4133a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 13:14:09 -0600 Subject: [PATCH 055/364] Ignore .eggs/ directory --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index a41da07..dfc3773 100644 --- a/.gitignore +++ b/.gitignore @@ -3,3 +3,5 @@ *.pyc *.sw* *.egg-info + +.eggs/ From 4d13923bfbe4b1c6dae24693199af73d084420b2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 13:15:14 -0600 Subject: [PATCH 056/364] Ignore test log files --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index dfc3773..612baf8 100644 --- a/.gitignore +++ b/.gitignore @@ -3,5 +3,5 @@ *.pyc *.sw* *.egg-info - +*.log .eggs/ From 03fd22c23073cf2faffb3d9577c2ef134acf5e4d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 13:15:25 -0600 Subject: [PATCH 057/364] Add a docstring to conftest --- tests/conftest.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/conftest.py b/tests/conftest.py index 7fa72ff..9bf4f95 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,3 +1,4 @@ +"""Test configuration for py.test.""" import sys import flake8 From a8576aff1239c3eb3e4d3e3ab28e82d045665e8e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 13:22:18 -0600 Subject: [PATCH 058/364] Fix bugs found while writing parse_cli_config test --- flake8/options/config.py | 4 +++- flake8/options/manager.py | 3 ++- tests/unit/test_merged_config_parser.py | 21 ++++++++++++++++++++- 3 files changed, 25 insertions(+), 3 deletions(-) diff --git a/flake8/options/config.py b/flake8/options/config.py index 558ec93..7acddd5 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -186,9 +186,11 @@ class MergedConfigParser(object): value = method(self.program_name, option_name) LOG.debug('Option "%s" returned value: %r', option_name, value) - final_value = self._normalize_value(value) + final_value = self._normalize_value(option, value) config_dict[option_name] = final_value + return config_dict + def is_configured_by(self, config): """Check if the specified config parser has an appropriate section.""" return config.has_section(self.program_name) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index fafc1e9..c4da8e0 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -104,7 +104,8 @@ class Option(object): 'dest={dest}, type={type}, callback={callback}, help={help},' ' callback={callback}, callback_args={callback_args}, ' 'callback_kwargs={callback_kwargs}, metavar={metavar})' - ).format(*self.option_args, **self.option_kwargs) + ).format(self.short_option_name, self.long_option_name, + **self.option_kwargs) def _make_dest(self, dest): if dest: diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index 2e79b4a..028ad18 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -1,4 +1,6 @@ """Unit tests for flake8.options.config.MergedConfigParser.""" +import os + import mock import pytest @@ -38,4 +40,21 @@ def test_creates_its_own_config_file_finder(args, extra_config_files, def test_parse_cli_config(optmanager): - pass + optmanager.add_option('--exclude', parse_from_config=True, + comma_separated_list=True, + normalize_paths=True) + optmanager.add_option('--ignore', parse_from_config=True, + comma_separated_list=True) + parser = config.MergedConfigParser(optmanager) + + parsed_config = parser.parse_cli_config( + 'tests/fixtures/config_files/cli-specified.ini' + ) + assert parsed_config == { + 'ignore': ['E123', 'W234', 'E111'], + 'exclude': [ + os.path.abspath('foo/'), + os.path.abspath('bar/'), + os.path.abspath('bogus/'), + ] + } From 59035767329af1a6e29b9b16f172064102d4bdee Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 14:36:52 -0600 Subject: [PATCH 059/364] Ignore only D203 and fix Flake8 errors --- flake8/__init__.py | 14 ++++++++++++++ flake8/_trie.py | 2 +- flake8/notifier.py | 3 +++ flake8/options/__init__.py | 12 ++++++++++++ flake8/options/manager.py | 4 ++-- flake8/style_guide.py | 1 + tests/unit/test_merged_config_parser.py | 1 + tox.ini | 3 +++ 8 files changed, 37 insertions(+), 3 deletions(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index 6f33c82..50dc71c 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -1,3 +1,14 @@ +"""Top-level module for Flake8. + +This module + +- initializes logging for the command-line tool +- tracks the version of the package +- provides a way to configure logging for the command-line tool + +.. autofunction:: flake8.configure_logging + +""" import logging import sys @@ -5,7 +16,10 @@ try: from logging import NullHandler except ImportError: class NullHandler(logging.Handler): + """Shim for version of Python < 2.7.""" + def emit(self, record): + """Do nothing.""" pass LOG = logging.getLogger(__name__) diff --git a/flake8/_trie.py b/flake8/_trie.py index 858c067..4871abb 100644 --- a/flake8/_trie.py +++ b/flake8/_trie.py @@ -5,7 +5,7 @@ __all__ = ('Trie', 'TrieNode') def _iterate_stringlike_objects(string): for i in range(len(string)): - yield string[i:i+1] + yield string[i:i + 1] class Trie(object): diff --git a/flake8/notifier.py b/flake8/notifier.py index 6bf51ef..e7bf3ba 100644 --- a/flake8/notifier.py +++ b/flake8/notifier.py @@ -1,7 +1,10 @@ +"""Implementation of the class that registers and notifies listeners.""" from flake8 import _trie class Notifier(object): + """Object that tracks and notifies listener objects.""" + def __init__(self): """Initialize an empty notifier object.""" self.listeners = _trie.Trie() diff --git a/flake8/options/__init__.py b/flake8/options/__init__.py index e69de29..cc20daa 100644 --- a/flake8/options/__init__.py +++ b/flake8/options/__init__.py @@ -0,0 +1,12 @@ +"""Package containing the option manager and config management logic. + +- :mod:`flake8.options.config` contains the logic for finding, parsing, and + merging configuration files. + +- :mod:`flake8.options.manager` contains the logic for managing customized + Flake8 command-line and configuration options. + +- :mod:`flake8.options.aggregator` uses objects from both of the above modules + to aggregate configuration into one object used by plugins and Flake8. + +""" diff --git a/flake8/options/manager.py b/flake8/options/manager.py index c4da8e0..d328a1f 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -104,8 +104,8 @@ class Option(object): 'dest={dest}, type={type}, callback={callback}, help={help},' ' callback={callback}, callback_args={callback_args}, ' 'callback_kwargs={callback_kwargs}, metavar={metavar})' - ).format(self.short_option_name, self.long_option_name, - **self.option_kwargs) + ).format(self.short_option_name, self.long_option_name, + **self.option_kwargs) def _make_dest(self, dest): if dest: diff --git a/flake8/style_guide.py b/flake8/style_guide.py index e69de29..a3f63eb 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -0,0 +1 @@ +"""Implementation of the StyleGuide used by Flake8.""" diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index 028ad18..96f6aae 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -40,6 +40,7 @@ def test_creates_its_own_config_file_finder(args, extra_config_files, def test_parse_cli_config(optmanager): + """Parse the specified config file as a cli config file.""" optmanager.add_option('--exclude', parse_from_config=True, comma_separated_list=True, normalize_paths=True) diff --git a/tox.ini b/tox.ini index 650c16e..8d9d425 100644 --- a/tox.ini +++ b/tox.ini @@ -7,3 +7,6 @@ deps = pytest commands = py.test {posargs} + +[flake8] +ignore = D203 From 8c872c4bb99c13d82b7b82db037eb3cb29a243e4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 14:47:06 -0600 Subject: [PATCH 060/364] Add more docstrings and fix lint errors --- setup.py | 2 ++ tests/unit/__init__.py | 0 tests/unit/test_merged_config_parser.py | 1 + tests/unit/test_notifier.py | 5 +++++ tests/unit/test_option.py | 1 + tests/unit/test_option_manager.py | 1 + tests/unit/test_trie.py | 1 + tests/unit/test_utils.py | 1 + tox.ini | 1 + 9 files changed, 13 insertions(+) delete mode 100644 tests/unit/__init__.py diff --git a/setup.py b/setup.py index 92ea08f..064b28e 100644 --- a/setup.py +++ b/setup.py @@ -1,3 +1,4 @@ +"""Packaging logic for Flake8.""" # -*- coding: utf-8 -*- from __future__ import with_statement import setuptools @@ -25,6 +26,7 @@ if mock is None: def get_long_description(): + """Generate a long description from the README and CHANGES files.""" descr = [] for fname in ('README.rst', 'CHANGES.rst'): with open(fname) as f: diff --git a/tests/unit/__init__.py b/tests/unit/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index 96f6aae..8287d1b 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -10,6 +10,7 @@ from flake8.options import manager @pytest.fixture def optmanager(): + """Generate an OptionManager with simple values.""" return manager.OptionManager(prog='flake8', version='3.0.0a1') diff --git a/tests/unit/test_notifier.py b/tests/unit/test_notifier.py index b0365f3..84b8a45 100644 --- a/tests/unit/test_notifier.py +++ b/tests/unit/test_notifier.py @@ -1,7 +1,9 @@ +"""Unit tests for the Notifier object.""" import pytest from flake8 import notifier + class _Listener(object): def __init__(self, error_code): self.error_code = error_code @@ -13,8 +15,11 @@ class _Listener(object): class TestNotifier(object): + """Notifier unit tests.""" + @pytest.fixture(autouse=True) def setup(self): + """Set up each TestNotifier instance.""" self.notifier = notifier.Notifier() self.listener_map = {} diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py index 55cae5c..5facf2a 100644 --- a/tests/unit/test_option.py +++ b/tests/unit/test_option.py @@ -60,5 +60,6 @@ def test_config_name_needs_long_option_name(): def test_dest_is_not_overridden(): + """Show that we do not override custom destinations.""" opt = manager.Option('-s', '--short', dest='something_not_short') assert opt.dest == 'something_not_short' diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index a9e641d..b0e4ee7 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -9,6 +9,7 @@ from flake8.options import manager @pytest.fixture def optmanager(): + """Generate a simple OptionManager with default test arguments.""" return manager.OptionManager(prog='flake8', version='3.0.0b1') diff --git a/tests/unit/test_trie.py b/tests/unit/test_trie.py index 0a5a0a9..4cbd8c5 100644 --- a/tests/unit/test_trie.py +++ b/tests/unit/test_trie.py @@ -1,3 +1,4 @@ +"""Unit test for the _trie module.""" from flake8 import _trie as trie diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index b0e243c..a6096d4 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -38,4 +38,5 @@ def test_normalize_path(value, expected): [os.path.abspath("../" + p) for p in RELATIVE_PATHS]), ]) def test_normalize_paths(value, expected): + """Verify we normalize comma-separated paths provided to the tool.""" assert utils.normalize_paths(value) == expected diff --git a/tox.ini b/tox.ini index 8d9d425..2267e15 100644 --- a/tox.ini +++ b/tox.ini @@ -9,4 +9,5 @@ commands = py.test {posargs} [flake8] +# Ignore some flake8-docstrings errors ignore = D203 From 629f796fc17820a28a5d08e9f039c636fa529a29 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 14:53:26 -0600 Subject: [PATCH 061/364] Add flake8 linting as a testenv --- tox.ini | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 2267e15..15218a1 100644 --- a/tox.ini +++ b/tox.ini @@ -1,5 +1,6 @@ [tox] -envlist = py26,py27,py32,py33,py34,py35 +minversion=2.3.1 +envlist = py26,py27,py32,py33,py34,py35,flake8 [testenv] deps = @@ -8,6 +9,16 @@ deps = commands = py.test {posargs} +[testenv:flake8] +skipsdist = true +skip_install = true +use_develop = false +deps = + flake8 + flake8-docstrings +commands = + flake8 + [flake8] # Ignore some flake8-docstrings errors ignore = D203 From 53d5f0e49af43a3598d261d9e9904aeb21f08c0e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 15:19:34 -0600 Subject: [PATCH 062/364] Add a pre-specified log format --- flake8/__init__.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/flake8/__init__.py b/flake8/__init__.py index 50dc71c..a0991cc 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -63,6 +63,9 @@ def configure_logging(verbosity, filename=None): else: handler = logging.FileHandler(filename) + handler.setFormatter( + logging.Formatter('%(asctime)s %(levelname)s %(message)s') + ) LOG.addHandler(handler) LOG.setLevel(log_level) LOG.debug('Added a %s logging handler to logger root at %s', From d04cb245d3d9a1f61a5a8cc60d1fbf742d7b46d3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 15:19:57 -0600 Subject: [PATCH 063/364] Move user and local config merging to own method --- flake8/options/config.py | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/flake8/options/config.py b/flake8/options/config.py index 7acddd5..2484cdd 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -228,6 +228,22 @@ class MergedConfigParser(object): LOG.debug('Parsing CLI configuration files.') return self._parse_config(config) + def merge_user_and_local_config(self): + """Merge the parsed user and local configuration files. + + :returns: + Dictionary of the parsed and merged configuration options. + :rtype: + dict + """ + user_config = self.parse_user_config() + config = self.parse_local_config() + + for option, value in user_config.items(): + config.setdefault(option, value) + + return config + def parse(self, cli_config=None, isolated=False): """Parse and return the local and user config files. @@ -257,10 +273,4 @@ class MergedConfigParser(object): '--config by the user', cli_config) return self.parse_cli_config(cli_config) - user_config = self.parse_user_config() - config = self.parse_local_config() - - for option, value in user_config.items(): - config.setdefault(option, value) - - return config + return self.merge_user_and_local_config() From 18bfa1f8a2b4232620200c738d6bbda7b17d0a6f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 15:20:08 -0600 Subject: [PATCH 064/364] Add test for is_configured_by --- tests/unit/test_merged_config_parser.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index 8287d1b..ba3100f 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -60,3 +60,15 @@ def test_parse_cli_config(optmanager): os.path.abspath('bogus/'), ] } + + +@pytest.mark.parametrize('filename,is_configured_by', [ + ('tests/fixtures/config_files/cli-specified.ini', True), + ('tests/fixtures/config_files/no-flake8-section.ini', False), +]) +def test_is_configured_by(filename, is_configured_by, optmanager): + """Verify the behaviour of the is_configured_by method.""" + parsed_config, _ = config.ConfigFileFinder._read_config(filename) + parser = config.MergedConfigParser(optmanager) + + assert parser.is_configured_by(parsed_config) is is_configured_by From ebe47ec7407e309f5b39ed6d0560aaac8b0b2853 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 21:05:51 -0600 Subject: [PATCH 065/364] Add fixture file without a flake8 section --- .../config_files/no-flake8-section.ini | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 tests/fixtures/config_files/no-flake8-section.ini diff --git a/tests/fixtures/config_files/no-flake8-section.ini b/tests/fixtures/config_files/no-flake8-section.ini new file mode 100644 index 0000000..a85b709 --- /dev/null +++ b/tests/fixtures/config_files/no-flake8-section.ini @@ -0,0 +1,20 @@ +[tox] +minversion=2.3.1 +envlist = py26,py27,py32,py33,py34,py35,flake8 + +[testenv] +deps = + mock + pytest +commands = + py.test {posargs} + +[testenv:flake8] +skipsdist = true +skip_install = true +use_develop = false +deps = + flake8 + flake8-docstrings +commands = + flake8 From e5c3ae5cd217bb83545114a2bcbe35a0a336e39d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 21:43:00 -0600 Subject: [PATCH 066/364] Add tests for parse_{user,local}_config methods --- tests/unit/test_merged_config_parser.py | 49 +++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index ba3100f..d0f53fd 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -72,3 +72,52 @@ def test_is_configured_by(filename, is_configured_by, optmanager): parser = config.MergedConfigParser(optmanager) assert parser.is_configured_by(parsed_config) is is_configured_by + + +def test_parse_user_config(optmanager): + """Verify parsing of user config files.""" + optmanager.add_option('--exclude', parse_from_config=True, + comma_separated_list=True, + normalize_paths=True) + optmanager.add_option('--ignore', parse_from_config=True, + comma_separated_list=True) + parser = config.MergedConfigParser(optmanager) + + with mock.patch.object(parser.config_finder, 'user_config_file') as usercf: + usercf.return_value = 'tests/fixtures/config_files/cli-specified.ini' + parsed_config = parser.parse_user_config() + + assert parsed_config == { + 'ignore': ['E123', 'W234', 'E111'], + 'exclude': [ + os.path.abspath('foo/'), + os.path.abspath('bar/'), + os.path.abspath('bogus/'), + ] + } + + +def test_parse_local_config(optmanager): + """Verify parsing of local config files.""" + optmanager.add_option('--exclude', parse_from_config=True, + comma_separated_list=True, + normalize_paths=True) + optmanager.add_option('--ignore', parse_from_config=True, + comma_separated_list=True) + parser = config.MergedConfigParser(optmanager) + config_finder = parser.config_finder + + with mock.patch.object(config_finder, 'local_config_files') as localcfs: + localcfs.return_value = [ + 'tests/fixtures/config_files/cli-specified.ini' + ] + parsed_config = parser.parse_local_config() + + assert parsed_config == { + 'ignore': ['E123', 'W234', 'E111'], + 'exclude': [ + os.path.abspath('foo/'), + os.path.abspath('bar/'), + os.path.abspath('bogus/'), + ] + } From cb9231eb9ba88a01ccbd47910d4617405ef5e2dc Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 10 Jan 2016 22:07:30 -0600 Subject: [PATCH 067/364] Add tests around merge_user_and_local_config and parse --- tests/unit/test_merged_config_parser.py | 50 +++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index d0f53fd..2c4c5fd 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -121,3 +121,53 @@ def test_parse_local_config(optmanager): os.path.abspath('bogus/'), ] } + + +def test_merge_user_and_local_config(optmanager): + """Verify merging of parsed user and local config files.""" + optmanager.add_option('--exclude', parse_from_config=True, + comma_separated_list=True, + normalize_paths=True) + optmanager.add_option('--ignore', parse_from_config=True, + comma_separated_list=True) + optmanager.add_option('--select', parse_from_config=True, + comma_separated_list=True) + parser = config.MergedConfigParser(optmanager) + config_finder = parser.config_finder + + with mock.patch.object(config_finder, 'local_config_files') as localcfs: + localcfs.return_value = [ + 'tests/fixtures/config_files/local-config.ini' + ] + with mock.patch.object(config_finder, + 'user_config_file') as usercf: + usercf.return_value = ('tests/fixtures/config_files/' + 'user-config.ini') + parsed_config = parser.merge_user_and_local_config() + + assert parsed_config == { + 'exclude': [ + os.path.abspath('docs/') + ], + 'ignore': ['D203'], + 'select': ['E', 'W', 'F'], + } + + +@mock.patch('flake8.options.config.ConfigFileFinder') +def test_parse_isolates_config(ConfigFileManager, optmanager): + """Verify behaviour of the parse method with isolated=True.""" + parser = config.MergedConfigParser(optmanager) + + assert parser.parse(isolated=True) == {} + assert parser.config_finder.local_configs.called is False + assert parser.config_finder.user_config.called is False + + +@mock.patch('flake8.options.config.ConfigFileFinder') +def test_parse_uses_cli_config(ConfigFileManager, optmanager): + """Verify behaviour of the parse method with a specified config.""" + parser = config.MergedConfigParser(optmanager) + + parser.parse(cli_config='foo.ini') + parser.config_finder.cli_config.assert_called_once_with('foo.ini') From 6940ef3aafa20f72a7da616aafe05c1c7df93a9f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 11 Jan 2016 23:15:11 -0600 Subject: [PATCH 068/364] Make log format configurable --- flake8/__init__.py | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index a0991cc..a6520e4 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -39,7 +39,8 @@ _VERBOSITY_TO_LOG_LEVEL = { } -def configure_logging(verbosity, filename=None): +def configure_logging(verbosity, filename=None, + logformat='%(asctime)s %(levelname)s %(message)s'): """Configure logging for flake8. :param int verbosity: @@ -63,9 +64,7 @@ def configure_logging(verbosity, filename=None): else: handler = logging.FileHandler(filename) - handler.setFormatter( - logging.Formatter('%(asctime)s %(levelname)s %(message)s') - ) + handler.setFormatter(logging.Formatter(logformat)) LOG.addHandler(handler) LOG.setLevel(log_level) LOG.debug('Added a %s logging handler to logger root at %s', From 40189885852ca17a7bc3c05fc9540b4721551503 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 11 Jan 2016 23:15:23 -0600 Subject: [PATCH 069/364] Add a module with defaults --- flake8/defaults.py | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 flake8/defaults.py diff --git a/flake8/defaults.py b/flake8/defaults.py new file mode 100644 index 0000000..ac798b5 --- /dev/null +++ b/flake8/defaults.py @@ -0,0 +1,5 @@ +"""Constants that define defaults.""" + +EXCLUDE = '.svn,CVS,.bzr,.hg,.git,__pycache__,.tox' +IGNORE = 'E121,E123,E126,E226,E24,E704' +MAX_LINE_LENGTH = 79 From 8466d1462f5a3b3b3775f229b104d275e7584da9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 11 Jan 2016 23:15:53 -0600 Subject: [PATCH 070/364] Start bulding a bit of the cli for functional testing --- flake8/main/__init__.py | 0 flake8/main/cli.py | 122 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 122 insertions(+) create mode 100644 flake8/main/__init__.py create mode 100644 flake8/main/cli.py diff --git a/flake8/main/__init__.py b/flake8/main/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/flake8/main/cli.py b/flake8/main/cli.py new file mode 100644 index 0000000..5bd7c84 --- /dev/null +++ b/flake8/main/cli.py @@ -0,0 +1,122 @@ +"""Command-line implementation of flake8.""" +from flake8 import defaults + + +def register_default_options(option_manager): + """Register the default options on our OptionManager.""" + add_option = option_manager.add_option + + # pep8 options + add_option( + '-v', '--verbose', default=0, action='count', + parse_from_config=True, + help='Print more information about what is happening in flake8.' + ' This option is repeatable and will increase verbosity each ' + 'time it is repeated.', + ) + add_option( + '-q', '--quiet', default=0, action='count', + parse_from_config=True, + help='Report only file names, or nothing. This option is repeatable.', + ) + + add_option( + '--count', action='store_true', parse_from_config=True, + help='Print total number of errors and warnings to standard error and' + ' set the exit code to 1 if total is not empty.', + ) + + add_option( + '--diff', action='store_true', + help='Report changes only within line number ranges in the unified ' + 'diff provided on standard in by the user.', + ) + + add_option( + '--exclude', metavar='patterns', default=defaults.EXCLUDE, + comma_separated_list=True, parse_from_config=True, + normalize_paths=True, + help='Comma-separated list of files or directories to exclude.' + '(Default: %default)', + ) + + add_option( + '--filename', metavar='patterns', default='*.py', + parse_from_config=True, comma_separated_list=True, + help='Only check for filenames matching the patterns in this comma-' + 'separated list. (Default: %default)', + ) + + # TODO(sigmavirus24): Figure out --first/--repeat + + add_option( + '--format', metavar='format', default='default', choices=['default'], + parse_from_config=True, + help='Format errors according to the chosen formatter.', + ) + + add_option( + '--hang-closing', action='store_true', parse_from_config=True, + help='Hang closing bracket instead of matching indentation of opening' + " bracket's line.", + ) + + add_option( + '--ignore', metavar='errors', default=defaults.IGNORE, + parse_from_config=True, comma_separated_list=True, + help='Comma-separated list of errors and warnings to ignore (or skip).' + ' For example, ``--ignore=E4,E51,W234``. (Default: %default)', + ) + + add_option( + '--max-line-length', type='int', metavar='n', + default=defaults.MAX_LINE_LENGTH, parse_from_config=True, + help='Maximum allowed line length for the entirety of this run. ' + '(Default: %default)', + ) + + add_option( + '--select', metavar='errors', default='', + parse_from_config=True, comma_separated_list=True, + help='Comma-separated list of errors and warnings to enable.' + ' For example, ``--select=E4,E51,W234``. (Default: %default)', + ) + + # TODO(sigmavirus24): Decide what to do about --show-pep8 + + add_option( + '--show-source', action='store_true', parse_from_config=True, + help='Show the source generate each error or warning.', + ) + + add_option( + '--statistics', action='store_true', parse_from_config=True, + help='Count errors and warnings.', + ) + + # Flake8 options + add_option( + '--enabled-extensions', default='', parse_from_config=True, + comma_separated_list=True, type='string', + help='Enable plugins and extensions that are otherwise disabled ' + 'by default', + ) + + add_option( + '--exit-zero', action='store_true', + help='Exit with status code "0" even if there are errors.', + ) + + add_option( + '-j', '--jobs', type='string', default='auto', parse_from_config=True, + help='Number of subprocesses to use to run checks in parallel. ' + 'This is ignored on Windows. The default, "auto", will ' + 'auto-detect the number of processors available to use.' + ' (Default: %default)', + ) + + add_option( + '--output-file', default=None, type='string', parse_from_config=True, + # callback=callbacks.redirect_stdout, + help='Redirect report to a file.', + ) From 18708111372a4cbc0cd961cd8a32fc66e83df781 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 14 Jan 2016 22:25:02 -0600 Subject: [PATCH 071/364] Start working on plugin management --- flake8/plugins/__init__.py | 0 flake8/plugins/manager.py | 199 +++++++++++++++++++++++++++++++++++++ 2 files changed, 199 insertions(+) create mode 100644 flake8/plugins/__init__.py create mode 100644 flake8/plugins/manager.py diff --git a/flake8/plugins/__init__.py b/flake8/plugins/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py new file mode 100644 index 0000000..0bc4a78 --- /dev/null +++ b/flake8/plugins/manager.py @@ -0,0 +1,199 @@ +"""Plugin loading and management logic and classes.""" +import logging + +import pkg_resources + +LOG = logging.getLogger(__name__) + + +class Plugin(object): + """Wrap an EntryPoint from setuptools and other logic.""" + + def __init__(self, name, entry_point): + """"Initialize our Plugin. + + :param str name: + Name of the entry-point as it was registered with setuptools. + :param entry_point: + EntryPoint returned by setuptools. + :type entry_point: + setuptools.EntryPoint + """ + self.name = name + self.entry_point = entry_point + self.plugin = None + + def __repr__(self): + return 'Plugin(name="{0}", entry_point="{1}")'.format( + self.name, self.entry_point + ) + + def execute(self, *args, **kwargs): + r"""Call the plugin with \*args and \*\*kwargs.""" + return self.plugin(*args, **kwargs) + + def load(self, verify_requirements=False): + """Retrieve the plugin for this entry-point. + + This loads the plugin, stores it on the instance and then returns it. + It does not reload it after the first time, it merely returns the + cached plugin. + + :param bool verify_requirements: + Whether or not to make setuptools verify that the requirements for + the plugin are satisfied. + :returns: + The plugin resolved from the entry-point. + """ + if self.plugin is None: + LOG.debug('Loading plugin "%s" from entry-point.', self.name) + # Avoid relying on hasattr() here. + resolve = getattr(self.entry_point, 'resolve', None) + require = getattr(self.entry_point, 'require', None) + if resolve and require: + if verify_requirements: + LOG.debug('Verifying plugin "%s"\'s requirements.', + self.name) + require() + self.plugin = resolve() + else: + self.plugin = self.entry_point.load( + require=verify_requirements + ) + + return self.plugin + + def provide_options(self, optmanager, options, extra_args): + """Pass the parsed options and extra arguments to the plugin.""" + plugin = self.load() + parse_options = getattr(plugin, 'parse_options', None) + if parse_options is not None: + LOG.debug('Providing options to plugin "%s".', self.name) + try: + parse_options(optmanager, options, extra_args) + except TypeError: + parse_options(options) + + def register_options(self, optmanager): + """Register the plugin's command-line options on the OptionManager. + + :param optmanager: + Instantiated OptionManager to register options on. + :type optmanager: + flake8.options.manager.OptionManager + :returns: + Nothing + """ + plugin = self.load() + add_options = getattr(plugin, 'add_options', None) + if add_options is not None: + LOG.debug( + 'Registering options from plugin "%s" on OptionManager %r', + self.name, optmanager + ) + plugin.add_options(optmanager) + + +class PluginManager(object): + """Find and manage plugins consistently.""" + + def __init__(self, namespace, verify_requirements=False): + """Initialize the manager. + + :param str namespace: + Namespace of the plugins to manage, e.g., 'flake8.extension'. + :param bool verify_requirements: + Whether or not to make setuptools verify that the requirements for + the plugin are satisfied. + """ + self.namespace = namespace + self.verify_requirements = verify_requirements + self.plugins = {} + self.names = [] + self._load_all_plugins() + + def __contains__(self, name): + LOG.debug('Checking for "%s" in plugin manager.', name) + return name in self.plugins + + def __getitem__(self, name): + LOG.debug('Retrieving plugin for "%s".', name) + return self.plugins[name] + + def _load_all_plugins(self): + LOG.debug('Loading entry-points for "%s".', self.namespace) + for entry_point in pkg_resources.iter_entry_points(self.namespace): + name = entry_point.name + self.plugins[name] = Plugin(name, entry_point) + self.names.append(name) + LOG.info('Loaded %r for plugin "%s".', self.plugins[name], name) + + def map(self, func, *args, **kwargs): + """Call ``func`` with the plugin and *args and **kwargs after. + + This yields the return value from ``func`` for each plugin. + + :param collections.Callable func: + Function to call with each plugin. Signature should at least be: + + .. code-block:: python + + def myfunc(plugin): + pass + + Any extra positional or keyword arguments specified with map will + be passed along to this function after the plugin. + :param args: + Positional arguments to pass to ``func`` after each plugin. + :param kwargs: + Keyword arguments to pass to ``func`` after each plugin. + """ + for name in self.names: + yield func(self.plugins[name], *args, **kwargs) + + +class Checkers(object): + """All of the checkers registered through entry-ponits.""" + + def __init__(self, namespace='flake8.extension'): + """Initialize the Checkers collection.""" + self.manager = PluginManager(namespace) + + @property + def names(self): + return self.manager.names + + def register_options(self, optmanager): + """Register all of the checkers' options to the OptionManager.""" + def call_register_options(plugin, optmanager): + return plugin.register_options(optmanager) + + list(self.map(call_register_options, optmanager)) + + def provide_options(self, optmanager, options, extra_args): + def call_provide_options(plugin, optmanager, options, extra_args): + return plugin.provide_options(optmanager, options, extra_args) + + list(self.map(call_provide_options, optmanager, options, extra_args)) + + +class Listeners(object): + """All of the listeners registered through entry-points.""" + + def __init__(self, namespace='flake8.listener'): + self.manager = PluginManager(namespace) + + @property + def names(self): + return self.manager.names + + +class ReportFormatters(object): + """All of the report formatters registered through entry-points.""" + + def __init__(self, namespace='flake8.report'): + self.manager = PluginManager(namespace) + + @property + def names(self): + return self.manager.names From f1010b77334dc8f260fd21383baaacaac74926d8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 15 Jan 2016 10:59:51 -0600 Subject: [PATCH 072/364] Make plugin and version loading lazy Abstract plugin type management into one class and provide specific classes for each of our known types of plugins --- flake8/plugins/manager.py | 101 +++++++++++++++++++++++++++----------- 1 file changed, 72 insertions(+), 29 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 0bc4a78..f18b666 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -1,4 +1,5 @@ """Plugin loading and management logic and classes.""" +import collections import logging import pkg_resources @@ -21,13 +22,22 @@ class Plugin(object): """ self.name = name self.entry_point = entry_point - self.plugin = None + self._plugin = None def __repr__(self): return 'Plugin(name="{0}", entry_point="{1}")'.format( self.name, self.entry_point ) + @property + def plugin(self): + self.load_plugin() + return self._plugin + + @property + def version(self): + return self.plugin.version + def execute(self, *args, **kwargs): r"""Call the plugin with \*args and \*\*kwargs.""" return self.plugin(*args, **kwargs) @@ -45,7 +55,22 @@ class Plugin(object): :returns: The plugin resolved from the entry-point. """ - if self.plugin is None: + return self.plugin + + def load_plugin(self, verify_requirements=False): + """Retrieve the plugin for this entry-point. + + This loads the plugin, stores it on the instance and then returns it. + It does not reload it after the first time, it merely returns the + cached plugin. + + :param bool verify_requirements: + Whether or not to make setuptools verify that the requirements for + the plugin are satisfied. + :returns: + Nothing + """ + if self._plugin is None: LOG.debug('Loading plugin "%s" from entry-point.', self.name) # Avoid relying on hasattr() here. resolve = getattr(self.entry_point, 'resolve', None) @@ -55,14 +80,12 @@ class Plugin(object): LOG.debug('Verifying plugin "%s"\'s requirements.', self.name) require() - self.plugin = resolve() + self._plugin = resolve() else: - self.plugin = self.entry_point.load( + self._plugin = self.entry_point.load( require=verify_requirements ) - return self.plugin - def provide_options(self, optmanager, options, extra_args): """Pass the parsed options and extra arguments to the plugin.""" plugin = self.load() @@ -152,48 +175,68 @@ class PluginManager(object): yield func(self.plugins[name], *args, **kwargs) -class Checkers(object): - """All of the checkers registered through entry-ponits.""" +class PluginTypeManager(object): + """Parent class for most of the specific plugin types.""" - def __init__(self, namespace='flake8.extension'): - """Initialize the Checkers collection.""" - self.manager = PluginManager(namespace) + def __init__(self): + """Initialize the plugin type's manager.""" + self.manager = PluginManager(self.namespace) @property def names(self): + """Proxy attribute to underlying manager.""" return self.manager.names + @property + def plugins(self): + """Proxy attribute to underlying manager.""" + return self.manager.plugins + + @staticmethod + def _generate_call_function(method_name, optmanager, *args, **kwargs): + def generated_function(plugin): + method = getattr(plugin, method_name, None) + if (method is not None and + isinstance(method, collections.Callable)): + return method(optmanager, *args, **kwargs) + + def load_plugins(self): + def load_plugin(plugin): + return plugin.load() + + return list(self.manager.map(load_plugin)) + def register_options(self, optmanager): """Register all of the checkers' options to the OptionManager.""" - def call_register_options(plugin, optmanager): - return plugin.register_options(optmanager) + call_register_options = self._generate_call_function( + 'register_options', optmanager, + ) - list(self.map(call_register_options, optmanager)) + list(self.manager.map(call_register_options, optmanager)) def provide_options(self, optmanager, options, extra_args): - def call_provide_options(plugin, optmanager, options, extra_args): - return plugin.provide_options(optmanager, options, extra_args) + """Provide parsed options and extra arguments to the plugins.""" + call_provide_options = self._generate_call_function( + 'provide_options', optmanager, options, extra_args, + ) - list(self.map(call_provide_options, optmanager, options, extra_args)) + list(self.manager.map(call_provide_options, optmanager, options, + extra_args)) + + +class Checkers(PluginTypeManager): + """All of the checkers registered through entry-ponits.""" + + namespace = 'flake8.extension' class Listeners(object): """All of the listeners registered through entry-points.""" - def __init__(self, namespace='flake8.listener'): - self.manager = PluginManager(namespace) - - @property - def names(self): - return self.manager.names + namespace = 'flake8.listen' class ReportFormatters(object): """All of the report formatters registered through entry-points.""" - def __init__(self, namespace='flake8.report'): - self.manager = PluginManager(namespace) - - @property - def names(self): - return self.manager.names + namespace = 'flake8.report' From d2f4e97c315fac01d2c91f073083e01dd122374a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 15 Jan 2016 21:39:33 -0600 Subject: [PATCH 073/364] Delete unnecessary and dead code --- flake8/plugins/manager.py | 23 +++-------------------- 1 file changed, 3 insertions(+), 20 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index f18b666..6e71f7e 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -42,21 +42,6 @@ class Plugin(object): r"""Call the plugin with \*args and \*\*kwargs.""" return self.plugin(*args, **kwargs) - def load(self, verify_requirements=False): - """Retrieve the plugin for this entry-point. - - This loads the plugin, stores it on the instance and then returns it. - It does not reload it after the first time, it merely returns the - cached plugin. - - :param bool verify_requirements: - Whether or not to make setuptools verify that the requirements for - the plugin are satisfied. - :returns: - The plugin resolved from the entry-point. - """ - return self.plugin - def load_plugin(self, verify_requirements=False): """Retrieve the plugin for this entry-point. @@ -88,8 +73,7 @@ class Plugin(object): def provide_options(self, optmanager, options, extra_args): """Pass the parsed options and extra arguments to the plugin.""" - plugin = self.load() - parse_options = getattr(plugin, 'parse_options', None) + parse_options = getattr(self.plugin, 'parse_options', None) if parse_options is not None: LOG.debug('Providing options to plugin "%s".', self.name) try: @@ -107,14 +91,13 @@ class Plugin(object): :returns: Nothing """ - plugin = self.load() - add_options = getattr(plugin, 'add_options', None) + add_options = getattr(self.plugin, 'add_options', None) if add_options is not None: LOG.debug( 'Registering options from plugin "%s" on OptionManager %r', self.name, optmanager ) - plugin.add_options(optmanager) + add_options(optmanager) class PluginManager(object): From 6ef9089eb7aa0b7b3f302e38d0e2819571da118a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 16 Jan 2016 07:02:36 -0600 Subject: [PATCH 074/364] Add main function entry-point to flake8 * Add --isolated option * Add --append-config * Add docstring to flake8.main --- flake8/main/__init__.py | 1 + flake8/main/cli.py | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+) diff --git a/flake8/main/__init__.py b/flake8/main/__init__.py index e69de29..d3aa1de 100644 --- a/flake8/main/__init__.py +++ b/flake8/main/__init__.py @@ -0,0 +1 @@ +"""Module containing the logic for the Flake8 entry-points.""" diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 5bd7c84..05af72e 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -1,5 +1,9 @@ """Command-line implementation of flake8.""" +import flake8 from flake8 import defaults +from flake8.options import aggregator +from flake8.options import manager +from flake8.plugins import manager as plugin_manager def register_default_options(option_manager): @@ -120,3 +124,39 @@ def register_default_options(option_manager): # callback=callbacks.redirect_stdout, help='Redirect report to a file.', ) + + # Config file options + + add_option( + '--isolated', default=False, action='store_true', + help='Ignore all found configuration files.', + ) + + add_option( + '--append-config', action='append', + help='Provide extra config files to parse in addition to the files ' + 'found by Flake8 by default. These files are the last ones read ' + 'and so they take the highest precedence when multiple files ' + 'provide the same option.', + ) + + +def main(argv=None): + """Main entry-point for the flake8 command-line tool.""" + option_manager = manager.OptionManager( + prog='flake8', version=flake8.__version__ + ) + # Load our plugins + check_plugins = plugin_manager.Checkers() + listening_plugins = plugin_manager.Listeners() + formatting_plugins = plugin_manager.ReportFormatters() + + # Register all command-line and config-file options + register_default_options(option_manager) + check_plugins.register_options(option_manager) + listening_plugins.register_options(option_manager) + formatting_plugins.register_options(option_manager) + + # Parse out our options from our found config files and user-provided CLI + # options + options, args = aggregator.aggregate_options(option_manager) From 45d470927c85a680fcc4a4f902fe7da7f6e7606d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 16 Jan 2016 07:03:37 -0600 Subject: [PATCH 075/364] Load plugins of each type idempotently * Do not pass too many arguments to provide_options or register_options * Sub-class PluginTypeManager for both Listeners and Formatters --- flake8/plugins/manager.py | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 6e71f7e..81abdc4 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -164,6 +164,7 @@ class PluginTypeManager(object): def __init__(self): """Initialize the plugin type's manager.""" self.manager = PluginManager(self.namespace) + self.plugins_loaded = False @property def names(self): @@ -182,20 +183,29 @@ class PluginTypeManager(object): if (method is not None and isinstance(method, collections.Callable)): return method(optmanager, *args, **kwargs) + return generated_function def load_plugins(self): - def load_plugin(plugin): - return plugin.load() + """Load all plugins of this type that are managed by this manager.""" + if self.plugins_loaded: + return - return list(self.manager.map(load_plugin)) + def load_plugin(plugin): + return plugin.load_plugin() + + plugins = list(self.manager.map(load_plugin)) + # Do not set plugins_loaded if we run into an exception + self.plugins_loaded = True + return plugins def register_options(self, optmanager): """Register all of the checkers' options to the OptionManager.""" + self.load_plugins() call_register_options = self._generate_call_function( 'register_options', optmanager, ) - list(self.manager.map(call_register_options, optmanager)) + list(self.manager.map(call_register_options)) def provide_options(self, optmanager, options, extra_args): """Provide parsed options and extra arguments to the plugins.""" @@ -203,8 +213,7 @@ class PluginTypeManager(object): 'provide_options', optmanager, options, extra_args, ) - list(self.manager.map(call_provide_options, optmanager, options, - extra_args)) + list(self.manager.map(call_provide_options)) class Checkers(PluginTypeManager): @@ -213,13 +222,13 @@ class Checkers(PluginTypeManager): namespace = 'flake8.extension' -class Listeners(object): +class Listeners(PluginTypeManager): """All of the listeners registered through entry-points.""" namespace = 'flake8.listen' -class ReportFormatters(object): +class ReportFormatters(PluginTypeManager): """All of the report formatters registered through entry-points.""" namespace = 'flake8.report' From 4d86e44992291487a5303d5d65d5e0c84fcf393a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 16 Jan 2016 07:04:28 -0600 Subject: [PATCH 076/364] Add pyflakes shim from previous flake8 Modify the shim plugin to work with new flake8 --- flake8/plugins/pyflakes.py | 130 +++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 flake8/plugins/pyflakes.py diff --git a/flake8/plugins/pyflakes.py b/flake8/plugins/pyflakes.py new file mode 100644 index 0000000..c801e54 --- /dev/null +++ b/flake8/plugins/pyflakes.py @@ -0,0 +1,130 @@ +# -*- coding: utf-8 -*- +from __future__ import absolute_import +try: + # The 'demandimport' breaks pyflakes and flake8.plugins.pyflakes + from mercurial import demandimport +except ImportError: + pass +else: + demandimport.disable() +import os + +import pyflakes +import pyflakes.checker + +from flake8 import utils + + +def patch_pyflakes(): + """Add error codes to Pyflakes messages.""" + codes = dict([line.split()[::-1] for line in ( + 'F401 UnusedImport', + 'F402 ImportShadowedByLoopVar', + 'F403 ImportStarUsed', + 'F404 LateFutureImport', + 'F810 Redefined', # XXX Obsolete? + 'F811 RedefinedWhileUnused', + 'F812 RedefinedInListComp', + 'F821 UndefinedName', + 'F822 UndefinedExport', + 'F823 UndefinedLocal', + 'F831 DuplicateArgument', + 'F841 UnusedVariable', + )]) + + for name, obj in vars(pyflakes.messages).items(): + if name[0].isupper() and obj.message: + obj.flake8_msg = '%s %s' % (codes.get(name, 'F999'), obj.message) +patch_pyflakes() + + +class FlakesChecker(pyflakes.checker.Checker): + """Subclass the Pyflakes checker to conform with the flake8 API.""" + name = 'pyflakes' + version = pyflakes.__version__ + + def __init__(self, tree, filename): + filename = utils.normalize_paths(filename)[0] + withDoctest = self.withDoctest + included_by = [include for include in self.include_in_doctest + if include != '' and filename.startswith(include)] + if included_by: + withDoctest = True + + for exclude in self.exclude_from_doctest: + if exclude != '' and filename.startswith(exclude): + withDoctest = False + overlaped_by = [include for include in included_by + if include.startswith(exclude)] + + if overlaped_by: + withDoctest = True + + super(FlakesChecker, self).__init__(tree, filename, + withDoctest=withDoctest) + + @classmethod + def add_options(cls, parser): + parser.add_option( + '--builtins', parse_from_config=True, comma_separated_list=True, + help="define more built-ins, comma separated", + ) + parser.add_option( + '--doctests', default=False, action='store_true', + parse_from_config=True, + help="check syntax of the doctests", + ) + parser.add_option( + '--include-in-doctest', default='', + dest='include_in_doctest', parse_from_config=True, + comma_separated_list=True, normalize_paths=True, + help='Run doctests only on these files', + type='string', + ) + parser.add_option( + '--exclude-from-doctest', default='', + dest='exclude_from_doctest', parse_from_config=True, + comma_separated_list=True, normalize_paths=True, + help='Skip these files when running doctests', + type='string', + ) + + @classmethod + def parse_options(cls, options): + if options.builtins: + cls.builtIns = cls.builtIns.union(options.builtins.split(',')) + cls.withDoctest = options.doctests + + included_files = [] + for included_file in options.include_in_doctest: + if included_file == '': + continue + if not included_file.startswith((os.sep, './', '~/')): + included_files.append('./' + included_file) + else: + included_files.append(included_file) + cls.include_in_doctest = utils.normalize_paths(included_files) + + excluded_files = [] + for excluded_file in options.exclude_from_doctest.split(','): + if excluded_file == '': + continue + if not excluded_file.startswith((os.sep, './', '~/')): + excluded_files.append('./' + excluded_file) + else: + excluded_files.append(excluded_file) + cls.exclude_from_doctest = utils.normalize_paths(excluded_files) + + inc_exc = set(cls.include_in_doctest).intersection( + cls.exclude_from_doctest + ) + if inc_exc: + raise ValueError('"%s" was specified in both the ' + 'include-in-doctest and exclude-from-doctest ' + 'options. You are not allowed to specify it in ' + 'both for doctesting.' % inc_exc) + + def run(self): + for m in self.messages: + col = getattr(m, 'col', 0) + yield m.lineno, col, (m.flake8_msg % m.message_args), m.__class__ From ef0e018fa2367188ba0aebbbc32897c63e6d4313 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 16 Jan 2016 07:04:55 -0600 Subject: [PATCH 077/364] Update setup.py for new entry-points and packages Comment out mccabe because it is currently broken with flake8 3.0 --- setup.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/setup.py b/setup.py index 064b28e..93e4fdb 100644 --- a/setup.py +++ b/setup.py @@ -45,17 +45,22 @@ setuptools.setup( maintainer="Ian Cordasco", maintainer_email="graffatcolmingov@gmail.com", url="https://gitlab.com/pycqa/flake8", - packages=["flake8", "flake8.options"], + packages=[ + "flake8", + "flake8.main", + "flake8.options", + "flake8.plugins", + ], install_requires=[ "pyflakes >= 0.8.1, < 1.1", "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", - "mccabe >= 0.2.1, < 0.4", + # "mccabe >= 0.2.1, < 0.4", ], entry_points={ 'distutils.commands': ['flake8 = flake8.main:Flake8Command'], - 'console_scripts': ['flake8 = flake8.main:main'], + 'console_scripts': ['flake8 = flake8.main.cli:main'], 'flake8.extension': [ - 'F = flake8._pyflakes:FlakesChecker', + 'F = flake8.plugins.pyflakes:FlakesChecker', ], }, classifiers=[ From ba56c34494f035a7ba48baf6f42094827a0be498 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 16 Jan 2016 07:05:12 -0600 Subject: [PATCH 078/364] Add venv testenv for testing the new version of flake8 --- tox.ini | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tox.ini b/tox.ini index 15218a1..77df69a 100644 --- a/tox.ini +++ b/tox.ini @@ -9,6 +9,11 @@ deps = commands = py.test {posargs} +[testenv:venv] +deps = + . +commands = {posargs} + [testenv:flake8] skipsdist = true skip_install = true From dfac6a2131403a037f0823c932ca85b1f6bccb38 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 16 Jan 2016 09:50:18 -0600 Subject: [PATCH 079/364] Record registered plugins and format their versions --- flake8/options/manager.py | 41 +++++++++++++++++++++++++++++++++++++++ flake8/plugins/manager.py | 5 +++++ 2 files changed, 46 insertions(+) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index d328a1f..c69dba2 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -155,6 +155,11 @@ class OptionManager(object): self.options = [] self.program_name = prog self.version = version + self.registered_plugins = set() + + @staticmethod + def format_plugin(plugin_tuple): + return dict(zip(["entry", "name", "version"], plugin_tuple)) def add_option(self, *args, **kwargs): """Create and register a new option. @@ -176,11 +181,47 @@ class OptionManager(object): self.config_options_dict[option.config_name] = option LOG.debug('Registered option "%s".', option) + def generate_versions(self, format_str='%(name)s: %(version)s'): + """Generate a comma-separated list of versions of plugins.""" + return ', '.join( + format_str % self.format_plugin(plugin) + for plugin in self.registered_plugins + ) + + def update_version_string(self): + """Update the flake8 version string.""" + self.parser.version = (self.version + ' (' + + self.generate_versions() + ')') + + def generate_epilog(self): + """Create an epilog with the version and name of each of plugin.""" + plugin_version_format = '%(name)s(%(entry)s): %(version)s' + self.parser.epilog = 'Installed plugins: ' + self.generate_versions( + plugin_version_format + ) + def parse_args(self, args=None, values=None): """Simple proxy to calling the OptionParser's parse_args method.""" + self.generate_epilog() + self.update_version_string() options, xargs = self.parser.parse_args(args, values) for option in self.options: old_value = getattr(options, option.dest) setattr(options, option.dest, option.normalize(old_value)) return options, xargs + + def register_plugin(self, entry_point_name, name, version): + """Register a plugin relying on the OptionManager. + + :param str entry_point_name: + The name of the entry-point loaded with pkg_resources. For + example, if the entry-point looks like: ``C90 = mccabe.Checker`` + then the ``entry_point_name`` would be ``C90``. + :param str name: + The name of the checker itself. This will be the ``name`` + attribute of the class or function loaded from the entry-point. + :param str version: + The version of the checker that we're using. + """ + self.registered_plugins.add((entry_point_name, name, version)) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 81abdc4..1afdaa9 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -98,6 +98,11 @@ class Plugin(object): self.name, optmanager ) add_options(optmanager) + optmanager.register_plugin( + entry_point_name=self.name, + name=self.plugin.name, + version=self.plugin.version + ) class PluginManager(object): From d6dbf797bcef595d7aa62f1b19e777709b92d549 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 17 Jan 2016 13:08:52 -0600 Subject: [PATCH 080/364] Add forgotten config file fixtures --- tests/fixtures/config_files/local-config.ini | 3 +++ tests/fixtures/config_files/user-config.ini | 5 +++++ 2 files changed, 8 insertions(+) create mode 100644 tests/fixtures/config_files/local-config.ini create mode 100644 tests/fixtures/config_files/user-config.ini diff --git a/tests/fixtures/config_files/local-config.ini b/tests/fixtures/config_files/local-config.ini new file mode 100644 index 0000000..348751a --- /dev/null +++ b/tests/fixtures/config_files/local-config.ini @@ -0,0 +1,3 @@ +[flake8] +exclude = docs/ +select = E,W,F diff --git a/tests/fixtures/config_files/user-config.ini b/tests/fixtures/config_files/user-config.ini new file mode 100644 index 0000000..b06c24f --- /dev/null +++ b/tests/fixtures/config_files/user-config.ini @@ -0,0 +1,5 @@ +[flake8] +exclude = + tests/fixtures/, + docs/ +ignore = D203 From dcf696d6786f936ed9dcfbdbf8e2d883266971fa Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 17 Jan 2016 21:45:32 -0600 Subject: [PATCH 081/364] Add tests around new OptionManager methods --- flake8/options/manager.py | 2 +- tests/unit/test_option_manager.py | 73 ++++++++++++++++++++++++++++++- 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index c69dba2..46e9f86 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -139,7 +139,7 @@ class OptionManager(object): """Manage Options and OptionParser while adding post-processing.""" def __init__(self, prog=None, version=None, - usage='%prog [options] input'): + usage='%prog [options] file file ...'): """Initialize an instance of an OptionManager. :param str prog: diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index b0e4ee7..660b1a6 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -6,11 +6,13 @@ import pytest from flake8.options import manager +TEST_VERSION = '3.0.0b1' + @pytest.fixture def optmanager(): """Generate a simple OptionManager with default test arguments.""" - return manager.OptionManager(prog='flake8', version='3.0.0b1') + return manager.OptionManager(prog='flake8', version=TEST_VERSION) def test_option_manager_creates_option_parser(optmanager): @@ -113,3 +115,72 @@ def test_parse_args_normalize_paths(optmanager): 'tox.ini', os.path.abspath('flake8/some-other.cfg'), ] + + +def test_format_plugin(): + """Verify that format_plugin turns a tuple into a dictionary.""" + plugin = manager.OptionManager.format_plugin(('T101', 'Testing', '0.0.0')) + assert plugin['entry'] == 'T101' + assert plugin['name'] == 'Testing' + assert plugin['version'] == '0.0.0' + + +def test_generate_versions(optmanager): + """Verify a comma-separated string is generated of registered plugins.""" + optmanager.registered_plugins = [ + ('T100', 'Testing 100', '0.0.0'), + ('T101', 'Testing 101', '0.0.0'), + ('T300', 'Testing 300', '0.0.0'), + ] + assert (optmanager.generate_versions() == + 'Testing 100: 0.0.0, Testing 101: 0.0.0, Testing 300: 0.0.0') + + +def test_generate_versions_with_format_string(optmanager): + """Verify a comma-separated string is generated of registered plugins.""" + optmanager.registered_plugins = [ + ('T100', 'Testing', '0.0.0'), + ('T101', 'Testing', '0.0.0'), + ('T300', 'Testing', '0.0.0'), + ] + assert ( + optmanager.generate_versions('%(name)s(%(entry)s): %(version)s') == + 'Testing(T100): 0.0.0, Testing(T101): 0.0.0, Testing(T300): 0.0.0' + ) + + +def test_update_version_string(optmanager): + """Verify we update the version string idempotently.""" + assert optmanager.version == TEST_VERSION + assert optmanager.parser.version == TEST_VERSION + + optmanager.registered_plugins = [ + ('T100', 'Testing 100', '0.0.0'), + ('T101', 'Testing 101', '0.0.0'), + ('T300', 'Testing 300', '0.0.0'), + ] + + optmanager.update_version_string() + + assert optmanager.version == TEST_VERSION + assert (optmanager.parser.version == TEST_VERSION + ' (' + 'Testing 100: 0.0.0, Testing 101: 0.0.0, Testing 300: 0.0.0)') + + +def test_generate_epilog(optmanager): + """Verify how we generate the epilog for help text.""" + assert optmanager.parser.epilog is None + + optmanager.registered_plugins = [ + ('T100', 'Testing 100', '0.0.0'), + ('T101', 'Testing 101', '0.0.0'), + ('T300', 'Testing 300', '0.0.0'), + ] + + expected_value = ( + 'Installed plugins: Testing 100(T100): 0.0.0, Testing 101(T101): ' + '0.0.0, Testing 300(T300): 0.0.0' + ) + + optmanager.generate_epilog() + assert optmanager.parser.epilog == expected_value From 4e294af81c14a627fa6a4771617a00a8a1abd293 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 06:43:47 -0600 Subject: [PATCH 082/364] Update fixture README with descriptions --- tests/fixtures/config_files/README.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/fixtures/config_files/README.rst b/tests/fixtures/config_files/README.rst index b69e080..b00adad 100644 --- a/tests/fixtures/config_files/README.rst +++ b/tests/fixtures/config_files/README.rst @@ -20,3 +20,19 @@ Purposes of existing fixtures This should only be used when providing config file(s) specified by the user on the command-line. + +``tests/fixtures/config_files/local-config.ini`` + + This should be used when providing config files that would have been found + by looking for config files in the current working project directory. + + +``tests/fixtures/config_files/no-flake8-section.ini`` + + This should be used when parsing an ini file without a ``[flake8]`` + section. + +``tests/fixtures/config_files/user-config.ini`` + + This is an example configuration file that would be found in the user's + home directory (or XDG Configuration Directory). From e66fc2efa81f9a05e52c6886b9606c4039259a5c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 07:19:16 -0600 Subject: [PATCH 083/364] Start creating documentation --- .gitignore | 1 + docs/build/.keep | 0 docs/source/conf.py | 295 ++++++++++++++++++++++++++++++++++++++++++ docs/source/index.rst | 22 ++++ tox.ini | 22 ++++ 5 files changed, 340 insertions(+) create mode 100644 docs/build/.keep create mode 100644 docs/source/conf.py create mode 100644 docs/source/index.rst diff --git a/.gitignore b/.gitignore index 612baf8..8e4e9dc 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,4 @@ *.egg-info *.log .eggs/ +docs/build/html/* diff --git a/docs/build/.keep b/docs/build/.keep new file mode 100644 index 0000000..e69de29 diff --git a/docs/source/conf.py b/docs/source/conf.py new file mode 100644 index 0000000..3c4be7f --- /dev/null +++ b/docs/source/conf.py @@ -0,0 +1,295 @@ +# -*- coding: utf-8 -*- +# +# flake8 documentation build configuration file, created by +# sphinx-quickstart on Tue Jan 19 07:14:10 2016. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys +import os + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +#sys.path.insert(0, os.path.abspath('.')) + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +#needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.doctest', + 'sphinx.ext.intersphinx', + 'sphinx.ext.todo', + 'sphinx.ext.coverage', + 'sphinx.ext.viewcode', +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The encoding of source files. +#source_encoding = 'utf-8-sig' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'flake8' +copyright = u'2016, Ian Cordasco' +author = u'Ian Cordasco' + +import flake8 +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = flake8.__version__ +# The full version, including alpha/beta/rc tags. +release = flake8.__version__ + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +#today = '' +# Else, today_fmt is used as the format for a strftime call. +#today_fmt = '%B %d, %Y' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +exclude_patterns = [] + +# The reST default role (used for this markup: `text`) to use for all +# documents. +#default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +#add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +#add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +#show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +#modindex_common_prefix = [] + +# If true, keep warnings as "system message" paragraphs in the built documents. +#keep_warnings = False + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = True + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +html_theme = 'alabaster' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +#html_theme_options = {} + +# Add any paths that contain custom themes here, relative to this directory. +#html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# " v documentation". +#html_title = None + +# A shorter title for the navigation bar. Default is the same as html_title. +#html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +#html_logo = None + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +#html_favicon = None + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +# Add any extra paths that contain custom files (such as robots.txt or +# .htaccess) here, relative to this directory. These files are copied +# directly to the root of the documentation. +#html_extra_path = [] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +#html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +#html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +#html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +#html_additional_pages = {} + +# If false, no module index is generated. +#html_domain_indices = True + +# If false, no index is generated. +#html_use_index = True + +# If true, the index is split into individual pages for each letter. +#html_split_index = False + +# If true, links to the reST sources are added to the pages. +#html_show_sourcelink = True + +# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. +#html_show_sphinx = True + +# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. +#html_show_copyright = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +#html_use_opensearch = '' + +# This is the file name suffix for HTML files (e.g. ".xhtml"). +#html_file_suffix = None + +# Language to be used for generating the HTML full-text search index. +# Sphinx supports the following languages: +# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' +# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' +#html_search_language = 'en' + +# A dictionary with options for the search language support, empty by default. +# Now only 'ja' uses this config value +#html_search_options = {'type': 'default'} + +# The name of a javascript file (relative to the configuration directory) that +# implements a search results scorer. If empty, the default will be used. +#html_search_scorer = 'scorer.js' + +# Output file base name for HTML help builder. +htmlhelp_basename = 'flake8doc' + +# -- Options for LaTeX output --------------------------------------------- + +latex_elements = { +# The paper size ('letterpaper' or 'a4paper'). +#'papersize': 'letterpaper', + +# The font size ('10pt', '11pt' or '12pt'). +#'pointsize': '10pt', + +# Additional stuff for the LaTeX preamble. +#'preamble': '', + +# Latex figure (float) alignment +#'figure_align': 'htbp', +} + +# Grouping the document tree into LaTeX files. List of tuples +# (source start file, target name, title, +# author, documentclass [howto, manual, or own class]). +latex_documents = [ + (master_doc, 'flake8.tex', u'flake8 Documentation', + u'Ian Cordasco', 'manual'), +] + +# The name of an image file (relative to this directory) to place at the top of +# the title page. +#latex_logo = None + +# For "manual" documents, if this is true, then toplevel headings are parts, +# not chapters. +#latex_use_parts = False + +# If true, show page references after internal links. +#latex_show_pagerefs = False + +# If true, show URL addresses after external links. +#latex_show_urls = False + +# Documents to append as an appendix to all manuals. +#latex_appendices = [] + +# If false, no module index is generated. +#latex_domain_indices = True + + +# -- Options for manual page output --------------------------------------- + +# One entry per manual page. List of tuples +# (source start file, name, description, authors, manual section). +man_pages = [ + (master_doc, 'flake8', u'flake8 Documentation', + [author], 1) +] + +# If true, show URL addresses after external links. +#man_show_urls = False + + +# -- Options for Texinfo output ------------------------------------------- + +# Grouping the document tree into Texinfo files. List of tuples +# (source start file, target name, title, author, +# dir menu entry, description, category) +texinfo_documents = [ + (master_doc, 'flake8', u'flake8 Documentation', + author, 'flake8', 'One line description of project.', + 'Miscellaneous'), +] + +# Documents to append as an appendix to all manuals. +#texinfo_appendices = [] + +# If false, no module index is generated. +#texinfo_domain_indices = True + +# How to display URL addresses: 'footnote', 'no', or 'inline'. +#texinfo_show_urls = 'footnote' + +# If true, do not generate a @detailmenu in the "Top" node's menu. +#texinfo_no_detailmenu = False + + +# Example configuration for intersphinx: refer to the Python standard library. +intersphinx_mapping = {'https://docs.python.org/': None} diff --git a/docs/source/index.rst b/docs/source/index.rst new file mode 100644 index 0000000..904ca87 --- /dev/null +++ b/docs/source/index.rst @@ -0,0 +1,22 @@ +.. flake8 documentation master file, created by + sphinx-quickstart on Tue Jan 19 07:14:10 2016. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to flake8's documentation! +================================== + +Contents: + +.. toctree:: + :maxdepth: 2 + + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` + diff --git a/tox.ini b/tox.ini index 77df69a..b2c0a91 100644 --- a/tox.ini +++ b/tox.ini @@ -24,6 +24,28 @@ deps = commands = flake8 +[testenv:docs] +deps = + sphinx>=1.3.0 +commands = + sphinx-build -E -W -c docs/source/ -b html docs/source/ docs/build/html + +[testenv:serve-docs] +basepython = python3.4 +skipsdist = true +skip_install = true +use_develop = false +changedir = docs/_build/html +deps = +commands = + python -m http.server {posargs} + +[testenv:readme] +deps = + readme_renderer +commands = + python setup.py check -r -s + [flake8] # Ignore some flake8-docstrings errors ignore = D203 From 11be73dbbb7c40ac973c71599b4647d2c076d237 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 07:27:39 -0600 Subject: [PATCH 084/364] Note that we need at least Sphinx 1.3 Exclude sphinx conf.py from flake8 --- docs/source/conf.py | 2 +- tox.ini | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 3c4be7f..835a673 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -23,7 +23,7 @@ import os # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' +needs_sphinx = '1.3' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom diff --git a/tox.ini b/tox.ini index b2c0a91..67c9478 100644 --- a/tox.ini +++ b/tox.ini @@ -49,3 +49,7 @@ commands = [flake8] # Ignore some flake8-docstrings errors ignore = D203 +# NOTE(sigmavirus24): Once we release 3.0.0 this exclude option can be specified +# across multiple lines. Presently it cannot be specified across multiple lines. +# :-( +exclude = .git,__pycache__,docs/source/conf.py From 83dd81a44526eab408ffc466e355263e76c8b2e0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 07:45:25 -0600 Subject: [PATCH 085/364] Add missing docstrings to plugin submodule --- flake8/options/manager.py | 1 + flake8/plugins/__init__.py | 1 + flake8/plugins/manager.py | 16 ++++++++++++++++ flake8/plugins/pyflakes.py | 6 ++++++ 4 files changed, 24 insertions(+) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 46e9f86..1dd6c8d 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -159,6 +159,7 @@ class OptionManager(object): @staticmethod def format_plugin(plugin_tuple): + """Convert a plugin tuple into a dictionary mapping name to value.""" return dict(zip(["entry", "name", "version"], plugin_tuple)) def add_option(self, *args, **kwargs): diff --git a/flake8/plugins/__init__.py b/flake8/plugins/__init__.py index e69de29..fda6a44 100644 --- a/flake8/plugins/__init__.py +++ b/flake8/plugins/__init__.py @@ -0,0 +1 @@ +"""Submodule of built-in plugins and plugin managers.""" diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 1afdaa9..e4ad20e 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -6,6 +6,14 @@ import pkg_resources LOG = logging.getLogger(__name__) +__all__ = ( + 'Checkers', + 'Listeners', + 'Plugin', + 'PluginManager', + 'ReportFormatters', +) + class Plugin(object): """Wrap an EntryPoint from setuptools and other logic.""" @@ -25,17 +33,23 @@ class Plugin(object): self._plugin = None def __repr__(self): + """Provide an easy to read description of the current plugin.""" return 'Plugin(name="{0}", entry_point="{1}")'.format( self.name, self.entry_point ) @property def plugin(self): + """The loaded (and cached) plugin associated with the entry-point. + + This property implicitly loads the plugin and then caches it. + """ self.load_plugin() return self._plugin @property def version(self): + """Return the version attribute on the plugin.""" return self.plugin.version def execute(self, *args, **kwargs): @@ -124,10 +138,12 @@ class PluginManager(object): self._load_all_plugins() def __contains__(self, name): + """Check if the entry-point name is in this plugin manager.""" LOG.debug('Checking for "%s" in plugin manager.', name) return name in self.plugins def __getitem__(self, name): + """Retrieve a plugin by its entry-point name.""" LOG.debug('Retrieving plugin for "%s".', name) return self.plugins[name] diff --git a/flake8/plugins/pyflakes.py b/flake8/plugins/pyflakes.py index c801e54..85465fc 100644 --- a/flake8/plugins/pyflakes.py +++ b/flake8/plugins/pyflakes.py @@ -1,3 +1,4 @@ +"""Plugin built-in to Flake8 to treat pyflakes as a plugin.""" # -*- coding: utf-8 -*- from __future__ import absolute_import try: @@ -40,10 +41,12 @@ patch_pyflakes() class FlakesChecker(pyflakes.checker.Checker): """Subclass the Pyflakes checker to conform with the flake8 API.""" + name = 'pyflakes' version = pyflakes.__version__ def __init__(self, tree, filename): + """Initialize the PyFlakes plugin with an AST tree and filename.""" filename = utils.normalize_paths(filename)[0] withDoctest = self.withDoctest included_by = [include for include in self.include_in_doctest @@ -65,6 +68,7 @@ class FlakesChecker(pyflakes.checker.Checker): @classmethod def add_options(cls, parser): + """Register options for PyFlakes on the Flake8 OptionManager.""" parser.add_option( '--builtins', parse_from_config=True, comma_separated_list=True, help="define more built-ins, comma separated", @@ -91,6 +95,7 @@ class FlakesChecker(pyflakes.checker.Checker): @classmethod def parse_options(cls, options): + """Parse option values from Flake8's OptionManager.""" if options.builtins: cls.builtIns = cls.builtIns.union(options.builtins.split(',')) cls.withDoctest = options.doctests @@ -125,6 +130,7 @@ class FlakesChecker(pyflakes.checker.Checker): 'both for doctesting.' % inc_exc) def run(self): + """Run the plugin.""" for m in self.messages: col = getattr(m, 'col', 0) yield m.lineno, col, (m.flake8_msg % m.message_args), m.__class__ From 15d7f7679c5240d09d26ecfc98b4c63bfda53dbe Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 07:45:43 -0600 Subject: [PATCH 086/364] Builtins will not be parsed by flake8 for us --- flake8/plugins/pyflakes.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/plugins/pyflakes.py b/flake8/plugins/pyflakes.py index 85465fc..d787344 100644 --- a/flake8/plugins/pyflakes.py +++ b/flake8/plugins/pyflakes.py @@ -97,7 +97,7 @@ class FlakesChecker(pyflakes.checker.Checker): def parse_options(cls, options): """Parse option values from Flake8's OptionManager.""" if options.builtins: - cls.builtIns = cls.builtIns.union(options.builtins.split(',')) + cls.builtIns = cls.builtIns.union(options.builtins) cls.withDoctest = options.doctests included_files = [] From 5a1e6943d8209b72cc697295f06f1219ba59d2c0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 07:46:41 -0600 Subject: [PATCH 087/364] Stop calling split on a list --exclude-from-doctests is also parsed by flake8 for us --- flake8/plugins/pyflakes.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/plugins/pyflakes.py b/flake8/plugins/pyflakes.py index d787344..7f6107a 100644 --- a/flake8/plugins/pyflakes.py +++ b/flake8/plugins/pyflakes.py @@ -111,7 +111,7 @@ class FlakesChecker(pyflakes.checker.Checker): cls.include_in_doctest = utils.normalize_paths(included_files) excluded_files = [] - for excluded_file in options.exclude_from_doctest.split(','): + for excluded_file in options.exclude_from_doctest: if excluded_file == '': continue if not excluded_file.startswith((os.sep, './', '~/')): From 56a75cc39187f5fd1aca3a2c931e27a211977e49 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 22:57:04 -0600 Subject: [PATCH 088/364] Add tests for the Plugin class --- tests/unit/test_plugin.py | 144 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 tests/unit/test_plugin.py diff --git a/tests/unit/test_plugin.py b/tests/unit/test_plugin.py new file mode 100644 index 0000000..28b95c9 --- /dev/null +++ b/tests/unit/test_plugin.py @@ -0,0 +1,144 @@ +"""Tests for flake8.plugins.manager.Plugin.""" +import mock + +from flake8.plugins import manager + + +def test_load_plugin_fallsback_on_old_setuptools(): + """Verify we fallback gracefully to on old versions of setuptools.""" + entry_point = mock.Mock(spec=['load']) + plugin = manager.Plugin('T000', entry_point) + + plugin.load_plugin() + entry_point.load.assert_called_once_with(require=False) + + +def test_load_plugin_avoids_deprecated_entry_point_methods(): + """Verify we use the preferred methods on new versions of setuptools.""" + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin = manager.Plugin('T000', entry_point) + + plugin.load_plugin(verify_requirements=True) + assert entry_point.load.called is False + entry_point.require.assert_called_once_with() + entry_point.resolve.assert_called_once_with() + + +def test_load_plugin_is_idempotent(): + """Verify we use the preferred methods on new versions of setuptools.""" + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin = manager.Plugin('T000', entry_point) + + plugin.load_plugin(verify_requirements=True) + plugin.load_plugin(verify_requirements=True) + plugin.load_plugin() + assert entry_point.load.called is False + entry_point.require.assert_called_once_with() + entry_point.resolve.assert_called_once_with() + + +def test_load_plugin_only_calls_require_when_verifying_requirements(): + """Verify we do not call require when verify_requirements is False.""" + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin = manager.Plugin('T000', entry_point) + + plugin.load_plugin() + assert entry_point.load.called is False + assert entry_point.require.called is False + entry_point.resolve.assert_called_once_with() + + +def test_plugin_property_loads_plugin_on_first_use(): + """Verify that we load our plugin when we first try to use it.""" + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin = manager.Plugin('T000', entry_point) + + assert plugin.plugin is not None + entry_point.resolve.assert_called_once_with() + + +def test_execute_calls_plugin_with_passed_arguments(): + """Verify that we pass arguments directly to the plugin.""" + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin_obj = mock.Mock() + plugin = manager.Plugin('T000', entry_point) + plugin._plugin = plugin_obj + + plugin.execute('arg1', 'arg2', kwarg1='value1', kwarg2='value2') + plugin_obj.assert_called_once_with( + 'arg1', 'arg2', kwarg1='value1', kwarg2='value2' + ) + + # Extra assertions + assert entry_point.load.called is False + assert entry_point.require.called is False + assert entry_point.resolve.called is False + + +def test_version_proxies_to_the_plugin(): + """Verify that we pass arguments directly to the plugin.""" + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin_obj = mock.Mock(spec_set=['version']) + plugin_obj.version = 'a.b.c' + plugin = manager.Plugin('T000', entry_point) + plugin._plugin = plugin_obj + + assert plugin.version == 'a.b.c' + + +def test_register_options(): + """Verify we call add_options on the plugin only if it exists.""" + # Set up our mocks and Plugin object + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin_obj = mock.Mock(spec_set=['name', 'version', 'add_options', + 'parse_options']) + option_manager = mock.Mock(spec=['register_plugin']) + plugin = manager.Plugin('T000', entry_point) + plugin._plugin = plugin_obj + + # Call the method we're testing. + plugin.register_options(option_manager) + + # Assert that we call add_options + plugin_obj.add_options.assert_called_once_with(option_manager) + # Assert that we register the plugin + option_manager.register_plugin.assert_called_once_with( + entry_point_name='T000', + version=plugin_obj.version, + name=plugin_obj.name, + ) + + +def test_register_options_checks_plugin_for_method(): + """Verify we call add_options on the plugin only if it exists.""" + # Set up our mocks and Plugin object + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin_obj = mock.Mock(spec_set=['name', 'version', 'parse_options']) + option_manager = mock.Mock(spec=['register_plugin']) + plugin = manager.Plugin('T000', entry_point) + plugin._plugin = plugin_obj + + # Call the method we're testing. + plugin.register_options(option_manager) + + # Assert that we register the plugin + assert option_manager.register_plugin.called is False + + +def test_provide_options(): + """Verify we call add_options on the plugin only if it exists.""" + # Set up our mocks and Plugin object + entry_point = mock.Mock(spec=['require', 'resolve', 'load']) + plugin_obj = mock.Mock(spec_set=['name', 'version', 'add_options', + 'parse_options']) + option_manager = mock.Mock() + plugin = manager.Plugin('T000', entry_point) + plugin._plugin = plugin_obj + + # Call the method we're testing. + plugin.provide_options(option_manager, 'options', None) + + # Assert that we call add_options + plugin_obj.parse_options.assert_called_once_with( + option_manager, 'options', None + ) From 6a72e36681aa237ae5e0fc04381ee1d6ab4d1e08 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 19 Jan 2016 23:01:07 -0600 Subject: [PATCH 089/364] Start tests for PluginManager --- tests/unit/test_plugin_manager.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 tests/unit/test_plugin_manager.py diff --git a/tests/unit/test_plugin_manager.py b/tests/unit/test_plugin_manager.py new file mode 100644 index 0000000..8bafd2b --- /dev/null +++ b/tests/unit/test_plugin_manager.py @@ -0,0 +1,13 @@ +"""Tests for flake8.plugins.manager.PluginManager.""" +import mock + +from flake8.plugins import manager + + +@mock.patch('pkg_resources.iter_entry_points') +def test_calls_pkg_resources_on_instantiation(iter_entry_points): + """Verify that we call iter_entry_points when we create a manager.""" + iter_entry_points.return_value = [] + manager.PluginManager(namespace='testing.pkg_resources') + + iter_entry_points.assert_called_once_with('testing.pkg_resources') From 50e8fef125341dc529f770dc7a6f48783c66a89e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 21 Jan 2016 08:26:05 -0600 Subject: [PATCH 090/364] Add more PluginManager tests --- tests/unit/test_plugin_manager.py | 49 +++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/tests/unit/test_plugin_manager.py b/tests/unit/test_plugin_manager.py index 8bafd2b..022848f 100644 --- a/tests/unit/test_plugin_manager.py +++ b/tests/unit/test_plugin_manager.py @@ -4,6 +4,13 @@ import mock from flake8.plugins import manager +def create_entry_point_mock(name): + """Create a mocked EntryPoint.""" + ep = mock.Mock(spec=['name']) + ep.name = name + return ep + + @mock.patch('pkg_resources.iter_entry_points') def test_calls_pkg_resources_on_instantiation(iter_entry_points): """Verify that we call iter_entry_points when we create a manager.""" @@ -11,3 +18,45 @@ def test_calls_pkg_resources_on_instantiation(iter_entry_points): manager.PluginManager(namespace='testing.pkg_resources') iter_entry_points.assert_called_once_with('testing.pkg_resources') + + +@mock.patch('pkg_resources.iter_entry_points') +def test_calls_pkg_resources_creates_plugins_automaticaly(iter_entry_points): + """Verify that we create Plugins on instantiation.""" + iter_entry_points.return_value = [ + create_entry_point_mock('T100'), + create_entry_point_mock('T200'), + ] + plugin_mgr = manager.PluginManager(namespace='testing.pkg_resources') + + iter_entry_points.assert_called_once_with('testing.pkg_resources') + assert 'T100' in plugin_mgr.plugins + assert 'T200' in plugin_mgr.plugins + assert isinstance(plugin_mgr.plugins['T100'], manager.Plugin) + assert isinstance(plugin_mgr.plugins['T200'], manager.Plugin) + + +@mock.patch('pkg_resources.iter_entry_points') +def test_proxies_contains_to_plugins_dictionary(iter_entry_points): + """Verify that we can use the PluginManager like a dictionary.""" + iter_entry_points.return_value = [ + create_entry_point_mock('T100'), + create_entry_point_mock('T200'), + ] + plugin_mgr = manager.PluginManager(namespace='testing.pkg_resources') + + assert 'T100' in plugin_mgr + assert 'T200' in plugin_mgr + + +@mock.patch('pkg_resources.iter_entry_points') +def test_proxies_getitm_to_plugins_dictionary(iter_entry_points): + """Verify that we can use the PluginManager like a dictionary.""" + iter_entry_points.return_value = [ + create_entry_point_mock('T100'), + create_entry_point_mock('T200'), + ] + plugin_mgr = manager.PluginManager(namespace='testing.pkg_resources') + + assert isinstance(plugin_mgr['T100'], manager.Plugin) + assert isinstance(plugin_mgr['T200'], manager.Plugin) From 916bec859cfa0a0df879b2674527bce9c0926a46 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 21 Jan 2016 08:32:35 -0600 Subject: [PATCH 091/364] Finish testing PluginManager --- tests/unit/test_plugin_manager.py | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/tests/unit/test_plugin_manager.py b/tests/unit/test_plugin_manager.py index 022848f..05073cf 100644 --- a/tests/unit/test_plugin_manager.py +++ b/tests/unit/test_plugin_manager.py @@ -50,7 +50,7 @@ def test_proxies_contains_to_plugins_dictionary(iter_entry_points): @mock.patch('pkg_resources.iter_entry_points') -def test_proxies_getitm_to_plugins_dictionary(iter_entry_points): +def test_proxies_getitem_to_plugins_dictionary(iter_entry_points): """Verify that we can use the PluginManager like a dictionary.""" iter_entry_points.return_value = [ create_entry_point_mock('T100'), @@ -60,3 +60,17 @@ def test_proxies_getitm_to_plugins_dictionary(iter_entry_points): assert isinstance(plugin_mgr['T100'], manager.Plugin) assert isinstance(plugin_mgr['T200'], manager.Plugin) + + +@mock.patch('pkg_resources.iter_entry_points') +def test_handles_mapping_functions_across_plugins(iter_entry_points): + """Verify we can use the PluginManager call functions on all plugins.""" + entry_point_mocks = [ + create_entry_point_mock('T100'), + create_entry_point_mock('T200'), + ] + iter_entry_points.return_value = entry_point_mocks + plugin_mgr = manager.PluginManager(namespace='testing.pkg_resources') + plugins = [plugin_mgr[name] for name in plugin_mgr.names] + + assert list(plugin_mgr.map(lambda x: x)) == plugins From c67735792b2d6481e9a5d0de468062c776834759 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 21 Jan 2016 08:32:46 -0600 Subject: [PATCH 092/364] Add documentation to PluginManager.map --- flake8/plugins/manager.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index e4ad20e..8fd3a24 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -169,7 +169,8 @@ class PluginManager(object): pass Any extra positional or keyword arguments specified with map will - be passed along to this function after the plugin. + be passed along to this function after the plugin. The plugin + passed is a :class:`~flake8.plugins.manager.Plugin`. :param args: Positional arguments to pass to ``func`` after each plugin. :param kwargs: From 64edc550617123b661f6e7ee9899ae4b0fc19af4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 21 Jan 2016 22:15:17 -0600 Subject: [PATCH 093/364] Add method to build Tries from Notifiers --- flake8/plugins/manager.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 8fd3a24..ce74e08 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -4,6 +4,8 @@ import logging import pkg_resources +from flake8 import _trie + LOG = logging.getLogger(__name__) __all__ = ( @@ -249,6 +251,13 @@ class Listeners(PluginTypeManager): namespace = 'flake8.listen' + def build_trie(self): + """Build a Trie for our Listeners.""" + trie = _trie.Trie() + for name in self.names: + trie.add(name, self.manager[name]) + return trie + class ReportFormatters(PluginTypeManager): """All of the report formatters registered through entry-points.""" From 6546cf41d4b41e3734e382eea6765aaf51b563e5 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 21 Jan 2016 22:15:38 -0600 Subject: [PATCH 094/364] Add tests for the PluginTypeManager --- tests/unit/test_plugin_type_manager.py | 55 ++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 tests/unit/test_plugin_type_manager.py diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py new file mode 100644 index 0000000..c0aa724 --- /dev/null +++ b/tests/unit/test_plugin_type_manager.py @@ -0,0 +1,55 @@ +"""Tests for flake8.plugins.manager.PluginTypeManager.""" +import collections +import mock + +from flake8.plugins import manager + +TEST_NAMESPACE = "testing.plugin-type-manager" + + +class TestType(manager.PluginTypeManager): + """Fake PluginTypeManager.""" + + namespace = TEST_NAMESPACE + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_instantiates_a_manager(PluginManager): + """Verify we create a PluginManager on instantiation.""" + TestType() + + PluginManager.assert_called_once_with(TEST_NAMESPACE) + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_proxies_names_to_manager(PluginManager): + """Verify we proxy the names attribute.""" + PluginManager.return_value = mock.Mock(names=[ + 'T100', 'T200', 'T300' + ]) + type_mgr = TestType() + + assert type_mgr.names == ['T100', 'T200', 'T300'] + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_proxies_plugins_to_manager(PluginManager): + """Verify we proxy the plugins attribute.""" + PluginManager.return_value = mock.Mock(plugins=[ + 'T100', 'T200', 'T300' + ]) + type_mgr = TestType() + + assert type_mgr.plugins == ['T100', 'T200', 'T300'] + + +def test_generate_call_function(): + """Verify the function we generate.""" + optmanager = object() + plugin = mock.Mock(method_name=lambda x: x) + func = manager.PluginTypeManager._generate_call_function( + 'method_name', optmanager, + ) + + assert isinstance(func, collections.Callable) + assert func(plugin) is optmanager From 3b64ff2a1f1b1b90f74f6f3e9fad7a6723c6f68a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 22 Jan 2016 09:04:52 -0600 Subject: [PATCH 095/364] Make Plugin.load_plugin raise a Flake8 exception Let's catch exceptions, log them, and re-raise as a FailedToLoad exception. This also refactors the actually loading of plugins into a private method on the Plugin class. --- flake8/exceptions.py | 25 +++++++++++++++++++++++++ flake8/plugins/manager.py | 37 +++++++++++++++++++++++++------------ tests/unit/test_plugin.py | 12 ++++++++++++ 3 files changed, 62 insertions(+), 12 deletions(-) create mode 100644 flake8/exceptions.py diff --git a/flake8/exceptions.py b/flake8/exceptions.py new file mode 100644 index 0000000..ac78769 --- /dev/null +++ b/flake8/exceptions.py @@ -0,0 +1,25 @@ +"""Exception classes for all of Flake8.""" + + +class Flake8Exception(Exception): + """Plain Flake8 exception.""" + + pass + + +class FailedToLoadPlugin(Flake8Exception): + """Exception raised when a plugin fails to load.""" + + FORMAT = 'Flake8 failed to load plugin "%(name)s" due to %(exc)s.' + + def __init__(self, *args, **kwargs): + """Initialize our FailedToLoadPlugin exception.""" + self.plugin = kwargs.pop('plugin') + self.ep_name = self.plugin.name + self.original_exception = kwargs.pop('exception') + super(FailedToLoadPlugin, self).__init__(*args, **kwargs) + + def __str__(self): + """Return a nice string for our exception.""" + return self.FORMAT % {'name': self.ep_name, + 'exc': self.original_exception} diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index ce74e08..a80020e 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -5,6 +5,7 @@ import logging import pkg_resources from flake8 import _trie +from flake8 import exceptions LOG = logging.getLogger(__name__) @@ -58,6 +59,21 @@ class Plugin(object): r"""Call the plugin with \*args and \*\*kwargs.""" return self.plugin(*args, **kwargs) + def _load(self, verify_requirements): + # Avoid relying on hasattr() here. + resolve = getattr(self.entry_point, 'resolve', None) + require = getattr(self.entry_point, 'require', None) + if resolve and require: + if verify_requirements: + LOG.debug('Verifying plugin "%s"\'s requirements.', + self.name) + require() + self._plugin = resolve() + else: + self._plugin = self.entry_point.load( + require=verify_requirements + ) + def load_plugin(self, verify_requirements=False): """Retrieve the plugin for this entry-point. @@ -73,19 +89,16 @@ class Plugin(object): """ if self._plugin is None: LOG.debug('Loading plugin "%s" from entry-point.', self.name) - # Avoid relying on hasattr() here. - resolve = getattr(self.entry_point, 'resolve', None) - require = getattr(self.entry_point, 'require', None) - if resolve and require: - if verify_requirements: - LOG.debug('Verifying plugin "%s"\'s requirements.', - self.name) - require() - self._plugin = resolve() - else: - self._plugin = self.entry_point.load( - require=verify_requirements + try: + self._load(verify_requirements) + except Exception as load_exception: + LOG.exception(load_exception, exc_info=True) + failed_to_load = exceptions.FailedToLoadPlugin( + plugin=self, + exception=load_exception, ) + LOG.critical(str(failed_to_load)) + raise failed_to_load def provide_options(self, optmanager, options, extra_args): """Pass the parsed options and extra arguments to the plugin.""" diff --git a/tests/unit/test_plugin.py b/tests/unit/test_plugin.py index 28b95c9..01454a1 100644 --- a/tests/unit/test_plugin.py +++ b/tests/unit/test_plugin.py @@ -1,6 +1,8 @@ """Tests for flake8.plugins.manager.Plugin.""" import mock +import pytest +from flake8 import exceptions from flake8.plugins import manager @@ -48,6 +50,16 @@ def test_load_plugin_only_calls_require_when_verifying_requirements(): entry_point.resolve.assert_called_once_with() +def test_load_plugin_catches_and_reraises_exceptions(): + """Verify we raise our own FailedToLoadPlugin.""" + entry_point = mock.Mock(spec=['require', 'resolve']) + entry_point.resolve.side_effect = ValueError('Test failure') + plugin = manager.Plugin('T000', entry_point) + + with pytest.raises(exceptions.FailedToLoadPlugin): + plugin.load_plugin() + + def test_plugin_property_loads_plugin_on_first_use(): """Verify that we load our plugin when we first try to use it.""" entry_point = mock.Mock(spec=['require', 'resolve', 'load']) From 6e8e2b966988d2f4e30d644d95abd8c03fe36e55 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 22 Jan 2016 09:06:24 -0600 Subject: [PATCH 096/364] Write more PluginTypeManager tests --- tests/unit/test_plugin_type_manager.py | 80 +++++++++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-) diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index c0aa724..1f8c646 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -1,12 +1,26 @@ """Tests for flake8.plugins.manager.PluginTypeManager.""" import collections -import mock +import mock +import pytest + +from flake8 import exceptions from flake8.plugins import manager TEST_NAMESPACE = "testing.plugin-type-manager" +def create_plugin_mock(raise_exception=False): + """Create an auto-spec'd mock of a flake8 Plugin.""" + plugin = mock.create_autospec(manager.Plugin, instance=True) + if raise_exception: + plugin.load_plugin.side_effect = exceptions.FailedToLoadPlugin( + plugin=mock.Mock(name='T101'), + exception=ValueError('Test failure'), + ) + return plugin + + class TestType(manager.PluginTypeManager): """Fake PluginTypeManager.""" @@ -53,3 +67,67 @@ def test_generate_call_function(): assert isinstance(func, collections.Callable) assert func(plugin) is optmanager + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_load_plugins(PluginManager): + """Verify load plugins loads *every* plugin.""" + # Create a bunch of fake plugins + plugins = [create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock()] + + # Have a function that will actually call the method underneath + def fake_map(func): + for plugin in plugins: + yield func(plugin) + + # Mock out the PluginManager instance + manager_mock = mock.Mock(spec=['map']) + # Replace the map method + manager_mock.map = fake_map + # Return our PluginManager mock + PluginManager.return_value = manager_mock + + type_mgr = TestType() + # Load the tests (do what we're actually testing) + assert len(type_mgr.load_plugins()) == 8 + # Assert that our closure does what we think it does + for plugin in plugins: + plugin.load_plugin.assert_called_once_with() + assert type_mgr.plugins_loaded is True + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_load_plugins_fails(PluginManager): + """Verify load plugins bubbles up exceptions.""" + plugins = [create_plugin_mock(), create_plugin_mock(True), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock()] + + # Have a function that will actually call the method underneath + def fake_map(func): + for plugin in plugins: + yield func(plugin) + + # Mock out the PluginManager instance + manager_mock = mock.Mock(spec=['map']) + # Replace the map method + manager_mock.map = fake_map + # Return our PluginManager mock + PluginManager.return_value = manager_mock + + type_mgr = TestType() + with pytest.raises(exceptions.FailedToLoadPlugin): + type_mgr.load_plugins() + + # Assert we didn't finish loading plugins + assert type_mgr.plugins_loaded is False + # Assert the first two plugins had their load_plugin method called + plugins[0].load_plugin.assert_called_once_with() + plugins[1].load_plugin.assert_called_once_with() + # Assert the rest of the plugins were not loaded + for plugin in plugins[2:]: + assert plugin.load_plugin.called is False From 04556f1a1b9c26f73ee0aabacac8e0ba542d6a65 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 23 Jan 2016 23:01:30 -0600 Subject: [PATCH 097/364] Build a Notifier instead of a Trie --- flake8/plugins/manager.py | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index a80020e..8d5d12c 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -4,7 +4,7 @@ import logging import pkg_resources -from flake8 import _trie +from flake8 import notifier from flake8 import exceptions LOG = logging.getLogger(__name__) @@ -264,12 +264,19 @@ class Listeners(PluginTypeManager): namespace = 'flake8.listen' - def build_trie(self): - """Build a Trie for our Listeners.""" - trie = _trie.Trie() + def build_notifier(self): + """Build a Notifier for our Listeners. + + :returns: + Object to notify our listeners of certain error codes and + warnings. + :rtype: + :class:`~flake8.notifier.Notifier` + """ + notifier_trie = notifier.Notifier() for name in self.names: - trie.add(name, self.manager[name]) - return trie + notifier_trie.register_listener(name, self.manager[name]) + return notifier_trie class ReportFormatters(PluginTypeManager): From b0a258fe79c7f8713708b03f64cdbc832a449bf4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 23 Jan 2016 23:02:06 -0600 Subject: [PATCH 098/364] Refactor our PluginTypeManager tests Add tests for PluginTypeManager#{register,provide}_optoins --- tests/unit/test_plugin_type_manager.py | 78 ++++++++++++++++++-------- 1 file changed, 56 insertions(+), 22 deletions(-) diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index 1f8c646..cb121c6 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -21,6 +21,20 @@ def create_plugin_mock(raise_exception=False): return plugin +def create_mapping_manager_mock(plugins): + """Create a mock for the PluginManager.""" + # Have a function that will actually call the method underneath + def fake_map(func): + for plugin in plugins: + yield func(plugin) + + # Mock out the PluginManager instance + manager_mock = mock.Mock(spec=['map']) + # Replace the map method + manager_mock.map = fake_map + return manager_mock + + class TestType(manager.PluginTypeManager): """Fake PluginTypeManager.""" @@ -77,18 +91,8 @@ def test_load_plugins(PluginManager): create_plugin_mock(), create_plugin_mock(), create_plugin_mock(), create_plugin_mock(), create_plugin_mock(), create_plugin_mock()] - - # Have a function that will actually call the method underneath - def fake_map(func): - for plugin in plugins: - yield func(plugin) - - # Mock out the PluginManager instance - manager_mock = mock.Mock(spec=['map']) - # Replace the map method - manager_mock.map = fake_map # Return our PluginManager mock - PluginManager.return_value = manager_mock + PluginManager.return_value = create_mapping_manager_mock(plugins) type_mgr = TestType() # Load the tests (do what we're actually testing) @@ -106,18 +110,8 @@ def test_load_plugins_fails(PluginManager): create_plugin_mock(), create_plugin_mock(), create_plugin_mock(), create_plugin_mock(), create_plugin_mock(), create_plugin_mock()] - - # Have a function that will actually call the method underneath - def fake_map(func): - for plugin in plugins: - yield func(plugin) - - # Mock out the PluginManager instance - manager_mock = mock.Mock(spec=['map']) - # Replace the map method - manager_mock.map = fake_map # Return our PluginManager mock - PluginManager.return_value = manager_mock + PluginManager.return_value = create_mapping_manager_mock(plugins) type_mgr = TestType() with pytest.raises(exceptions.FailedToLoadPlugin): @@ -131,3 +125,43 @@ def test_load_plugins_fails(PluginManager): # Assert the rest of the plugins were not loaded for plugin in plugins[2:]: assert plugin.load_plugin.called is False + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_register_options(PluginManager): + """Test that we map over every plugin to register options.""" + plugins = [create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock()] + # Return our PluginManager mock + PluginManager.return_value = create_mapping_manager_mock(plugins) + optmanager = object() + + type_mgr = TestType() + type_mgr.register_options(optmanager) + + for plugin in plugins: + plugin.register_options.assert_called_with(optmanager) + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_provide_options(PluginManager): + """Test that we map over every plugin to provide parsed options.""" + plugins = [create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock(), + create_plugin_mock(), create_plugin_mock()] + # Return our PluginManager mock + PluginManager.return_value = create_mapping_manager_mock(plugins) + optmanager = object() + options = object() + extra_args = [] + + type_mgr = TestType() + type_mgr.provide_options(optmanager, options, extra_args) + + for plugin in plugins: + plugin.provide_options.assert_called_with(optmanager, + options, + extra_args) From ebdc935ffc44bbfaae863b3b07bc165c601dd88c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 15:13:58 -0600 Subject: [PATCH 099/364] Refactor NotifierBuilder into its own mixin This allows for easier unit testing --- flake8/plugins/manager.py | 24 +++++++++++++--------- tests/unit/test_plugin_type_manager.py | 28 ++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 10 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 8d5d12c..f73d006 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -253,16 +253,8 @@ class PluginTypeManager(object): list(self.manager.map(call_provide_options)) -class Checkers(PluginTypeManager): - """All of the checkers registered through entry-ponits.""" - - namespace = 'flake8.extension' - - -class Listeners(PluginTypeManager): - """All of the listeners registered through entry-points.""" - - namespace = 'flake8.listen' +class NotifierBuilder(object): + """Mixin class that builds a Notifier from a PluginManager.""" def build_notifier(self): """Build a Notifier for our Listeners. @@ -279,6 +271,18 @@ class Listeners(PluginTypeManager): return notifier_trie +class Checkers(PluginTypeManager): + """All of the checkers registered through entry-ponits.""" + + namespace = 'flake8.extension' + + +class Listeners(PluginTypeManager, NotifierBuilder): + """All of the listeners registered through entry-points.""" + + namespace = 'flake8.listen' + + class ReportFormatters(PluginTypeManager): """All of the report formatters registered through entry-points.""" diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index cb121c6..2735dec 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -165,3 +165,31 @@ def test_provide_options(PluginManager): plugin.provide_options.assert_called_with(optmanager, options, extra_args) + + +class FakePluginTypeManager(manager.NotifierBuilder): + """Provide an easy way to test the NotifierBuilder.""" + + def __init__(self, manager): + """Initialize with our fake manager.""" + self.names = sorted(manager.keys()) + self.manager = manager + + +@pytest.fixture +def notifier_builder(): + """Create a fake plugin type manager.""" + return FakePluginTypeManager(manager={ + 'T100': object(), + 'T101': object(), + 'T110': object(), + }) + + +def test_build_notifier(notifier_builder): + """Verify we properly build a Notifier object.""" + notifier = notifier_builder.build_notifier() + for name in ('T100', 'T101', 'T110'): + assert list(notifier.listeners_for(name)) == [ + notifier_builder.manager[name] + ] From 5037fab13242a2feb48811f2e71be7e1a69c5214 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 15:16:18 -0600 Subject: [PATCH 100/364] Update design document --- DESIGN.rst | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/DESIGN.rst b/DESIGN.rst index e472671..cac20f9 100644 --- a/DESIGN.rst +++ b/DESIGN.rst @@ -81,7 +81,7 @@ What we *might* want is for a autofix plugin to register something like :: - 'flake8.autofix_extension': [ + 'flake8.listen': [ 'E1 = my_fixer.E1Listener', 'E2 = my_fixer.E2Listener', ] @@ -100,20 +100,15 @@ compose formatters would allow for certain formatters to highlight more important information over less important information as the user deems necessary. +:: + + 'flake8.format': [ + 'json = my_formatter.JsonFormatter', + 'xml = my_formatter.XMLFormatter', + ] + See https://gitlab.com/pycqa/flake8/issues/66 -.. _report-generation: - -Support for Report Generation -+++++++++++++++++++++++++++++ - -Flake8 should support pluggable report formats. See also pluggable report -formats for https://github.com/openstack/bandit - -Report generation plugins may also choose to implement a way to store previous -runs of flake8. As such these plugins should be designed to be composable as -well. - .. _options-passing: Support for Plugins Require Parsed Options From 551bce11a3e7ac328a3859faa0ab854c719863c1 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 15:19:44 -0600 Subject: [PATCH 101/364] Add extend_default_ignore list to OptionManager --- flake8/options/manager.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 1dd6c8d..da3f73e 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -156,6 +156,7 @@ class OptionManager(object): self.program_name = prog self.version = version self.registered_plugins = set() + self.extended_default_ignore = set() @staticmethod def format_plugin(plugin_tuple): @@ -182,6 +183,15 @@ class OptionManager(object): self.config_options_dict[option.config_name] = option LOG.debug('Registered option "%s".', option) + def extend_default_ignore(self, error_codes): + """Extend the default ignore list with the error codes provided. + + :param list error_codes: + List of strings that are the error/warning codes with which to + extend the default ignore list. + """ + self.extended_default_ignore.update(error_codes) + def generate_versions(self, format_str='%(name)s: %(version)s'): """Generate a comma-separated list of versions of plugins.""" return ', '.join( From 3a490971b2984b52bcc56cc7eb74c275c3fbe5b7 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 22:13:06 -0600 Subject: [PATCH 102/364] Add --config option flag --- flake8/main/cli.py | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 05af72e..7f58fb2 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -127,11 +127,6 @@ def register_default_options(option_manager): # Config file options - add_option( - '--isolated', default=False, action='store_true', - help='Ignore all found configuration files.', - ) - add_option( '--append-config', action='append', help='Provide extra config files to parse in addition to the files ' @@ -140,6 +135,18 @@ def register_default_options(option_manager): 'provide the same option.', ) + add_option( + '--config', default=None, + help='Path to the config file that will be the authoritative config ' + 'source. This will cause Flake8 to ignore all other ' + 'configuration files.' + ) + + add_option( + '--isolated', default=False, action='store_true', + help='Ignore all found configuration files.', + ) + def main(argv=None): """Main entry-point for the flake8 command-line tool.""" From 51bfc6a5b1129133df4944d612f12342cbeb495a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 22:13:21 -0600 Subject: [PATCH 103/364] Pass arglist to first parse_args call for testing --- flake8/options/aggregator.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/options/aggregator.py b/flake8/options/aggregator.py index 863fa05..380b2d7 100644 --- a/flake8/options/aggregator.py +++ b/flake8/options/aggregator.py @@ -17,7 +17,7 @@ def aggregate_options(manager, arglist=None, values=None): default_values, _ = manager.parse_args([], values=values) # Get original CLI values so we can find additional config file paths and # see if --config was specified. - original_values, original_args = manager.parse_args() + original_values, original_args = manager.parse_args(arglist) extra_config_files = utils.normalize_paths(original_values.append_config) # Make our new configuration file mergerator From a66f43bc393e4d68b879a6b12c025d59799a5484 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 22:13:57 -0600 Subject: [PATCH 104/364] Extend the default ignore list with plugins' defaults --- flake8/options/aggregator.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/flake8/options/aggregator.py b/flake8/options/aggregator.py index 380b2d7..e12952a 100644 --- a/flake8/options/aggregator.py +++ b/flake8/options/aggregator.py @@ -31,6 +31,12 @@ def aggregate_options(manager, arglist=None, values=None): parsed_config = config_parser.parse(original_values.config, original_values.isolated) + # Extend the default ignore value with the extended default ignore list, + # registered by plugins. + extended_default_ignore = manager.extended_default_ignore.copy() + extended_default_ignore.update(default_values.ignore) + default_values.ignore = list(extended_default_ignore) + # Merge values parsed from config onto the default values returned for config_name, value in parsed_config.items(): dest_name = config_name From 2cba89c56891d3270c8c36e4236d57a468ccbab2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 22:14:21 -0600 Subject: [PATCH 105/364] Log extended default values and add a test --- flake8/options/manager.py | 1 + tests/unit/test_option_manager.py | 10 ++++++++++ 2 files changed, 11 insertions(+) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index da3f73e..9faaf7c 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -190,6 +190,7 @@ class OptionManager(object): List of strings that are the error/warning codes with which to extend the default ignore list. """ + LOG.debug('Extending default ignore list with %r', error_codes) self.extended_default_ignore.update(error_codes) def generate_versions(self, format_str='%(name)s: %(version)s'): diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 660b1a6..99c5d18 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -184,3 +184,13 @@ def test_generate_epilog(optmanager): optmanager.generate_epilog() assert optmanager.parser.epilog == expected_value + + +def test_extend_default_ignore(optmanager): + """Verify that we update the extended default ignore list.""" + assert optmanager.extended_default_ignore == set() + + optmanager.extend_default_ignore(['T100', 'T101', 'T102']) + assert optmanager.extended_default_ignore == set(['T100', + 'T101', + 'T102']) From 17a7826dc7e878a0da3fb8b4853ea80c9967ba24 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 24 Jan 2016 22:14:36 -0600 Subject: [PATCH 106/364] Add integration tests for aggregate_options --- tests/integration/test_aggregator.py | 48 ++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 tests/integration/test_aggregator.py diff --git a/tests/integration/test_aggregator.py b/tests/integration/test_aggregator.py new file mode 100644 index 0000000..3f64fa3 --- /dev/null +++ b/tests/integration/test_aggregator.py @@ -0,0 +1,48 @@ +"""Test aggregation of config files and command-line options.""" +import os + +import pytest + +from flake8.main import cli +from flake8.options import aggregator +from flake8.options import manager + +CLI_SPECIFIED_CONFIG = 'tests/fixtures/config_files/cli-specified.ini' + + +@pytest.fixture +def optmanager(): + """Create a new OptionManager.""" + option_manager = manager.OptionManager( + prog='flake8', + version='3.0.0', + ) + cli.register_default_options(option_manager) + return option_manager + + +def test_aggregate_options_with_config(optmanager): + """Verify we aggregate options and config values appropriately.""" + arguments = ['flake8', '--config', CLI_SPECIFIED_CONFIG, '--select', + 'E11,E34,E402,W,F', '--exclude', 'tests/*'] + options, args = aggregator.aggregate_options(optmanager, arguments) + + assert options.config == CLI_SPECIFIED_CONFIG + assert options.select == ['E11', 'E34', 'E402', 'W', 'F'] + assert options.ignore == ['E123', 'W234', 'E111'] + assert options.exclude == [os.path.abspath('tests/*')] + + +def test_aggregate_options_when_isolated(optmanager): + """Verify we aggregate options and config values appropriately.""" + arguments = ['flake8', '--isolated', '--select', 'E11,E34,E402,W,F', + '--exclude', 'tests/*'] + optmanager.extend_default_ignore(['E8']) + options, args = aggregator.aggregate_options(optmanager, arguments) + + assert options.isolated is True + assert options.select == ['E11', 'E34', 'E402', 'W', 'F'] + assert sorted(options.ignore) == [ + 'E121', 'E123', 'E126', 'E226', 'E24', 'E704', 'E8', + ] + assert options.exclude == [os.path.abspath('tests/*')] From f2ca91372aec1a1ca8f13f00df7bd3013b613830 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 25 Jan 2016 20:40:33 -0600 Subject: [PATCH 107/364] Start structuring documentation --- dev/.keep | 0 docs/source/index.rst | 23 ++++++++++++++++++----- internal/.keep | 0 user/.keep | 0 4 files changed, 18 insertions(+), 5 deletions(-) create mode 100644 dev/.keep create mode 100644 internal/.keep create mode 100644 user/.keep diff --git a/dev/.keep b/dev/.keep new file mode 100644 index 0000000..e69de29 diff --git a/docs/source/index.rst b/docs/source/index.rst index 904ca87..3557d34 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,14 +3,28 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Welcome to flake8's documentation! -================================== +Flake8: Your Tool For Style Guide Enforcement +============================================= -Contents: +User Guide +---------- .. toctree:: - :maxdepth: 2 + :maxdepth: 2 +Plugin Developer Guide +---------------------- + +.. toctree:: + :maxdepth: 2 + + internal/option_handling.rst + +Developer Guide +--------------- + +.. toctree:: + :maxdepth: 2 Indices and tables @@ -19,4 +33,3 @@ Indices and tables * :ref:`genindex` * :ref:`modindex` * :ref:`search` - diff --git a/internal/.keep b/internal/.keep new file mode 100644 index 0000000..e69de29 diff --git a/user/.keep b/user/.keep new file mode 100644 index 0000000..e69de29 From 6590fab3f2dc168e8f3e35cec0afa2e8d108359a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 25 Jan 2016 21:15:04 -0600 Subject: [PATCH 108/364] Move docs directories into docs/source --- {dev => docs/source/dev}/.keep | 0 {internal => docs/source/internal}/.keep | 0 {user => docs/source/user}/.keep | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename {dev => docs/source/dev}/.keep (100%) rename {internal => docs/source/internal}/.keep (100%) rename {user => docs/source/user}/.keep (100%) diff --git a/dev/.keep b/docs/source/dev/.keep similarity index 100% rename from dev/.keep rename to docs/source/dev/.keep diff --git a/internal/.keep b/docs/source/internal/.keep similarity index 100% rename from internal/.keep rename to docs/source/internal/.keep diff --git a/user/.keep b/docs/source/user/.keep similarity index 100% rename from user/.keep rename to docs/source/user/.keep From 7898b5c12efcd9accfd71da5b1a0cab0df8584eb Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 25 Jan 2016 22:05:46 -0600 Subject: [PATCH 109/364] Add documentation around Option Management --- docs/source/conf.py | 2 +- docs/source/index.rst | 2 +- docs/source/internal/option_handling.rst | 133 +++++++++++++++++++++++ 3 files changed, 135 insertions(+), 2 deletions(-) create mode 100644 docs/source/internal/option_handling.rst diff --git a/docs/source/conf.py b/docs/source/conf.py index 835a673..991579f 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -292,4 +292,4 @@ texinfo_documents = [ # Example configuration for intersphinx: refer to the Python standard library. -intersphinx_mapping = {'https://docs.python.org/': None} +intersphinx_mapping = {'python': ('https://docs.python.org/3.4', None)} diff --git a/docs/source/index.rst b/docs/source/index.rst index 3557d34..0263664 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,7 +18,7 @@ Plugin Developer Guide .. toctree:: :maxdepth: 2 - internal/option_handling.rst + internal/option_handling Developer Guide --------------- diff --git a/docs/source/internal/option_handling.rst b/docs/source/internal/option_handling.rst new file mode 100644 index 0000000..f81a4be --- /dev/null +++ b/docs/source/internal/option_handling.rst @@ -0,0 +1,133 @@ +Option and Configuration Handling +================================= + +Option Management +----------------- + +Command-line options are often also set in configuration files for Flake8. +While not all options are meant to be parsed from configuration files, many +default options are also parsed from configuration files as are most plugin +options. + +In Flake8 2, plugins received a :class:`optparse.OptionParser` instance and +called :meth:`optparse.OptionParser.add_option` to register options. If the +plugin author also wanted to have that option parsed from config files they +also had to do something like: + +.. code-block:: python + + parser.config_options.append('my_config_option') + parser.config_options.extend(['config_opt1', 'config_opt2']) + +This was previously undocumented and led to a lot of confusion as to why +registered options were not automatically parsed from configuration files. + +Since Flake8 3 was rewritten from scratch, we decided to take a different +approach to configuration file parsing. Instead of needing to know about an +undocumented attribute that pep8 looks for, Flake8 3 now accepts a parameter +to ``add_option``, specifically ``parse_from_config`` which is a boolean +value. + +Flake8 does this by creating its own abstractions on top of :mod:`optparse`. +The first abstraction is the :class:`flake8.options.manager.Option` class. The +second is the :class:`flake8.options.manager.OptionManager`. In fact, we add +three new parameters: + +- ``parse_from_config`` + +- ``comma_separated_list`` + +- ``normalize_paths`` + +The last two are not specifically for configuration file handling, but they +do improve that dramatically. We found that there were options that when +specified in a configuration file, lended themselves to being split across +multiple lines and those options were almost always comma-separated. For +example, let's consider a user's list of ignored error codes for a project: + +.. code-block:: ini + + [flake8] + ignore = + E111, # Reasoning + E711, # Reasoning + E712, # Reasoning + E121, # Reasoning + E122, # Reasoning + E123, # Reasoning + E131, # Reasoning + E251 # Reasoning + +It makes sense here to allow users to specify the value this way, but, the +standard libary's :class:`configparser.RawConfigParser` class does returns a +string that looks like + +.. code-block:: python + + "\nE111, \nE711, \nE712, \nE121, \nE122, \nE123, \nE131, \nE251 " + +This means that a typical call to :meth:`str.split` with ``','`` will not be +sufficient here. Telling Flake8 that something is a comma-separated list +(e.g., ``comma_separated_list=True``) will handle this for you. Flake8 will +return: + +.. code-block:: python + + ["E111", "E711", "E712", "E121", "E122", "E123", "E131", "E251"] + +Next let's look at how users might like to specify their ``exclude`` list. +Presently OpenStack's Nova project has this line in their `tox.ini`_: + +.. code-block:: ini + + exclude = .venv,.git,.tox,dist,doc,*openstack/common/*,*lib/python*,*egg,build,tools/xenserver*,releasenotes + +I think we can all agree that this would be easier to read like this: + +.. code-block:: ini + + exclude = + .venv, + .git, + .tox, + dist, + doc, + *openstack/common/*, + *lib/python*, + *egg, + build, + tools/xenserver*, + releasenotes + +In this case, since these are actually intended to be paths, we would specify +both ``comma_separated_list=True`` and ``normalize_paths=True`` because we +want the paths to be provided to us with some consistency (either all absolute +paths or not). + +Now let's look at how this would actually be utilized. Most plugin developers +will receive an instance of :class:`~flake8.options.manager.OptionManager` so +to ease the transition we kept the same API as the +:class:`optparse.OptionParser` object. The only difference is that +:meth:`~flake8.options.manager.OptionManager.add_option` accepts the three +extra arguments we highlighted above. + +.. _tox.ini: + https://github.com/openstack/nova/blob/3eb190c4cfc0eefddac6c2cc1b94a699fb1687f8/tox.ini#L155 + +Configuration File Management +----------------------------- + +.. todo:: Add notes about Config File Management + +API Documentation +----------------- + +.. autoclass:: flake8.options.manager.Option + :members: __init__, normalize, to_optparse + +.. autoclass:: flake8.options.manager.OptionManager + :members: + :special-members: + +.. autoclass:: flake8.options.config.MergedConfigParser + :members: From 938d2c450aca196e3b20996f2a17c5913d5a9a89 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 26 Jan 2016 08:40:45 -0600 Subject: [PATCH 110/364] Switch to the RTD theme --- docs/source/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 991579f..0d10af2 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -115,7 +115,7 @@ todo_include_todos = True # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. -html_theme = 'alabaster' +html_theme = 'sphinx_rtd_theme' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the From f59152cc01d7da57a279ccdb1e14550b3573f607 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 26 Jan 2016 08:40:56 -0600 Subject: [PATCH 111/364] Start documenting configuration parsing --- docs/source/internal/option_handling.rst | 64 +++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/docs/source/internal/option_handling.rst b/docs/source/internal/option_handling.rst index f81a4be..01b3862 100644 --- a/docs/source/internal/option_handling.rst +++ b/docs/source/internal/option_handling.rst @@ -117,7 +117,64 @@ extra arguments we highlighted above. Configuration File Management ----------------------------- -.. todo:: Add notes about Config File Management +In Flake8 2, configuration file discovery and management was handled by pep8. +In pep8's 1.6 release series, it drastically broke how discovery and merging +worked (as a result of trying to improve it). To avoid a dependency breaking +Flake8 again in the future, we have created our own discovery and management. +As part of managing this ourselves, we decided to change management/discovery +for 3.0.0. We have done the following: + +- User files (files stored in a user's home directory or in the XDG directory + inside their home directory) are the first files read. For example, if the + user has a ``~/.flake8`` file, we will read that first. + +- Project files (files stored in the current directory) are read next and + merged on top of the user file. In other words, configuration in project + files takes precedence over configuration in user files. + +- **New in 3.0.0** The user can specify ``--append-config `` + repeatedly to include extra configuration files that should be read and + take precedence over user and project files. + +- **New in 3.0.0** The user can specify ``--config `` to so this + file is the only configuration file used. This is a change from Flake8 2 + where pep8 would simply merge this configuration file into the configuration + generated by user and project files (where this takes precedence). + +- **New in 3.0.0** The user can specify ``--isolated`` to disable + configuration via discovered configuration files. + +To facilitate the configuration file management, we've taken a different +approach to discovery and management of files than pep8. In pep8 1.5, 1.6, and +1.7 configuration discovery and management was centralized in `66 lines of +very terse python`_ which was confusing and not very explicit. The terseness +of this function (Flake8's authors believe) caused the confusion and problems +with pep8's 1.6 series. As such, Flake8 has separated out discovery, +management, and merging into a module to make reasoning about each of these +pieces easier and more explicit (as well as easier to test). + +Configuration file discovery is managed by the +:class:`~flake8.options.config.ConfigFileFinder` object. This object needs to +know information about the program's name, any extra arguments passed to it, +and any configuration files that should be appended to the list of discovered +files. It provides methods for finding the files and similiar methods for +parsing those fles. For example, it provides +:meth:`~flake8.options.config.ConfigFileFinder.local_config_files` to find +known local config files (and append the extra configuration files) and it +also provides :meth:`~flake8.options.config.ConfigFileFinder.local_configs` +to parse those configuration files. + +.. note:: ``local_config_files`` also filters out non-existent files. + +Configuration file merging and managemnt is controlled by the +:class:`~flake8.options.config.MergedConfigParser`. This requires the instance +of :class:`~flake8.options.manager.OptionManager` that the program is using, +the list of appended config files, and the list of extra arguments. + +.. todo:: Describe how the MergedConfigParser parses and merges config options + +.. _66 lines of very terse python: + https://github.com/PyCQA/pep8/blob/b8088a2b6bc5b76bece174efad877f764529bc74/pep8.py#L1981..L2047 API Documentation ----------------- @@ -129,5 +186,10 @@ API Documentation :members: :special-members: +.. autoclass:: flake8.options.config.ConfigFileFinder + :members: + :special-members: + .. autoclass:: flake8.options.config.MergedConfigParser :members: + :special-members: From 4cf48b9b772c558363340d64bacb35a867c945c6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 27 Jan 2016 21:32:40 -0600 Subject: [PATCH 112/364] Add documentation about option aggregation Add logging around default ignore lists --- docs/source/internal/option_handling.rst | 34 ++++++++++++++++++++++-- flake8/options/aggregator.py | 20 +++++++++++++- 2 files changed, 51 insertions(+), 3 deletions(-) diff --git a/docs/source/internal/option_handling.rst b/docs/source/internal/option_handling.rst index 01b3862..8ab1911 100644 --- a/docs/source/internal/option_handling.rst +++ b/docs/source/internal/option_handling.rst @@ -169,9 +169,37 @@ to parse those configuration files. Configuration file merging and managemnt is controlled by the :class:`~flake8.options.config.MergedConfigParser`. This requires the instance of :class:`~flake8.options.manager.OptionManager` that the program is using, -the list of appended config files, and the list of extra arguments. +the list of appended config files, and the list of extra arguments. This +object is currently the sole user of the +:class:`~flake8.options.config.ConfigFileFinder` object. It appropriately +initializes the object and uses it in each of -.. todo:: Describe how the MergedConfigParser parses and merges config options +- :meth:`~flake8.options.config.MergedConfigParser.parse_cli_config` +- :meth:`~flake8.options.config.MergedConfigParser.parse_local_config` +- :meth:`~flake8.options.config.MergedConfigParser.parse_user_config` + +Finally, +:meth:`~flake8.options.config.MergedConfigParser.merge_user_and_local_config` +takes the user and local configuration files that are parsed by +:meth:`~flake8.options.config.MergedConfigParser.parse_local_config` and +:meth:`~flake8.options.config.MergedConfigParser.parse_user_config`. The +main usage of the ``MergedConfigParser`` is in +:func:`~flake8.options.aggregator.aggregate_options`. + +Aggregating Configuration File and Command Line Arguments +--------------------------------------------------------- + +:func:`~flake8.options.aggregator.aggregate_options` accepts an instance of +:class:`~flake8.options.maanger.OptionManager` and does the work to parse the +command-line arguments passed by the user necessary for creating an instance +of :class:`~flake8.options.config.MergedConfigParser`. + +After parsing the configuration file, we determine the default ignore list. We +use the defaults from the OptionManager and update those with the parsed +configuration files. Finally we parse the user-provided options one last time +using the option defaults and configuration file values as defaults. The +parser merges on the command-line specified arguments for us so we have our +final, definitive, aggregated options. .. _66 lines of very terse python: https://github.com/PyCQA/pep8/blob/b8088a2b6bc5b76bece174efad877f764529bc74/pep8.py#L1981..L2047 @@ -179,6 +207,8 @@ the list of appended config files, and the list of extra arguments. API Documentation ----------------- +.. autofunction:: flake8.options.aggregator.aggregate_options + .. autoclass:: flake8.options.manager.Option :members: __init__, normalize, to_optparse diff --git a/flake8/options/aggregator.py b/flake8/options/aggregator.py index e12952a..5c02730 100644 --- a/flake8/options/aggregator.py +++ b/flake8/options/aggregator.py @@ -12,7 +12,22 @@ LOG = logging.getLogger(__name__) def aggregate_options(manager, arglist=None, values=None): - """Function that aggregates the CLI and Config options.""" + """Aggregate and merge CLI and config file options. + + :param flake8.option.manager.OptionManager manager: + The instance of the OptionManager that we're presently using. + :param list arglist: + The list of arguments to pass to ``manager.parse_args``. In most cases + this will be None so ``parse_args`` uses ``sys.argv``. This is mostly + available to make testing easier. + :param optparse.Values values: + Previously parsed set of parsed options. + :returns: + Tuple of the parsed options and extra arguments returned by + ``manager.parse_args``. + :rtype: + tuple(optparse.Values, list) + """ # Get defaults from the option parser default_values, _ = manager.parse_args([], values=values) # Get original CLI values so we can find additional config file paths and @@ -34,8 +49,11 @@ def aggregate_options(manager, arglist=None, values=None): # Extend the default ignore value with the extended default ignore list, # registered by plugins. extended_default_ignore = manager.extended_default_ignore.copy() + LOG.debug('Extended default ignore list: %s', + list(extended_default_ignore)) extended_default_ignore.update(default_values.ignore) default_values.ignore = list(extended_default_ignore) + LOG.debug('Merged default ignore list: %s', default_values.ignore) # Merge values parsed from config onto the default values returned for config_name, value in parsed_config.items(): From f4f68b6442e26a29a86cb7bcd3a3b2cfe01438d6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 31 Jan 2016 23:59:53 -0600 Subject: [PATCH 113/364] Document our plugin handling --- docs/source/index.rst | 1 + docs/source/internal/plugin_handling.rst | 73 ++++++++++++++++++++++++ flake8/plugins/manager.py | 2 +- 3 files changed, 75 insertions(+), 1 deletion(-) create mode 100644 docs/source/internal/plugin_handling.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 0263664..4a50a5d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -19,6 +19,7 @@ Plugin Developer Guide :maxdepth: 2 internal/option_handling + internal/plugin_handling Developer Guide --------------- diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst new file mode 100644 index 0000000..547b5ca --- /dev/null +++ b/docs/source/internal/plugin_handling.rst @@ -0,0 +1,73 @@ +Plugin Handling +=============== + +Plugin Management +----------------- + +Flake8 3.0 added support for two other plugins besides those which define new +checks. It now supports: + +- extra checks + +- alternative report formatters + +- listeners to auto-correct violations of checks + +To facilitate this, Flake8 needed a more mature way of managing plugins. As +such, we developed the |PluginManager| which accepts a namespace and will load +the plugins for that namespace. A |PluginManager| creates and manages many +|Plugin| instances. + +A |Plugin| lazily loads the underlying entry-point provided by setuptools. +The entry-point will be loaded either by calling +:meth:`~flake8.plugins.manager.Plugin.load_plugin` or accessing the ``plugin`` +attribute. We also use this abstraction to retrieve options that the plugin +wishes to register and parse. + +The only public method that the |PluginManager| provides is +:meth:`~flake8.plugins.manager.PluginManager.map`. This will accept a function +(or other callable) and call it with each plugin as the first parameter. + +We build atop the |PluginManager| with the |PTM|. It is expected that users of +the |PTM| will subclass it and specify the ``namespace``, e.g., + +.. code-block:: python + + class ExamplePluginType(flake8.plugin.manager.PluginTypeManager): + namespace = 'example-plugins' + +This provides a few extra methods via the |PluginManager|'s ``map`` method. + +Finally, we create three classes of plugins: + +- :class:`~flake8.plugins.manager.Checkers` + +- :class:`~flake8.plugins.manager.Listeners` + +- :class:`~flake8.plugins.manager.ReportFormatters` + +These are used to interact with each of the types of plugins individually. + +API Documentation +----------------- + +.. autoclass:: flake8.plugins.manager.PluginManager + :members: + :special-members: __init__, __contains__, __getitem__ + +.. autoclass:: flake8.plugins.manager.Plugin + :members: + :special-members: __init__ + +.. autoclass:: flake8.plugins.manager.PluginTypeManager + :members: + +.. autoclass:: flake8.plugins.manager.Checkers + +.. autoclass:: flake8.plugins.manager.Listeners + +.. autoclass:: flake8.plugins.manager.ReportFormatters + +.. |PluginManager| replace:: :class:`~flake8.plugins.manager.PluginManager` +.. |Plugin| replace:: :class:`~flake8.plugins.manager.Plugin` +.. |PTM| replace:: :class:`~flake8.plugins.manager.PluginTypeManager` diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index f73d006..570c1d0 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -171,7 +171,7 @@ class PluginManager(object): LOG.info('Loaded %r for plugin "%s".', self.plugins[name], name) def map(self, func, *args, **kwargs): - """Call ``func`` with the plugin and *args and **kwargs after. + r"""Call ``func`` with the plugin and \*args and \**kwargs after. This yields the return value from ``func`` for each plugin. From 51d11d1a308e72f5e460d77bf37f98c818dd650f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 1 Feb 2016 00:00:29 -0600 Subject: [PATCH 114/364] Note inspiration --- docs/source/internal/plugin_handling.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst index 547b5ca..d69957d 100644 --- a/docs/source/internal/plugin_handling.rst +++ b/docs/source/internal/plugin_handling.rst @@ -48,6 +48,11 @@ Finally, we create three classes of plugins: These are used to interact with each of the types of plugins individually. +.. note:: + + Our inspiration for our plugin handling comes from the author's extensive + experience with ``stevedore``. + API Documentation ----------------- From 5dc7440a2ba228bc6d7b693ff4737d3a2453e965 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 1 Feb 2016 20:09:15 -0600 Subject: [PATCH 115/364] Start documenting our Trie implementation --- docs/source/internal/plugin_handling.rst | 3 +++ flake8/{ => plugins}/_trie.py | 0 flake8/plugins/manager.py | 2 +- flake8/{ => plugins}/notifier.py | 2 +- tests/unit/test_notifier.py | 2 +- tests/unit/test_trie.py | 2 +- 6 files changed, 7 insertions(+), 4 deletions(-) rename flake8/{ => plugins}/_trie.py (100%) rename flake8/{ => plugins}/notifier.py (97%) diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst index d69957d..e217497 100644 --- a/docs/source/internal/plugin_handling.rst +++ b/docs/source/internal/plugin_handling.rst @@ -53,6 +53,9 @@ These are used to interact with each of the types of plugins individually. Our inspiration for our plugin handling comes from the author's extensive experience with ``stevedore``. +Notifying Listener Plugins +-------------------------- + API Documentation ----------------- diff --git a/flake8/_trie.py b/flake8/plugins/_trie.py similarity index 100% rename from flake8/_trie.py rename to flake8/plugins/_trie.py diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 570c1d0..4a06c75 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -4,8 +4,8 @@ import logging import pkg_resources -from flake8 import notifier from flake8 import exceptions +from flake8.plugins import notifier LOG = logging.getLogger(__name__) diff --git a/flake8/notifier.py b/flake8/plugins/notifier.py similarity index 97% rename from flake8/notifier.py rename to flake8/plugins/notifier.py index e7bf3ba..323ea1d 100644 --- a/flake8/notifier.py +++ b/flake8/plugins/notifier.py @@ -1,5 +1,5 @@ """Implementation of the class that registers and notifies listeners.""" -from flake8 import _trie +from flake8.plugins import _trie class Notifier(object): diff --git a/tests/unit/test_notifier.py b/tests/unit/test_notifier.py index 84b8a45..effcc88 100644 --- a/tests/unit/test_notifier.py +++ b/tests/unit/test_notifier.py @@ -1,7 +1,7 @@ """Unit tests for the Notifier object.""" import pytest -from flake8 import notifier +from flake8.plugins import notifier class _Listener(object): diff --git a/tests/unit/test_trie.py b/tests/unit/test_trie.py index 4cbd8c5..152b5b6 100644 --- a/tests/unit/test_trie.py +++ b/tests/unit/test_trie.py @@ -1,5 +1,5 @@ """Unit test for the _trie module.""" -from flake8 import _trie as trie +from flake8.plugins import _trie as trie class TestTrie(object): From 4dc1cc386d91caa381ef9ca98eba309d23e070f9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 1 Feb 2016 20:33:18 -0600 Subject: [PATCH 116/364] Add documentation around plugin notifications and pyflakes --- docs/source/internal/plugin_handling.rst | 31 ++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst index e217497..e430bfb 100644 --- a/docs/source/internal/plugin_handling.rst +++ b/docs/source/internal/plugin_handling.rst @@ -56,6 +56,33 @@ These are used to interact with each of the types of plugins individually. Notifying Listener Plugins -------------------------- +One of the interesting challenges with allowing plugins to be notified each +time an error or warning is emitted by a checker is finding listeners quickly +and efficiently. It makes sense to allow a listener to listen for a certain +class of warnings or just a specific warning. As such, we need to allow all +plugins that listen to a specific warning or class to be notified. For +example, someone might register a listener for ``E1`` and another for ``E111`` +if ``E111`` is triggered by the code, both listeners should be notified. +If ``E112`` is returned, then only ``E1`` (and any other listeners) would be +notified. + +To implement this goal, we needed an object to store listeners in that would +allow for efficient look up - a Trie (or Prefix Tree). Given that none of the +existing packages on PyPI allowed for storing data on each node of the trie, +it was left up to write our own as :class:`~flake8.plugins._trie.Trie`. On +top of that we layer our :class:`~flake8.plugins.notifier.Notifier` class. + +Now when Flake8 receives an error or warning, we can easily call the +:meth:`~flake8.plugins.notifier.Notifier.notify` method and let plugins act on +that knowledge. + +Default Plugins +--------------- + +Finally, Flake8 has always provided its own plugin shim for Pyflakes. As part +of that we carry our own shim in-tree and now store that in +:mod:`flake8.plugins.pyflakes`. + API Documentation ----------------- @@ -76,6 +103,10 @@ API Documentation .. autoclass:: flake8.plugins.manager.ReportFormatters +.. autoclass:: flake8.plugins.notifier.Notifier + +.. autoclass:: flake8.plugins._trie.Trie + .. |PluginManager| replace:: :class:`~flake8.plugins.manager.PluginManager` .. |Plugin| replace:: :class:`~flake8.plugins.manager.Plugin` .. |PTM| replace:: :class:`~flake8.plugins.manager.PluginTypeManager` From 91327c75e0da54fb7f4479499009fb6946f59820 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 1 Feb 2016 20:55:57 -0600 Subject: [PATCH 117/364] Start work on our StyleGuide implementation --- flake8/style_guide.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index a3f63eb..ec000b7 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -1 +1,25 @@ """Implementation of the StyleGuide used by Flake8.""" + + +class StyleGuide(object): + """Manage a Flake8 user's style guide.""" + + def __init__(self, options, arguments, checker_plugins, listening_plugins, + formatting_plugins): + """Initialize our StyleGuide. + + .. todo:: Add parameter documentation. + """ + pass + + +# Should separate style guide logic from code that runs checks +# StyleGuide should manage select/ignore logic as well as include/exclude +# logic. See also https://github.com/PyCQA/pep8/pull/433 + +# StyleGuide shoud dispatch check execution in a way that can use +# multiprocessing but also retry in serial. See also: +# https://gitlab.com/pycqa/flake8/issues/74 + +# StyleGuide should interface with Reporter and aggregate errors/notify +# listeners From 0645ec3ef771c8e2e6ef39bdd5eb360067d147b9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 2 Feb 2016 16:53:24 -0600 Subject: [PATCH 118/364] Move __contains__ and __getitem__ to proper class --- flake8/plugins/manager.py | 20 ++++++++-------- tests/unit/test_plugin_manager.py | 28 +--------------------- tests/unit/test_plugin_type_manager.py | 33 ++++++++++++++++++++++++++ 3 files changed, 44 insertions(+), 37 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 4a06c75..02a136c 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -152,16 +152,6 @@ class PluginManager(object): self.names = [] self._load_all_plugins() - def __contains__(self, name): - """Check if the entry-point name is in this plugin manager.""" - LOG.debug('Checking for "%s" in plugin manager.', name) - return name in self.plugins - - def __getitem__(self, name): - """Retrieve a plugin by its entry-point name.""" - LOG.debug('Retrieving plugin for "%s".', name) - return self.plugins[name] - def _load_all_plugins(self): LOG.debug('Loading entry-points for "%s".', self.namespace) for entry_point in pkg_resources.iter_entry_points(self.namespace): @@ -203,6 +193,16 @@ class PluginTypeManager(object): self.manager = PluginManager(self.namespace) self.plugins_loaded = False + def __contains__(self, name): + """Check if the entry-point name is in this plugin type manager.""" + LOG.debug('Checking for "%s" in plugin type manager.', name) + return name in self.plugins + + def __getitem__(self, name): + """Retrieve a plugin by its name.""" + LOG.debug('Retrieving plugin for "%s".', name) + return self.plugins[name] + @property def names(self): """Proxy attribute to underlying manager.""" diff --git a/tests/unit/test_plugin_manager.py b/tests/unit/test_plugin_manager.py index 05073cf..8991b96 100644 --- a/tests/unit/test_plugin_manager.py +++ b/tests/unit/test_plugin_manager.py @@ -36,32 +36,6 @@ def test_calls_pkg_resources_creates_plugins_automaticaly(iter_entry_points): assert isinstance(plugin_mgr.plugins['T200'], manager.Plugin) -@mock.patch('pkg_resources.iter_entry_points') -def test_proxies_contains_to_plugins_dictionary(iter_entry_points): - """Verify that we can use the PluginManager like a dictionary.""" - iter_entry_points.return_value = [ - create_entry_point_mock('T100'), - create_entry_point_mock('T200'), - ] - plugin_mgr = manager.PluginManager(namespace='testing.pkg_resources') - - assert 'T100' in plugin_mgr - assert 'T200' in plugin_mgr - - -@mock.patch('pkg_resources.iter_entry_points') -def test_proxies_getitem_to_plugins_dictionary(iter_entry_points): - """Verify that we can use the PluginManager like a dictionary.""" - iter_entry_points.return_value = [ - create_entry_point_mock('T100'), - create_entry_point_mock('T200'), - ] - plugin_mgr = manager.PluginManager(namespace='testing.pkg_resources') - - assert isinstance(plugin_mgr['T100'], manager.Plugin) - assert isinstance(plugin_mgr['T200'], manager.Plugin) - - @mock.patch('pkg_resources.iter_entry_points') def test_handles_mapping_functions_across_plugins(iter_entry_points): """Verify we can use the PluginManager call functions on all plugins.""" @@ -71,6 +45,6 @@ def test_handles_mapping_functions_across_plugins(iter_entry_points): ] iter_entry_points.return_value = entry_point_mocks plugin_mgr = manager.PluginManager(namespace='testing.pkg_resources') - plugins = [plugin_mgr[name] for name in plugin_mgr.names] + plugins = [plugin_mgr.plugins[name] for name in plugin_mgr.names] assert list(plugin_mgr.map(lambda x: x)) == plugins diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index 2735dec..36fb92a 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -35,6 +35,13 @@ def create_mapping_manager_mock(plugins): return manager_mock +def create_manager_with_plugins(plugins): + """Create a fake PluginManager with a plugins dictionary.""" + manager_mock = mock.create_autospec(manager.PluginManager) + manager_mock.plugins = plugins + return manager_mock + + class TestType(manager.PluginTypeManager): """Fake PluginTypeManager.""" @@ -167,6 +174,32 @@ def test_provide_options(PluginManager): extra_args) +@mock.patch('flake8.plugins.manager.PluginManager') +def test_proxy_contains_to_managers_plugins_dict(PluginManager): + """Verify that we proxy __contains__ to the manager's dictionary.""" + plugins = {'T10%i' % i: create_plugin_mock() for i in range(8)} + # Return our PluginManager mock + PluginManager.return_value = create_manager_with_plugins(plugins) + + type_mgr = TestType() + for i in range(8): + key = 'T10%i' % i + assert key in type_mgr + + +@mock.patch('flake8.plugins.manager.PluginManager') +def test_proxies_getitem_to_managers_plugins_dictionary(PluginManager): + """Verify that we can use the PluginTypeManager like a dictionary.""" + plugins = {'T10%i' % i: create_plugin_mock() for i in range(8)} + # Return our PluginManager mock + PluginManager.return_value = create_manager_with_plugins(plugins) + + type_mgr = TestType() + for i in range(8): + key = 'T10%i' % i + assert type_mgr[key] is plugins[key] + + class FakePluginTypeManager(manager.NotifierBuilder): """Provide an easy way to test the NotifierBuilder.""" From eafc91ae6ad5c05b3e036733d73a0edac9070192 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 2 Feb 2016 20:48:26 -0600 Subject: [PATCH 119/364] Add handling and decision making for select and ignore --- flake8/style_guide.py | 106 ++++++++++++++++++++++++++- setup.cfg | 4 ++ setup.py | 17 +++-- tests/unit/test_style_guide.py | 128 +++++++++++++++++++++++++++++++++ 4 files changed, 249 insertions(+), 6 deletions(-) create mode 100644 tests/unit/test_style_guide.py diff --git a/flake8/style_guide.py b/flake8/style_guide.py index ec000b7..ef1cbae 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -1,4 +1,30 @@ """Implementation of the StyleGuide used by Flake8.""" +import enum + +__all__ = ( + 'StyleGuide', +) + + +class Selected(enum.Enum): + """Enum representing an explicitly or implicitly selected code.""" + + Explicitly = 'explicitly selected' + Implicitly = 'implicitly selected' + + +class Ignored(enum.Enum): + """Enum representing an explicitly or implicitly ignored code.""" + + Explicitly = 'explicitly ignored' + Implicitly = 'implicitly ignored' + + +class Decision(enum.Enum): + """Enum representing whether a code should be ignored or selected.""" + + Ignored = 'ignored error' + Selected = 'selected error' class StyleGuide(object): @@ -10,7 +36,85 @@ class StyleGuide(object): .. todo:: Add parameter documentation. """ - pass + self.options = options + self.arguments = arguments + self.checkers = checker_plugins + self.listeners = listening_plugins + self.formatters = formatting_plugins + self._selected = tuple(options.select) + self._ignored = tuple(options.ignore) + self._decision_cache = {} + + def is_user_selected(self, code): + """Determine if the code has been selected by the user. + + :param str code: + The code for the check that has been run. + :returns: + Selected.Implicitly if the selected list is empty, + Selected.Explicitly if the selected list is not empty and a match + was found, + Ignored.Implicitly if the selected list is not empty but no match + was found. + """ + if not self._selected: + return Selected.Implicitly + + if code.startswith(self._selected): + return Selected.Explicitly + + return Ignored.Implicitly + + def is_user_ignored(self, code): + """Determine if the code has been ignored by the user. + + :param str code: + The code for the check that has been run. + :returns: + Selected.Implicitly if the ignored list is empty, + Ignored.Explicitly if the ignored list is not empty and a match was + found, + Selected.Implicitly if the ignored list is not empty but no match + was found. + """ + if self._ignored and code.startswith(self._ignored): + return Ignored.Explicitly + + return Selected.Implicitly + + def _decision_for(self, code): + startswith = code.startswith + selected = sorted([s for s in self._selected if startswith(s)])[0] + ignored = sorted([i for i in self._ignored if startswith(i)])[0] + + if selected.startswith(ignored): + return Decision.Selected + return Decision.Ignored + + def should_report_error(self, code): + """Determine if the error code should be reported or ignored. + + :param str code: + The code for the check that has been run. + """ + decision = self._decision_cache.get(code) + if decision is None: + selected = self.is_user_selected(code) + ignored = self.is_user_ignored(code) + + if ((selected is Selected.Explicitly or + selected is Selected.Implicitly) and + ignored is Selected.Implicitly): + decision = Decision.Selected + elif (selected is Selected.Explicitly and + ignored is Ignored.Explicitly): + decision = self._decision_for(code) + elif (selected is Ignored.Implicitly or + ignored is Ignored.Explicitly): + decision = Decision.Ignored + + self._decision_cache[code] = decision + return decision # Should separate style guide logic from code that runs checks diff --git a/setup.cfg b/setup.cfg index f05f933..e96761b 100644 --- a/setup.cfg +++ b/setup.cfg @@ -3,3 +3,7 @@ test=pytest [bdist_wheel] universal=1 + +[metadata] +requires-dist = + enum34; python_version<"3.4" diff --git a/setup.py b/setup.py index 93e4fdb..0b90cf7 100644 --- a/setup.py +++ b/setup.py @@ -2,6 +2,7 @@ # -*- coding: utf-8 -*- from __future__ import with_statement import setuptools +import sys import flake8 @@ -25,6 +26,16 @@ if mock is None: tests_require += ['mock'] +requires = [ + "pyflakes >= 0.8.1, < 1.1", + "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", + # "mccabe >= 0.2.1, < 0.4", +] + +if sys.version_info < (3, 4): + requires.append("enum34") + + def get_long_description(): """Generate a long description from the README and CHANGES files.""" descr = [] @@ -51,11 +62,7 @@ setuptools.setup( "flake8.options", "flake8.plugins", ], - install_requires=[ - "pyflakes >= 0.8.1, < 1.1", - "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", - # "mccabe >= 0.2.1, < 0.4", - ], + install_requires=requires, entry_points={ 'distutils.commands': ['flake8 = flake8.main:Flake8Command'], 'console_scripts': ['flake8 = flake8.main.cli:main'], diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py new file mode 100644 index 0000000..51e9f61 --- /dev/null +++ b/tests/unit/test_style_guide.py @@ -0,0 +1,128 @@ +"""Tests for the flake8.style_guide.StyleGuide class.""" +import optparse + +from flake8 import style_guide + +import pytest + + +def create_options(**kwargs): + """Create and return an instance of optparse.Values.""" + kwargs.setdefault('select', []) + kwargs.setdefault('ignore', []) + return optparse.Values(kwargs) + + +@pytest.mark.parametrize('ignore_list,error_code', [ + (['E111', 'E121'], 'E111'), + (['E111', 'E121'], 'E121'), + (['E11', 'E12'], 'E121'), + (['E2', 'E12'], 'E121'), + (['E2', 'E12'], 'E211'), +]) +def test_is_user_ignored_ignores_errors(ignore_list, error_code): + """Verify we detect users explicitly ignoring an error.""" + guide = style_guide.StyleGuide(create_options(ignore=ignore_list), + arguments=[], + checker_plugins=None, + listening_plugins=None, + formatting_plugins=None) + + assert guide.is_user_ignored(error_code) is style_guide.Ignored.Explicitly + + +@pytest.mark.parametrize('ignore_list,error_code', [ + (['E111', 'E121'], 'E112'), + (['E111', 'E121'], 'E122'), + (['E11', 'E12'], 'W121'), + (['E2', 'E12'], 'E112'), + (['E2', 'E12'], 'E111'), +]) +def test_is_user_ignored_implicitly_selects_errors(ignore_list, error_code): + """Verify we detect users does not explicitly ignore an error.""" + guide = style_guide.StyleGuide(create_options(ignore=ignore_list), + arguments=[], + checker_plugins=None, + listening_plugins=None, + formatting_plugins=None) + + assert guide.is_user_ignored(error_code) is style_guide.Selected.Implicitly + + +@pytest.mark.parametrize('select_list,error_code', [ + (['E111', 'E121'], 'E111'), + (['E111', 'E121'], 'E121'), + (['E11', 'E12'], 'E121'), + (['E2', 'E12'], 'E121'), + (['E2', 'E12'], 'E211'), +]) +def test_is_user_selected_selects_errors(select_list, error_code): + """Verify we detect users explicitly selecting an error.""" + guide = style_guide.StyleGuide(create_options(select=select_list), + arguments=[], + checker_plugins=None, + listening_plugins=None, + formatting_plugins=None) + + assert (guide.is_user_selected(error_code) is + style_guide.Selected.Explicitly) + + +def test_is_user_selected_implicitly_selects_errors(): + """Verify we detect users implicitly selecting an error.""" + select_list = [] + error_code = 'E121' + guide = style_guide.StyleGuide(create_options(select=select_list), + arguments=[], + checker_plugins=None, + listening_plugins=None, + formatting_plugins=None) + + assert (guide.is_user_selected(error_code) is + style_guide.Selected.Implicitly) + + +@pytest.mark.parametrize('select_list,error_code', [ + (['E111', 'E121'], 'E112'), + (['E111', 'E121'], 'E122'), + (['E11', 'E12'], 'E132'), + (['E2', 'E12'], 'E321'), + (['E2', 'E12'], 'E410'), +]) +def test_is_user_selected_excludes_errors(select_list, error_code): + """Verify we detect users implicitly excludes an error.""" + guide = style_guide.StyleGuide(create_options(select=select_list), + arguments=[], + checker_plugins=None, + listening_plugins=None, + formatting_plugins=None) + + assert guide.is_user_selected(error_code) is style_guide.Ignored.Implicitly + + +@pytest.mark.parametrize('select_list,ignore_list,error_code,expected', [ + (['E111', 'E121'], [], 'E111', style_guide.Decision.Selected), + (['E111', 'E121'], [], 'E112', style_guide.Decision.Ignored), + (['E111', 'E121'], [], 'E121', style_guide.Decision.Selected), + (['E111', 'E121'], [], 'E122', style_guide.Decision.Ignored), + (['E11', 'E12'], [], 'E132', style_guide.Decision.Ignored), + (['E2', 'E12'], [], 'E321', style_guide.Decision.Ignored), + (['E2', 'E12'], [], 'E410', style_guide.Decision.Ignored), + (['E11', 'E121'], ['E1'], 'E112', style_guide.Decision.Selected), + (['E111', 'E121'], ['E2'], 'E122', style_guide.Decision.Ignored), + (['E11', 'E12'], ['E13'], 'E132', style_guide.Decision.Ignored), + (['E1', 'E3'], ['E32'], 'E321', style_guide.Decision.Ignored), + ([], ['E2', 'E12'], 'E410', style_guide.Decision.Selected), + (['E4'], ['E2', 'E12', 'E41'], 'E410', style_guide.Decision.Ignored), + (['E41'], ['E2', 'E12', 'E4'], 'E410', style_guide.Decision.Selected), +]) +def test_should_report_error(select_list, ignore_list, error_code, expected): + """Verify we decide when to report an error.""" + guide = style_guide.StyleGuide(create_options(select=select_list, + ignore=ignore_list), + arguments=[], + checker_plugins=None, + listening_plugins=None, + formatting_plugins=None) + + assert guide.should_report_error(error_code) is expected From 691bdc97eab5676f80d1874657e844c2931c7a7e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 3 Feb 2016 21:11:27 -0600 Subject: [PATCH 120/364] Add option to disable # noqa Sometimes it is nice to be able to ignore what someone has used #noqa on and see what the results would be. This can also be specified in a config file to prevent people from using # noqa in a code base. --- flake8/main/cli.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 7f58fb2..51071b5 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -86,6 +86,13 @@ def register_default_options(option_manager): ' For example, ``--select=E4,E51,W234``. (Default: %default)', ) + add_option( + '--disable-noqa', default=False, parse_from_config=True, + action='store_true', + help='Disable the effect of "# noqa". This will report errors on ' + 'lines with "# noqa" at the end.' + ) + # TODO(sigmavirus24): Decide what to do about --show-pep8 add_option( From 1543ff8bab8b3832d59b4d54360bcf39fd18ce46 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 3 Feb 2016 22:04:08 -0600 Subject: [PATCH 121/364] Add some logging to the style guide --- flake8/style_guide.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index ef1cbae..2177b78 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -1,11 +1,16 @@ """Implementation of the StyleGuide used by Flake8.""" +import logging + import enum __all__ = ( 'StyleGuide', ) +LOG = logging.getLogger(__name__) + +# TODO(sigmavirus24): Determine if we need to use enum/enum34 class Selected(enum.Enum): """Enum representing an explicitly or implicitly selected code.""" @@ -99,8 +104,11 @@ class StyleGuide(object): """ decision = self._decision_cache.get(code) if decision is None: + LOG.debug('Deciding if "%s" should be reported', code) selected = self.is_user_selected(code) ignored = self.is_user_ignored(code) + LOG.debug('The user configured "%s" to be "%s", "%s"', + code, selected, ignored) if ((selected is Selected.Explicitly or selected is Selected.Implicitly) and @@ -114,6 +122,7 @@ class StyleGuide(object): decision = Decision.Ignored self._decision_cache[code] = decision + LOG.debug('"%s" will be "%s"', code, decision) return decision From 253211f5ad09dbaef681f018860fe778026150b5 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 3 Feb 2016 22:30:03 -0600 Subject: [PATCH 122/364] Add caching stdin_get_value function --- flake8/utils.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/flake8/utils.py b/flake8/utils.py index 0dc16e2..dd83748 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -1,5 +1,7 @@ """Utility methods for flake8.""" +import io import os +import sys def parse_comma_separated_list(value): @@ -43,3 +45,16 @@ def normalize_path(path, parent=os.curdir): if '/' in path: path = os.path.abspath(os.path.join(parent, path)) return path.rstrip('/') + + +def stdin_get_value(): + """Get and cache it so plugins can use it.""" + cached_value = getattr(stdin_get_value, 'cached_stdin', None) + if cached_value is None: + stdin_value = sys.stdin.read() + if sys.version_info < (3, 0): + cached_value = io.BytesIO(stdin_value) + else: + cached_value = io.StringIO(stdin_value) + stdin_get_value.cached_stdin = cached_value + return cached_value.getvalue() From a4e051614f851cf4e642bf00991d5952af104723 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 3 Feb 2016 23:11:45 -0600 Subject: [PATCH 123/364] Document flake8.utils --- docs/source/internal/utils.rst | 47 ++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 docs/source/internal/utils.rst diff --git a/docs/source/internal/utils.rst b/docs/source/internal/utils.rst new file mode 100644 index 0000000..69dee45 --- /dev/null +++ b/docs/source/internal/utils.rst @@ -0,0 +1,47 @@ +=================== + Utility Functions +=================== + +Flake8 has a few utility functions that it uses and provides to plugins. + +.. autofunction:: flake8.utils.parse_comma_separated_list + +:func:`~flake8.utils.parse_comma_separated_list` takes either a string like + +.. code-block:: python + + "E121,W123,F904" + "E121,\nW123,\nF804" + "E121,\n\tW123,\n\tF804" + +Or it will take a list of strings (potentially with whitespace) such as + +.. code-block:: python + + [" E121\n", "\t\nW123 ", "\n\tF904\n "] + +And converts it to a list that looks as follows + +.. code-block:: python + + ["E121", "W123", "F904"] + +This function helps normalize any kind of comma-separated input you or Flake8 +might receive. This is most helpful when taking advantage of Flake8's +additional parameters to :class:`~flake8.options.manager.Option`. + +.. autofunction:: flake8.utils.normalize_path + +This utility takes a string that represents a path and returns the absolute +path if the string has a ``/`` in it. It also removes trailing ``/``\ s. + +.. autofunction:: flake8.utils.normalize_paths + +This function utilizes :func:`~flake8.utils.parse_comma_separated_list` and +:func:`~flake8.utils.normalize_path` to normalize it's input to a list of +strings that should be paths. + +.. autofunction:: flake8.utils.stdin_get_value + +This function retrieves and caches the value provided on ``sys.stdin``. This +allows plugins to use this to retrieve ``stdin`` if necessary. From 276f823de66164240cf25c237762d8af6d6c947a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 6 Feb 2016 09:32:38 -0600 Subject: [PATCH 124/364] Add type annotations to flake8.utils for fun --- flake8/utils.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/flake8/utils.py b/flake8/utils.py index dd83748..98e5cea 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -5,6 +5,7 @@ import sys def parse_comma_separated_list(value): + # type: (Union[Sequence[str], str]) -> List[str] """Parse a comma-separated list. :param value: @@ -24,6 +25,7 @@ def parse_comma_separated_list(value): def normalize_paths(paths, parent=os.curdir): + # type: (Union[Sequence[str], str], str) -> List[str] """Parse a comma-separated list of paths. :returns: @@ -35,6 +37,7 @@ def normalize_paths(paths, parent=os.curdir): def normalize_path(path, parent=os.curdir): + # type: (str, str) -> str """Normalize a single-path. :returns: @@ -48,6 +51,7 @@ def normalize_path(path, parent=os.curdir): def stdin_get_value(): + # type: () -> str """Get and cache it so plugins can use it.""" cached_value = getattr(stdin_get_value, 'cached_stdin', None) if cached_value is None: From 5870d136ed6f0891c78993b726c9d12e83bcd737 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 6 Feb 2016 13:51:35 -0600 Subject: [PATCH 125/364] Represent an error with a namedtuple --- flake8/style_guide.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 2177b78..e8451f1 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -1,4 +1,5 @@ """Implementation of the StyleGuide used by Flake8.""" +import collections import logging import enum @@ -32,6 +33,13 @@ class Decision(enum.Enum): Selected = 'selected error' +Error = collections.namedtuple('Error', ['code', + 'filename', + 'line_number', + 'column_number', + 'text']) + + class StyleGuide(object): """Manage a Flake8 user's style guide.""" From 35bb32d94e56e13d4c98c0874b2a6d45b5bc2c91 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 6 Feb 2016 13:52:32 -0600 Subject: [PATCH 126/364] Start working on default formatters for Flake8 --- flake8/formatting/__init__.py | 1 + 1 file changed, 1 insertion(+) create mode 100644 flake8/formatting/__init__.py diff --git a/flake8/formatting/__init__.py b/flake8/formatting/__init__.py new file mode 100644 index 0000000..bf44801 --- /dev/null +++ b/flake8/formatting/__init__.py @@ -0,0 +1 @@ +"""Submodule containing the default formatters for Flake8.""" From 63b50bc1f4dbfb848d24cafd1c4a9e5e379508dc Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 6 Feb 2016 14:36:26 -0600 Subject: [PATCH 127/364] Add more functionality to the BaseFormatter Ensure setuptools knows about flake8.formatting --- flake8/formatting/base.py | 105 ++++++++++++++++++++++++++++++++++++++ setup.py | 1 + 2 files changed, 106 insertions(+) create mode 100644 flake8/formatting/base.py diff --git a/flake8/formatting/base.py b/flake8/formatting/base.py new file mode 100644 index 0000000..85a931c --- /dev/null +++ b/flake8/formatting/base.py @@ -0,0 +1,105 @@ +"""The base class and interface for all formatting plugins.""" +from __future__ import print_function + + +class BaseFormatter(object): + """Class defining the formatter interface. + + .. attribute:: options + + The options parsed from both configuration files and the command-line. + + .. attribute:: filename + + If specified by the user, the path to store the results of the run. + + .. attribute:: output_fd + + Initialized when the :meth:`start` is called. This will be a file + object opened for writing. + + .. attribute:: newline + + The string to add to the end of a line. This is only used when the + output filename has been specified. + """ + + def __init__(self, options): + """Initialize with the options parsed from config and cli. + + This also calls a hook, :meth:`after_init`, so subclasses do not need + to call super to call this method. + + :param optparse.Values options: + User specified configuration parsed from both configuration files + and the command-line interface. + """ + self.options = options + self.filename = options.output_file + self.output_fd = None + self.newline = '\n' + self.after_init() + + def after_init(self): + """Initialize the formatter further.""" + pass + + def start(self): + """Prepare the formatter to receive input. + + This defaults to initializing :attr:`output_fd` if :attr:`filename` + """ + if self.filename: + self.output_fd = open(self.filename, 'w') + + def handle(self, error): + """Handle an error reported by Flake8. + + This defaults to calling :meth:`format` and then :meth:`write`. To + extend how errors are handled, override this method. + + :param error: + This will be an instance of :class:`~flake8.style_guide.Error`. + :type error: + flake8.style_guide.Error + """ + line = self.format(error) + self.write(line) + + def format(self, error): + """Format an error reported by Flake8. + + This method **must** be implemented by subclasses. + + :param error: + This will be an instance of :class:`~flake8.style_guide.Error`. + :type error: + flake8.style_guide.Error + :returns: + The formatted error string. + :rtype: + str + """ + raise NotImplementedError('Subclass of BaseFormatter did not implement' + ' format.') + + def write(self, line): + """Write the line either to the output file or stdout. + + This handles deciding whether to write to a file or print to standard + out for subclasses. Override this if you want behaviour that differs + from the default. + + :param str line: + The formatted string to print or write. + """ + if self.output_fd is not None: + self.output_fd.write(line + self.newline) + else: + print(self.output_fd) + + def stop(self): + """Clean up after reporting is finished.""" + if self.output_fd is not None: + self.output_fd.close() + self.output_fd = None diff --git a/setup.py b/setup.py index 0b90cf7..00b5a68 100644 --- a/setup.py +++ b/setup.py @@ -58,6 +58,7 @@ setuptools.setup( url="https://gitlab.com/pycqa/flake8", packages=[ "flake8", + "flake8.formatting", "flake8.main", "flake8.options", "flake8.plugins", From fe1b628c0f702c2af0303b667aa0504b9b3348bb Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 6 Feb 2016 14:36:44 -0600 Subject: [PATCH 128/364] Add documentation around plugins and formatters --- docs/source/dev/formatters.rst | 16 ++++ docs/source/dev/registering_plugins.rst | 113 ++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 docs/source/dev/formatters.rst create mode 100644 docs/source/dev/registering_plugins.rst diff --git a/docs/source/dev/formatters.rst b/docs/source/dev/formatters.rst new file mode 100644 index 0000000..b96c40b --- /dev/null +++ b/docs/source/dev/formatters.rst @@ -0,0 +1,16 @@ +=========================================== + Developing a Formatting Plugin for Flake8 +=========================================== + +Flake8 added the ability to develop custom formatting plugins in version +3.0.0. Let's write a plugin together: + +.. code-block:: python + + from flake8.formatting import base + + + class Example(base.BaseFormatter): + """Flake8's example formatter.""" + + pass diff --git a/docs/source/dev/registering_plugins.rst b/docs/source/dev/registering_plugins.rst new file mode 100644 index 0000000..1444af0 --- /dev/null +++ b/docs/source/dev/registering_plugins.rst @@ -0,0 +1,113 @@ +================================== + Registering a Plugin with Flake8 +================================== + +To register any kind of plugin with Flake8, you need a few things: + +#. You need a way to install the plugin (whether it is packaged on its own or + as part of something else). In this section, we will use a ``setup.py`` + written for an example plugin. + +#. A name for your plugin that will (ideally) be unique. + +#. A somewhat recent version of setuptools (newer than 0.7.0 but preferably as + recent as you can attain). + +Flake8 presently relies on a functionality provided by setuptools called +`Entry Points`_. These allow any package to register a plugin with Flake8 via +that package's ``setup.py`` file. + +Let's presume that we already have our plugin written and it's in a module +called ``flake8_example``. We might have a ``setup.py`` that looks something +like: + +.. code-block:: python + + from __future__ import with_statement + import setuptools + + requires = [ + "flake8 > 3.0.0", + ] + + flake8_entry_point = # ... + + setuptools.setup( + name="flake8_example", + license="MIT", + version="0.1.0", + description="our extension to flake8", + author="Me", + author_email="example@example.com", + url="https://gitlab.com/me/flake8_example", + packages=[ + "flake8_example", + ], + install_requires=requires, + entry_points={ + flake8_entry_point: [ + 'X = flake8_example.ExamplePlugin', + ], + }, + classifiers=[ + "Environment :: Console", + "Intended Audience :: Developers", + "License :: OSI Approved :: MIT License", + "Programming Language :: Python", + "Programming Language :: Python :: 2", + "Programming Language :: Python :: 3", + "Topic :: Software Development :: Libraries :: Python Modules", + "Topic :: Software Development :: Quality Assurance", + ], + ) + +Note specifically these lines: + +.. code-block:: python + + flake8_entry_point = # ... + + setuptools.setup( + # snip ... + entry_points={ + flake8_entry_point: [ + 'X = flake8_example.ExamplePlugin', + ], + }, + # snip ... + ) + +We tell setuptools to register our entry point "X" inside the specific +grouping of entry-points that flake8 should look in. + +Flake8 presently looks at three groups: + +- ``flake8.extension`` + +- ``flake8.format`` + +- ``flake8.report`` + +If your plugin is one that adds checks to Flake8, you will use +``flake8.extension``. If your plugin formats the output and provides that to +the user, you will use ``flake8.format``. Finally, if your plugin performs +extra report handling (filtering, etc.) it will use ``flake8.report``. + +If our ``ExamplePlugin`` is something that adds checks, our code would look +like: + +.. code-block:: python + + setuptools.setup( + # snip ... + entry_points={ + 'flake8.extension': [ + 'X = flake8_example.ExamplePlugin', + ], + }, + # snip ... + ) + + +.. _Entry Points: + https://pythonhosted.org/setuptools/pkg_resources.html#entry-points From 7828b63002fdebad678698e16eb744c494713f93 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 6 Feb 2016 15:18:41 -0600 Subject: [PATCH 129/364] Finish developer documentation around formatter plugins --- docs/source/dev/formatters.rst | 34 +++++++++++++++++++++++++ docs/source/dev/registering_plugins.rst | 2 ++ 2 files changed, 36 insertions(+) diff --git a/docs/source/dev/formatters.rst b/docs/source/dev/formatters.rst index b96c40b..8f90892 100644 --- a/docs/source/dev/formatters.rst +++ b/docs/source/dev/formatters.rst @@ -14,3 +14,37 @@ Flake8 added the ability to develop custom formatting plugins in version """Flake8's example formatter.""" pass + +We notice, as soon as we start, that we inherit from Flake8's +:class:`~flake8.formatting.base.BaseFormatter` class. If we follow the +:ref:`instructions to register a plugin ` and try to use +our example formatter, e.g., ``flake8 --format=example`` then Flake8 will fail +because we did not implement the ``format`` method. Let's do that next. + +.. code-block:: python + + class Example(base.BaseFormatter): + """Flake8's example formatter.""" + + def format(self, error): + return 'Example formatter: {0!r}'.format(error) + +With that we're done. Obviously this isn't a very useful formatter, but it +should highlight the simplicitly of creating a formatter with Flake8. If we +wanted to instead create a formatter that aggregated the results and returned +XML, JSON, or subunit we could also do that. Flake8 interacts with the +formatter in two ways: + +#. It creates the formatter and provides it the options parsed from the + configuration files and command-line + +#. It uses the instance of the formatter and calls ``handle`` with the error. + +By default :meth:`flake8.formatting.base.BaseFormatter.handle` simply calls +the ``format`` method and then ``write``. Any extra handling you wish to do +for formatting purposes should override the ``handle`` method. + +API Documentation +================= + +.. autoclass:: flake8.formatting.base.BaseFormatter diff --git a/docs/source/dev/registering_plugins.rst b/docs/source/dev/registering_plugins.rst index 1444af0..c0edf63 100644 --- a/docs/source/dev/registering_plugins.rst +++ b/docs/source/dev/registering_plugins.rst @@ -1,3 +1,5 @@ +.. _register-a-plugin: + ================================== Registering a Plugin with Flake8 ================================== From 81495fd859192462ee738564c24fb14b51ebbfa1 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 6 Feb 2016 15:22:36 -0600 Subject: [PATCH 130/364] Add default formatting class --- flake8/formatting/default.py | 32 ++++++++++++++++++++++++++++++++ flake8/main/cli.py | 2 +- setup.py | 3 +++ 3 files changed, 36 insertions(+), 1 deletion(-) create mode 100644 flake8/formatting/default.py diff --git a/flake8/formatting/default.py b/flake8/formatting/default.py new file mode 100644 index 0000000..0b69d7a --- /dev/null +++ b/flake8/formatting/default.py @@ -0,0 +1,32 @@ +"""Default formatting class for Flake8.""" +from flake8.formatting import base + + +class Default(base.BaseFormatter): + """Default formatter for Flake8. + + This also handles backwards compatibility for people specifying a custom + format string. + """ + + error_format = '%(path)s:%(row)d:%(col)d: %(code)s %(text)s' + + def after_init(self): + """Check for a custom format string.""" + self.output_fd = None + if self.options.format.lower() != 'default': + self.error_format = self.options.format + + def format(self, error): + """Format and write error out. + + If an output filename is specified, write formatted errors to that + file. Otherwise, print the formatted error to standard out. + """ + return self.error_format % { + "code": error.code, + "text": error.text, + "path": error.filename, + "row": error.line_number, + "col": error.column_number, + } diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 51071b5..e77be84 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -54,7 +54,7 @@ def register_default_options(option_manager): # TODO(sigmavirus24): Figure out --first/--repeat add_option( - '--format', metavar='format', default='default', choices=['default'], + '--format', metavar='format', default='default', parse_from_config=True, help='Format errors according to the chosen formatter.', ) diff --git a/setup.py b/setup.py index 00b5a68..281ed00 100644 --- a/setup.py +++ b/setup.py @@ -70,6 +70,9 @@ setuptools.setup( 'flake8.extension': [ 'F = flake8.plugins.pyflakes:FlakesChecker', ], + 'flake8.format': [ + 'default = flake8.formatting.default.Default', + ], }, classifiers=[ "Environment :: Console", From 56976671d0bb9ba5ea725e8aeda3387c402ce938 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 7 Feb 2016 09:39:19 -0600 Subject: [PATCH 131/364] Fix entry-point names and example usage --- docs/source/dev/registering_plugins.rst | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/source/dev/registering_plugins.rst b/docs/source/dev/registering_plugins.rst index c0edf63..48e39fc 100644 --- a/docs/source/dev/registering_plugins.rst +++ b/docs/source/dev/registering_plugins.rst @@ -48,7 +48,7 @@ like: install_requires=requires, entry_points={ flake8_entry_point: [ - 'X = flake8_example.ExamplePlugin', + 'X = flake8_example:ExamplePlugin', ], }, classifiers=[ @@ -73,7 +73,7 @@ Note specifically these lines: # snip ... entry_points={ flake8_entry_point: [ - 'X = flake8_example.ExamplePlugin', + 'X = flake8_example:ExamplePlugin', ], }, # snip ... @@ -86,14 +86,14 @@ Flake8 presently looks at three groups: - ``flake8.extension`` -- ``flake8.format`` +- ``flake8.listen`` - ``flake8.report`` If your plugin is one that adds checks to Flake8, you will use -``flake8.extension``. If your plugin formats the output and provides that to -the user, you will use ``flake8.format``. Finally, if your plugin performs -extra report handling (filtering, etc.) it will use ``flake8.report``. +``flake8.extension``. If your plugin automatically fixes errors in code, you +will use ``flake8.listen``. Finally, if your plugin performs extra report +handling (formatting, filtering, etc.) it will use ``flake8.report``. If our ``ExamplePlugin`` is something that adds checks, our code would look like: @@ -104,7 +104,7 @@ like: # snip ... entry_points={ 'flake8.extension': [ - 'X = flake8_example.ExamplePlugin', + 'X = flake8_example:ExamplePlugin', ], }, # snip ... From 57174a0743b55c543556984105440685c4a69287 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 7 Feb 2016 09:39:58 -0600 Subject: [PATCH 132/364] Print the line not the output file object --- flake8/formatting/base.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/formatting/base.py b/flake8/formatting/base.py index 85a931c..e419470 100644 --- a/flake8/formatting/base.py +++ b/flake8/formatting/base.py @@ -96,7 +96,7 @@ class BaseFormatter(object): if self.output_fd is not None: self.output_fd.write(line + self.newline) else: - print(self.output_fd) + print(line) def stop(self): """Clean up after reporting is finished.""" From 853985e1dca99e3f265c02f0f733f9c4b33106ca Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 7 Feb 2016 09:40:18 -0600 Subject: [PATCH 133/364] Allow dictionary-like get behaviour with plugins --- flake8/plugins/manager.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 02a136c..3a20b91 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -203,6 +203,22 @@ class PluginTypeManager(object): LOG.debug('Retrieving plugin for "%s".', name) return self.plugins[name] + def get(self, name, default=None): + """Retrieve the plugin referred to by ``name`` or return the default. + + :param str name: + Name of the plugin to retrieve. + :param default: + Default value to return. + :returns: + Plugin object referred to by name, if it exists. + :rtype: + :class:`Plugin` + """ + if name in self: + return self[name] + return default + @property def names(self): """Proxy attribute to underlying manager.""" From 0b8f11acc2ab95f5c1a3cbe7ff7e70139104f884 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 7 Feb 2016 09:40:32 -0600 Subject: [PATCH 134/364] Fix our entry-points for report formatters --- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index 281ed00..2ab68dd 100644 --- a/setup.py +++ b/setup.py @@ -70,8 +70,8 @@ setuptools.setup( 'flake8.extension': [ 'F = flake8.plugins.pyflakes:FlakesChecker', ], - 'flake8.format': [ - 'default = flake8.formatting.default.Default', + 'flake8.report': [ + 'default = flake8.formatting.default:Default', ], }, classifiers=[ From 2fc853b7726c5fa917adba6e64bd0d62d461d3d3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 7 Feb 2016 10:44:46 -0600 Subject: [PATCH 135/364] Stop assigning output_fd in Default formatter --- flake8/formatting/default.py | 1 - flake8/main/cli.py | 4 ++++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/flake8/formatting/default.py b/flake8/formatting/default.py index 0b69d7a..95cc129 100644 --- a/flake8/formatting/default.py +++ b/flake8/formatting/default.py @@ -13,7 +13,6 @@ class Default(base.BaseFormatter): def after_init(self): """Check for a custom format string.""" - self.output_fd = None if self.options.format.lower() != 'default': self.error_format = self.options.format diff --git a/flake8/main/cli.py b/flake8/main/cli.py index e77be84..8cc4791 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -174,3 +174,7 @@ def main(argv=None): # Parse out our options from our found config files and user-provided CLI # options options, args = aggregator.aggregate_options(option_manager) + + # formatter = formatting_plugins.get( + # options.format, formatting_plugins['default'] + # ).execute(options) From b8a38c2573eb00f30a563ee233050ae0b5db1842 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 7 Feb 2016 10:45:19 -0600 Subject: [PATCH 136/364] Add pylint formatter --- flake8/formatting/default.py | 14 ++++++++++++++ setup.py | 1 + 2 files changed, 15 insertions(+) diff --git a/flake8/formatting/default.py b/flake8/formatting/default.py index 95cc129..036d13f 100644 --- a/flake8/formatting/default.py +++ b/flake8/formatting/default.py @@ -29,3 +29,17 @@ class Default(base.BaseFormatter): "row": error.line_number, "col": error.column_number, } + + +class Pylint(Default): + """Pylint formatter for Flake8.""" + + error_format = '%(path)s:%(row)d: [%(code)s] %(text)s' + + def after_init(self): + """Do not check the value of --format. + + In the default formatter, this makes sense for backwards + compatibility, but it does not make sense here. + """ + pass diff --git a/setup.py b/setup.py index 2ab68dd..f8c3ff2 100644 --- a/setup.py +++ b/setup.py @@ -72,6 +72,7 @@ setuptools.setup( ], 'flake8.report': [ 'default = flake8.formatting.default:Default', + 'pylint = flake8.formatting.default:Pylint', ], }, classifiers=[ From f824cbae9321527214b9fb9c3e7688c1d05f4b47 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 7 Feb 2016 19:54:12 -0600 Subject: [PATCH 137/364] Update docs index and tox env --- docs/source/dev/formatters.rst | 1 + docs/source/index.rst | 7 +++++-- tox.ini | 2 +- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/source/dev/formatters.rst b/docs/source/dev/formatters.rst index 8f90892..0b37abd 100644 --- a/docs/source/dev/formatters.rst +++ b/docs/source/dev/formatters.rst @@ -48,3 +48,4 @@ API Documentation ================= .. autoclass:: flake8.formatting.base.BaseFormatter + :members: diff --git a/docs/source/index.rst b/docs/source/index.rst index 4a50a5d..a2bd76a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,8 +18,8 @@ Plugin Developer Guide .. toctree:: :maxdepth: 2 - internal/option_handling - internal/plugin_handling + dev/formatters + dev/registering_plugins Developer Guide --------------- @@ -27,6 +27,9 @@ Developer Guide .. toctree:: :maxdepth: 2 + internal/option_handling + internal/plugin_handling + internal/utils Indices and tables ================== diff --git a/tox.ini b/tox.ini index 67c9478..8b952f3 100644 --- a/tox.ini +++ b/tox.ini @@ -35,7 +35,7 @@ basepython = python3.4 skipsdist = true skip_install = true use_develop = false -changedir = docs/_build/html +changedir = docs/build/html deps = commands = python -m http.server {posargs} From c20793b49c8421059b702943360d38fd2c76304f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 10 Feb 2016 22:44:23 -0600 Subject: [PATCH 138/364] Add internal documentation around default formatters --- docs/source/index.rst | 1 + docs/source/internal/formatters.rst | 47 +++++++++++++++++++++++++++ flake8/formatting/default.py | 49 +++++++++++++++++------------ 3 files changed, 77 insertions(+), 20 deletions(-) create mode 100644 docs/source/internal/formatters.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index a2bd76a..c8a4f35 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -27,6 +27,7 @@ Developer Guide .. toctree:: :maxdepth: 2 + internal/formatters internal/option_handling internal/plugin_handling internal/utils diff --git a/docs/source/internal/formatters.rst b/docs/source/internal/formatters.rst new file mode 100644 index 0000000..caa5718 --- /dev/null +++ b/docs/source/internal/formatters.rst @@ -0,0 +1,47 @@ +===================== + Built-in Formatters +===================== + +By default Flake8 has two formatters built-in, ``default`` and ``pylint``. +These correspond to two classes |DefaultFormatter| and |PylintFormatter|. + +In Flake8 2.0, pep8 handled formatting of errors and also allowed users to +specify an arbitrary format string as a parameter to ``--format``. In order +to allow for this backwards compatibility, Flake8 3.0 made two choices: + +#. To not limit a user's choices for ``--format`` to the format class names + +#. To make the default formatter attempt to use the string provided by the + user if it cannot find a formatter with that name. + +Default Formatter +================= + +The |DefaultFormatter| continues to use the same default format string as +pep8: ``'%(path)s:%(row)d:%(col)d: %(code)s %(text)s'``. + +In order to provide the default functionality it overrides two methods: + +#. ``after_init`` + +#. ``format`` + +The former allows us to inspect the value provided to ``--format`` by the +user and alter our own format based on that value. The second simply uses +that format string to format the error. + +.. autoclass:: flake8.formatters.default.Default + :members: + +Pylint Formatter +================ + +The |PylintFormatter| simply defines the default Pylint format string from +pep8: ``'%(path)s:%(row)d: [%(code)s] %(text)s'``. + +.. autoclass:: flake8.formatters.default.Pylint + :members: + + +.. |DefaultFormatter| replace:: :class:`~flake8.formatters.default.Default` +.. |PylintFormatter| replace:: :class:`~flake8.formatters.default.Pylint` diff --git a/flake8/formatting/default.py b/flake8/formatting/default.py index 036d13f..e0686a0 100644 --- a/flake8/formatting/default.py +++ b/flake8/formatting/default.py @@ -2,20 +2,22 @@ from flake8.formatting import base -class Default(base.BaseFormatter): - """Default formatter for Flake8. +class SimpleFormatter(base.BaseFormatter): + """Simple abstraction for Default and Pylint formatter commonality. + + Sub-classes of this need to define an ``error_format`` attribute in order + to succeed. The ``format`` method relies on that attribute and expects the + ``error_format`` string to use the old-style formatting strings with named + parameters: + + * code + * text + * path + * row + * col - This also handles backwards compatibility for people specifying a custom - format string. """ - error_format = '%(path)s:%(row)d:%(col)d: %(code)s %(text)s' - - def after_init(self): - """Check for a custom format string.""" - if self.options.format.lower() != 'default': - self.error_format = self.options.format - def format(self, error): """Format and write error out. @@ -31,15 +33,22 @@ class Default(base.BaseFormatter): } -class Pylint(Default): +class Default(SimpleFormatter): + """Default formatter for Flake8. + + This also handles backwards compatibility for people specifying a custom + format string. + """ + + error_format = '%(path)s:%(row)d:%(col)d: %(code)s %(text)s' + + def after_init(self): + """Check for a custom format string.""" + if self.options.format.lower() != 'default': + self.error_format = self.options.format + + +class Pylint(SimpleFormatter): """Pylint formatter for Flake8.""" error_format = '%(path)s:%(row)d: [%(code)s] %(text)s' - - def after_init(self): - """Do not check the value of --format. - - In the default formatter, this makes sense for backwards - compatibility, but it does not make sense here. - """ - pass From 1d241a22c926052bce40d2ed601920f2e58e66c6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 10 Feb 2016 22:45:20 -0600 Subject: [PATCH 139/364] Sketch out how the StyleGuide would be used from main --- flake8/main/cli.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 8cc4791..183b951 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -4,6 +4,7 @@ from flake8 import defaults from flake8.options import aggregator from flake8.options import manager from flake8.plugins import manager as plugin_manager +from flake8 import style_guide def register_default_options(option_manager): @@ -178,3 +179,5 @@ def main(argv=None): # formatter = formatting_plugins.get( # options.format, formatting_plugins['default'] # ).execute(options) + # listener_trie = listening_plugins.build_notifier() + # guide = style_guide.StyleGuide(options, args, listener_trie, formatter) From 99b96e95ceba764d6d227886bc486f243462f466 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 10 Feb 2016 22:45:50 -0600 Subject: [PATCH 140/364] Simplify StyleGuide and add handle_error method --- flake8/style_guide.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index e8451f1..7a39a4a 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -43,22 +43,21 @@ Error = collections.namedtuple('Error', ['code', class StyleGuide(object): """Manage a Flake8 user's style guide.""" - def __init__(self, options, arguments, checker_plugins, listening_plugins, - formatting_plugins): + def __init__(self, options, arguments, listener_trie, formatter): """Initialize our StyleGuide. .. todo:: Add parameter documentation. """ self.options = options self.arguments = arguments - self.checkers = checker_plugins - self.listeners = listening_plugins - self.formatters = formatting_plugins + self.listener = listener_trie + self.formatter = formatter self._selected = tuple(options.select) self._ignored = tuple(options.ignore) self._decision_cache = {} def is_user_selected(self, code): + # type: (Error) -> Union[Selected, Ignored] """Determine if the code has been selected by the user. :param str code: @@ -79,6 +78,7 @@ class StyleGuide(object): return Ignored.Implicitly def is_user_ignored(self, code): + # type: (Error) -> Union[Selected, Ignored] """Determine if the code has been ignored by the user. :param str code: @@ -96,6 +96,7 @@ class StyleGuide(object): return Selected.Implicitly def _decision_for(self, code): + # type: (Error) -> Decision startswith = code.startswith selected = sorted([s for s in self._selected if startswith(s)])[0] ignored = sorted([i for i in self._ignored if startswith(i)])[0] @@ -105,6 +106,7 @@ class StyleGuide(object): return Decision.Ignored def should_report_error(self, code): + # type: (Error) -> Decision """Determine if the error code should be reported or ignored. :param str code: @@ -133,6 +135,13 @@ class StyleGuide(object): LOG.debug('"%s" will be "%s"', code, decision) return decision + def handle_error(self, code, filename, line_number, column_number, text): + # type: (str, str, int, int, str) -> NoneType + """Handle an error reported by a check.""" + error = Error(code, filename, line_number, column_number, text) + if self.should_report_error(error): + self.formatter.handle(error) + self.notifier.notify(error.code, error) # Should separate style guide logic from code that runs checks # StyleGuide should manage select/ignore logic as well as include/exclude From a9a939cbbc93f860aee607d1d1f40068f10067a6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 15 Feb 2016 22:31:56 -0600 Subject: [PATCH 141/364] Update StyleGuide tests and add new tests for handle_error --- flake8/style_guide.py | 4 +- tests/unit/test_style_guide.py | 83 ++++++++++++++++++++++++++-------- 2 files changed, 67 insertions(+), 20 deletions(-) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 7a39a4a..98e21c8 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -139,9 +139,9 @@ class StyleGuide(object): # type: (str, str, int, int, str) -> NoneType """Handle an error reported by a check.""" error = Error(code, filename, line_number, column_number, text) - if self.should_report_error(error): + if self.should_report_error(error.code) is Decision.Selected: self.formatter.handle(error) - self.notifier.notify(error.code, error) + self.listener.notify(error.code, error) # Should separate style guide logic from code that runs checks # StyleGuide should manage select/ignore logic as well as include/exclude diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index 51e9f61..f1e18b6 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -1,8 +1,11 @@ """Tests for the flake8.style_guide.StyleGuide class.""" import optparse +from flake8.formatting import base +from flake8.plugins import notifier from flake8 import style_guide +import mock import pytest @@ -24,9 +27,8 @@ def test_is_user_ignored_ignores_errors(ignore_list, error_code): """Verify we detect users explicitly ignoring an error.""" guide = style_guide.StyleGuide(create_options(ignore=ignore_list), arguments=[], - checker_plugins=None, - listening_plugins=None, - formatting_plugins=None) + listener_trie=None, + formatter=None) assert guide.is_user_ignored(error_code) is style_guide.Ignored.Explicitly @@ -42,9 +44,8 @@ def test_is_user_ignored_implicitly_selects_errors(ignore_list, error_code): """Verify we detect users does not explicitly ignore an error.""" guide = style_guide.StyleGuide(create_options(ignore=ignore_list), arguments=[], - checker_plugins=None, - listening_plugins=None, - formatting_plugins=None) + listener_trie=None, + formatter=None) assert guide.is_user_ignored(error_code) is style_guide.Selected.Implicitly @@ -60,9 +61,8 @@ def test_is_user_selected_selects_errors(select_list, error_code): """Verify we detect users explicitly selecting an error.""" guide = style_guide.StyleGuide(create_options(select=select_list), arguments=[], - checker_plugins=None, - listening_plugins=None, - formatting_plugins=None) + listener_trie=None, + formatter=None) assert (guide.is_user_selected(error_code) is style_guide.Selected.Explicitly) @@ -74,9 +74,8 @@ def test_is_user_selected_implicitly_selects_errors(): error_code = 'E121' guide = style_guide.StyleGuide(create_options(select=select_list), arguments=[], - checker_plugins=None, - listening_plugins=None, - formatting_plugins=None) + listener_trie=None, + formatter=None) assert (guide.is_user_selected(error_code) is style_guide.Selected.Implicitly) @@ -93,9 +92,8 @@ def test_is_user_selected_excludes_errors(select_list, error_code): """Verify we detect users implicitly excludes an error.""" guide = style_guide.StyleGuide(create_options(select=select_list), arguments=[], - checker_plugins=None, - listening_plugins=None, - formatting_plugins=None) + listener_trie=None, + formatter=None) assert guide.is_user_selected(error_code) is style_guide.Ignored.Implicitly @@ -121,8 +119,57 @@ def test_should_report_error(select_list, ignore_list, error_code, expected): guide = style_guide.StyleGuide(create_options(select=select_list, ignore=ignore_list), arguments=[], - checker_plugins=None, - listening_plugins=None, - formatting_plugins=None) + listener_trie=None, + formatter=None) assert guide.should_report_error(error_code) is expected + + +@pytest.mark.parametrize('select_list,ignore_list,error_code', [ + (['E111', 'E121'], [], 'E111'), + (['E111', 'E121'], [], 'E121'), + (['E11', 'E121'], ['E1'], 'E112'), + ([], ['E2', 'E12'], 'E410'), + (['E41'], ['E2', 'E12', 'E4'], 'E410'), +]) +def test_handle_error_notifies_listeners(select_list, ignore_list, error_code): + """Verify that error codes notify the listener trie appropriately.""" + listener_trie = mock.create_autospec(notifier.Notifier, instance=True) + formatter = mock.create_autospec(base.BaseFormatter, instance=True) + guide = style_guide.StyleGuide(create_options(select=select_list, + ignore=ignore_list), + arguments=[], + listener_trie=listener_trie, + formatter=formatter) + + guide.handle_error(error_code, 'stdin', 1, 1, 'error found') + error = style_guide.Error(error_code, 'stdin', 1, 1, 'error found') + listener_trie.notify.assert_called_once_with(error_code, error) + formatter.handle.assert_called_once_with(error) + + +@pytest.mark.parametrize('select_list,ignore_list,error_code', [ + (['E111', 'E121'], [], 'E122'), + (['E11', 'E12'], [], 'E132'), + (['E2', 'E12'], [], 'E321'), + (['E2', 'E12'], [], 'E410'), + (['E111', 'E121'], ['E2'], 'E122'), + (['E11', 'E12'], ['E13'], 'E132'), + (['E1', 'E3'], ['E32'], 'E321'), + (['E4'], ['E2', 'E12', 'E41'], 'E410'), + (['E111', 'E121'], [], 'E112'), +]) +def test_handle_error_does_not_notify_listeners(select_list, ignore_list, + error_code): + """Verify that error codes notify the listener trie appropriately.""" + listener_trie = mock.create_autospec(notifier.Notifier, instance=True) + formatter = mock.create_autospec(base.BaseFormatter, instance=True) + guide = style_guide.StyleGuide(create_options(select=select_list, + ignore=ignore_list), + arguments=[], + listener_trie=listener_trie, + formatter=formatter) + + guide.handle_error(error_code, 'stdin', 1, 1, 'error found') + assert listener_trie.notify.called is False + assert formatter.handle.called is False From f4b18229a0294249be22076ad6a52e244acb5a5a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 09:08:24 -0600 Subject: [PATCH 142/364] Add checking for inline #noqa comments This also supports ignoring only specified codes --- flake8/style_guide.py | 50 +++++++++++++++++++++++++++++++++- tests/unit/test_style_guide.py | 6 ++-- 2 files changed, 53 insertions(+), 3 deletions(-) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 98e21c8..af68dce 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -1,9 +1,13 @@ """Implementation of the StyleGuide used by Flake8.""" import collections +import linecache import logging +import re import enum +from flake8 import utils + __all__ = ( 'StyleGuide', ) @@ -43,6 +47,20 @@ Error = collections.namedtuple('Error', ['code', class StyleGuide(object): """Manage a Flake8 user's style guide.""" + NOQA_INLINE_REGEXP = re.compile( + # We're looking for items that look like this: + # ``# noqa`` + # ``# noqa: E123`` + # ``# noqa: E123,W451,F921`` + # ``# NoQA: E123,W451,F921`` + # ``# NOQA: E123,W451,F921`` + # We do not care about the ``: `` that follows ``noqa`` + # We do not care about the casing of ``noqa`` + # We want a comma-separated list of errors + '# noqa(?:: )?(?P[A-Z0-9,]+)?$', + re.IGNORECASE + ) + def __init__(self, options, arguments, listener_trie, formatter): """Initialize our StyleGuide. @@ -109,6 +127,12 @@ class StyleGuide(object): # type: (Error) -> Decision """Determine if the error code should be reported or ignored. + This method only cares about the select and ignore rules as specified + by the user in their configuration files and command-line flags. + + This method does not look at whether the specific line is being + ignored in the file itself. + :param str code: The code for the check that has been run. """ @@ -135,11 +159,35 @@ class StyleGuide(object): LOG.debug('"%s" will be "%s"', code, decision) return decision + def is_inline_ignored(self, error): + """Determine if an comment has been added to ignore this line.""" + physical_line = linecache.getline(error.filename, error.line_number) + noqa_match = self.NOQA_INLINE_REGEXP.search(physical_line) + if noqa_match is None: + LOG.debug('%r is not inline ignored', error) + return False + + codes_str = noqa_match.groupdict()['codes'] + if codes_str is None: + LOG.debug('%r is ignored by a blanket ``# noqa``', error) + return True + + codes = set(utils.parse_comma_separated_list(codes_str)) + if error.code in codes: + LOG.debug('%r is ignored specifically inline with ``# noqa: %s``', + error, codes_str) + return True + + LOG.debug('%r is not ignored inline with ``# noqa: %s``', + error, codes_str) + return False + def handle_error(self, code, filename, line_number, column_number, text): # type: (str, str, int, int, str) -> NoneType """Handle an error reported by a check.""" error = Error(code, filename, line_number, column_number, text) - if self.should_report_error(error.code) is Decision.Selected: + if (self.should_report_error(error.code) is Decision.Selected and + self.is_inline_ignored(error) is False): self.formatter.handle(error) self.listener.notify(error.code, error) diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index f1e18b6..e02fdc7 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -142,7 +142,8 @@ def test_handle_error_notifies_listeners(select_list, ignore_list, error_code): listener_trie=listener_trie, formatter=formatter) - guide.handle_error(error_code, 'stdin', 1, 1, 'error found') + with mock.patch('linecache.getline', return_value=''): + guide.handle_error(error_code, 'stdin', 1, 1, 'error found') error = style_guide.Error(error_code, 'stdin', 1, 1, 'error found') listener_trie.notify.assert_called_once_with(error_code, error) formatter.handle.assert_called_once_with(error) @@ -170,6 +171,7 @@ def test_handle_error_does_not_notify_listeners(select_list, ignore_list, listener_trie=listener_trie, formatter=formatter) - guide.handle_error(error_code, 'stdin', 1, 1, 'error found') + with mock.patch('linecache.getline', return_value=''): + guide.handle_error(error_code, 'stdin', 1, 1, 'error found') assert listener_trie.notify.called is False assert formatter.handle.called is False From fbd5944f15f90114d4d9c2e299ec26a30c646d82 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 11:02:22 -0600 Subject: [PATCH 143/364] Add specific tests for is_inline_ignored Update the logic so someone can use a class in their ``# noqa`` ignore list --- flake8/style_guide.py | 2 +- tests/unit/test_style_guide.py | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index af68dce..6a05249 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -173,7 +173,7 @@ class StyleGuide(object): return True codes = set(utils.parse_comma_separated_list(codes_str)) - if error.code in codes: + if error.code in codes or error.code.startswith(tuple(codes)): LOG.debug('%r is ignored specifically inline with ``# noqa: %s``', error, codes_str) return True diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index e02fdc7..4ae4f5f 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -125,6 +125,26 @@ def test_should_report_error(select_list, ignore_list, error_code, expected): assert guide.should_report_error(error_code) is expected +@pytest.mark.parametrize('error_code,physical_line,expected_result', [ + ('E111', 'a = 1', False), + ('E121', 'a = 1 # noqa: E111', False), + ('E121', 'a = 1 # noqa: E111,W123,F821', False), + ('E111', 'a = 1 # noqa: E111,W123,F821', True), + ('W123', 'a = 1 # noqa: E111,W123,F821', True), + ('E111', 'a = 1 # noqa: E11,W123,F821', True), +]) +def test_is_inline_ignored(error_code, physical_line, expected_result): + """Verify that we detect inline usage of ``# noqa``.""" + guide = style_guide.StyleGuide(create_options(select=['E', 'W', 'F']), + arguments=[], + listener_trie=None, + formatter=None) + error = style_guide.Error(error_code, 'filename.py', 1, 1, 'error text') + + with mock.patch('linecache.getline', return_value=physical_line): + assert guide.is_inline_ignored(error) is expected_result + + @pytest.mark.parametrize('select_list,ignore_list,error_code', [ (['E111', 'E121'], [], 'E111'), (['E111', 'E121'], [], 'E121'), From 85c199ea34d39bdb811249d26fc535a2228139c6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 15:10:17 -0600 Subject: [PATCH 144/364] Add pylint config and testenv --- .pylintrc | 378 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ tox.ini | 11 ++ 2 files changed, 389 insertions(+) create mode 100644 .pylintrc diff --git a/.pylintrc b/.pylintrc new file mode 100644 index 0000000..8b82de9 --- /dev/null +++ b/.pylintrc @@ -0,0 +1,378 @@ +[MASTER] + +# Specify a configuration file. +#rcfile= + +# Python code to execute, usually for sys.path manipulation such as +# pygtk.require(). +#init-hook= + +# Add files or directories to the blacklist. They should be base names, not +# paths. +ignore=CVS,.git,flake8.egg-info + +# Pickle collected data for later comparisons. +persistent=yes + +# List of plugins (as comma separated values of python modules names) to load, +# usually to register additional checkers. +load-plugins= + +# Use multiple processes to speed up Pylint. +jobs=4 + +# Allow loading of arbitrary C extensions. Extensions are imported into the +# active Python interpreter and may run arbitrary code. +unsafe-load-any-extension=no + +# A comma-separated list of package or module names from where C extensions may +# be loaded. Extensions are loading into the active Python interpreter and may +# run arbitrary code +extension-pkg-whitelist= + +# Allow optimization of some AST trees. This will activate a peephole AST +# optimizer, which will apply various small optimizations. For instance, it can +# be used to obtain the result of joining multiple strings with the addition +# operator. Joining a lot of strings can lead to a maximum recursion error in +# Pylint and this flag can prevent that. It has one side effect, the resulting +# AST will be different than the one from reality. +optimize-ast=no + + +[MESSAGES CONTROL] + +# Only show warnings with the listed confidence levels. Leave empty to show +# all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED +confidence= + +# Enable the message, report, category or checker with the given id(s). You can +# either give multiple identifier separated by comma (,) or put this option +# multiple time. See also the "--disable" option for examples. +#enable= + +# Disable the message, report, category or checker with the given id(s). You +# can either give multiple identifiers separated by comma (,) or put this +# option multiple times (only on the command line, not in the configuration +# file where it should appear only once).You can also use "--disable=all" to +# disable everything first and then reenable specific checks. For example, if +# you want to run only the similarities checker, you can use "--disable=all +# --enable=similarities". If you want to run only the classes checker, but have +# no Warning level messages displayed, use"--disable=all --enable=classes +# --disable=W" +disable=intern-builtin,nonzero-method,parameter-unpacking,backtick,raw_input-builtin,dict-view-method,filter-builtin-not-iterating,long-builtin,unichr-builtin,input-builtin,unicode-builtin,file-builtin,map-builtin-not-iterating,delslice-method,apply-builtin,cmp-method,setslice-method,coerce-method,long-suffix,raising-string,import-star-module-level,buffer-builtin,reload-builtin,unpacking-in-except,print-statement,hex-method,old-octal-literal,metaclass-assignment,dict-iter-method,range-builtin-not-iterating,using-cmp-argument,indexing-exception,no-absolute-import,coerce-builtin,getslice-method,suppressed-message,execfile-builtin,round-builtin,useless-suppression,reduce-builtin,old-raise-syntax,zip-builtin-not-iterating,cmp-builtin,xrange-builtin,standarderror-builtin,old-division,oct-method,next-method-called,old-ne-operator,basestring-builtin + + +[REPORTS] + +# Set the output format. Available formats are text, parseable, colorized, msvs +# (visual studio) and html. You can also give a reporter class, eg +# mypackage.mymodule.MyReporterClass. +output-format=text + +# Put messages in a separate file for each module / package specified on the +# command line instead of printing them on stdout. Reports (if any) will be +# written in a file name "pylint_global.[txt|html]". +files-output=no + +# Tells whether to display a full report or only the messages +reports=no + +# Python expression which should return a note less than 10 (10 is the highest +# note). You have access to the variables errors warning, statement which +# respectively contain the number of errors / warnings messages and the total +# number of statements analyzed. This is used by the global evaluation report +# (RP0004). +evaluation=100.0 - ((float(5 * error + warning + refactor + convention) / statement) * 100) + +# Template used to display messages. This is a python new-style format string +# used to format the message information. See doc for all details +#msg-template= + + +[BASIC] + +# List of builtins function names that should not be used, separated by a comma +bad-functions=map,filter + +# Good variable names which should always be accepted, separated by a comma +good-names=i,j,k,ex,Run,_ + +# Bad variable names which should always be refused, separated by a comma +bad-names=foo,bar,baz,toto,tutu,tata + +# Colon-delimited sets of names that determine each other's naming style when +# the name regexes allow several styles. +name-group= + +# Include a hint for the correct naming format with invalid-name +include-naming-hint=yes + +# Regular expression matching correct argument names +argument-rgx=[a-z_][a-z0-9_]{2,30}$ + +# Naming hint for argument names +argument-name-hint=[a-z_][a-z0-9_]{2,30}$ + +# Regular expression matching correct attribute names +attr-rgx=[a-z_][a-z0-9_]{2,30}$ + +# Naming hint for attribute names +attr-name-hint=[a-z_][a-z0-9_]{2,30}$ + +# Regular expression matching correct constant names +const-rgx=(([A-Z_][A-Z0-9_]*)|(__.*__))$ + +# Naming hint for constant names +const-name-hint=(([A-Z_][A-Z0-9_]*)|(__.*__))$ + +# Regular expression matching correct class names +class-rgx=[A-Z_][a-zA-Z0-9]+$ + +# Naming hint for class names +class-name-hint=[A-Z_][a-zA-Z0-9]+$ + +# Regular expression matching correct inline iteration names +inlinevar-rgx=[A-Za-z_][A-Za-z0-9_]*$ + +# Naming hint for inline iteration names +inlinevar-name-hint=[A-Za-z_][A-Za-z0-9_]*$ + +# Regular expression matching correct class attribute names +class-attribute-rgx=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$ + +# Naming hint for class attribute names +class-attribute-name-hint=([A-Za-z_][A-Za-z0-9_]{2,30}|(__.*__))$ + +# Regular expression matching correct function names +function-rgx=[a-z_][a-z0-9_]{2,30}$ + +# Naming hint for function names +function-name-hint=[a-z_][a-z0-9_]{2,30}$ + +# Regular expression matching correct module names +module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$ + +# Naming hint for module names +module-name-hint=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+))$ + +# Regular expression matching correct method names +method-rgx=[a-z_][a-z0-9_]{2,30}$ + +# Naming hint for method names +method-name-hint=[a-z_][a-z0-9_]{2,30}$ + +# Regular expression matching correct variable names +variable-rgx=[a-z_][a-z0-9_]{2,30}$ + +# Naming hint for variable names +variable-name-hint=[a-z_][a-z0-9_]{2,30}$ + +# Regular expression which should only match function or class names that do +# not require a docstring. +no-docstring-rgx=^_ + +# Minimum line length for functions/classes that require docstrings, shorter +# ones are exempt. +docstring-min-length=-1 + + +[ELIF] + +# Maximum number of nested blocks for function / method body +max-nested-blocks=5 + + +[FORMAT] + +# Maximum number of characters on a single line. +max-line-length=100 + +# Regexp for a line that is allowed to be longer than the limit. +ignore-long-lines=^\s*(# )??$ + +# Allow the body of an if to be on the same line as the test if there is no +# else. +single-line-if-stmt=no + +# List of optional constructs for which whitespace checking is disabled. `dict- +# separator` is used to allow tabulation in dicts, etc.: {1 : 1,\n222: 2}. +# `trailing-comma` allows a space between comma and closing bracket: (a, ). +# `empty-line` allows space-only lines. +no-space-check=trailing-comma,dict-separator + +# Maximum number of lines in a module +max-module-lines=1000 + +# String used as indentation unit. This is usually " " (4 spaces) or "\t" (1 +# tab). +indent-string=' ' + +# Number of spaces of indent required inside a hanging or continued line. +indent-after-paren=4 + +# Expected format of line ending, e.g. empty (any line ending), LF or CRLF. +expected-line-ending-format= + + +[LOGGING] + +# Logging modules to check that the string format arguments are in logging +# function parameter format +logging-modules=logging + + +[MISCELLANEOUS] + +# List of note tags to take in consideration, separated by a comma. +notes=FIXME,XXX,TODO + + +[SIMILARITIES] + +# Minimum lines number of a similarity. +min-similarity-lines=4 + +# Ignore comments when computing similarities. +ignore-comments=yes + +# Ignore docstrings when computing similarities. +ignore-docstrings=yes + +# Ignore imports when computing similarities. +ignore-imports=no + + +[SPELLING] + +# Spelling dictionary name. Available dictionaries: none. To make it working +# install python-enchant package. +spelling-dict= + +# List of comma separated words that should not be checked. +spelling-ignore-words= + +# A path to a file that contains private dictionary; one word per line. +spelling-private-dict-file= + +# Tells whether to store unknown words to indicated private dictionary in +# --spelling-private-dict-file option instead of raising a message. +spelling-store-unknown-words=no + + +[TYPECHECK] + +# Tells whether missing members accessed in mixin class should be ignored. A +# mixin class is detected if its name ends with "mixin" (case insensitive). +ignore-mixin-members=yes + +# List of module names for which member attributes should not be checked +# (useful for modules/projects where namespaces are manipulated during runtime +# and thus existing member attributes cannot be deduced by static analysis. It +# supports qualified module names, as well as Unix pattern matching. +ignored-modules= + +# List of classes names for which member attributes should not be checked +# (useful for classes with attributes dynamically set). This supports can work +# with qualified names. +ignored-classes= + +# List of members which are set dynamically and missed by pylint inference +# system, and so shouldn't trigger E1101 when accessed. Python regular +# expressions are accepted. +generated-members= + + +[VARIABLES] + +# Tells whether we should check for unused import in __init__ files. +init-import=no + +# A regular expression matching the name of dummy variables (i.e. expectedly +# not used). +dummy-variables-rgx=_$|dummy + +# List of additional names supposed to be defined in builtins. Remember that +# you should avoid to define new builtins when possible. +additional-builtins= + +# List of strings which can identify a callback function by name. A callback +# name must start or end with one of those strings. +callbacks=cb_,_cb + + +[CLASSES] + +# List of method names used to declare (i.e. assign) instance attributes. +defining-attr-methods=__init__,__new__,setUp + +# List of valid names for the first argument in a class method. +valid-classmethod-first-arg=cls + +# List of valid names for the first argument in a metaclass class method. +valid-metaclass-classmethod-first-arg=mcs + +# List of member names, which should be excluded from the protected access +# warning. +exclude-protected=_asdict,_fields,_replace,_source,_make + + +[DESIGN] + +# Maximum number of arguments for function / method +max-args=20 + +# Argument names that match this expression will be ignored. Default to name +# with leading underscore +ignored-argument-names=_.* + +# Maximum number of locals for function / method body +max-locals=20 + +# Maximum number of return / yield for function / method body +max-returns=6 + +# Maximum number of branch for function / method body +max-branches=12 + +# Maximum number of statements in function / method body +max-statements=50 + +# Maximum number of parents for a class (see R0901). +max-parents=7 + +# Maximum number of attributes for a class (see R0902). +max-attributes=10 + +# Minimum number of public methods for a class (see R0903). +min-public-methods=2 + +# Maximum number of public methods for a class (see R0904). +max-public-methods=20 + +# Maximum number of boolean expressions in a if statement +max-bool-expr=5 + + +[IMPORTS] + +# Deprecated modules which should not be used, separated by a comma +deprecated-modules=optparse + +# Create a graph of every (i.e. internal and external) dependencies in the +# given file (report RP0402 must not be disabled) +import-graph= + +# Create a graph of external dependencies in the given file (report RP0402 must +# not be disabled) +ext-import-graph= + +# Create a graph of internal dependencies in the given file (report RP0402 must +# not be disabled) +int-import-graph= + + +[EXCEPTIONS] + +# Exceptions that will emit a warning when being caught. Defaults to +# "Exception" +overgeneral-exceptions=Exception diff --git a/tox.ini b/tox.ini index 8b952f3..9b56c4c 100644 --- a/tox.ini +++ b/tox.ini @@ -24,6 +24,17 @@ deps = commands = flake8 +[testenv:pylint] +basepython = python3 +skipsdist = true +skip_install = true +use_develop = false +deps = + . + pylint +commands = + pylint flake8 + [testenv:docs] deps = sphinx>=1.3.0 From 69b8be71dca9f0d8b9a22189801a4a4c8328f07f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 15:10:46 -0600 Subject: [PATCH 145/364] Make pylint happier --- flake8/__init__.py | 12 ++++++------ flake8/formatting/default.py | 2 ++ flake8/main/cli.py | 13 +++++++------ flake8/options/aggregator.py | 3 +-- flake8/options/config.py | 9 +++++---- flake8/options/manager.py | 16 +++++++++++----- flake8/plugins/manager.py | 12 ++++++++---- flake8/plugins/pyflakes.py | 23 +++++++++++++---------- flake8/style_guide.py | 8 ++++---- flake8/utils.py | 9 +++++---- tests/unit/test_config_file_finder.py | 4 ++-- 11 files changed, 64 insertions(+), 47 deletions(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index a6520e4..996f036 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -10,8 +10,6 @@ This module """ import logging -import sys - try: from logging import NullHandler except ImportError: @@ -21,6 +19,7 @@ except ImportError: def emit(self, record): """Do nothing.""" pass +import sys LOG = logging.getLogger(__name__) LOG.addHandler(NullHandler()) @@ -51,7 +50,6 @@ def configure_logging(verbosity, filename=None, If the name is "stdout" or "stderr" this will log to the appropriate stream. """ - global LOG if verbosity <= 0: return if verbosity > 2: @@ -60,12 +58,14 @@ def configure_logging(verbosity, filename=None, log_level = _VERBOSITY_TO_LOG_LEVEL[verbosity] if not filename or filename in ('stderr', 'stdout'): - handler = logging.StreamHandler(getattr(sys, filename)) + fileobj = getattr(sys, filename or 'stderr') + handler = logging.StreamHandler else: - handler = logging.FileHandler(filename) + fileobj = filename + handler = logging.FileHandler handler.setFormatter(logging.Formatter(logformat)) - LOG.addHandler(handler) + LOG.addHandler(handler(fileobj)) LOG.setLevel(log_level) LOG.debug('Added a %s logging handler to logger root at %s', filename, __name__) diff --git a/flake8/formatting/default.py b/flake8/formatting/default.py index e0686a0..bef8c88 100644 --- a/flake8/formatting/default.py +++ b/flake8/formatting/default.py @@ -18,6 +18,8 @@ class SimpleFormatter(base.BaseFormatter): """ + error_format = None + def format(self, error): """Format and write error out. diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 183b951..c9503f1 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -174,10 +174,11 @@ def main(argv=None): # Parse out our options from our found config files and user-provided CLI # options - options, args = aggregator.aggregate_options(option_manager) + options, args = aggregator.aggregate_options(option_manager, argv) - # formatter = formatting_plugins.get( - # options.format, formatting_plugins['default'] - # ).execute(options) - # listener_trie = listening_plugins.build_notifier() - # guide = style_guide.StyleGuide(options, args, listener_trie, formatter) + formatter = formatting_plugins.get( + options.format, formatting_plugins['default'] + ).execute(options) + listener_trie = listening_plugins.build_notifier() + guide = style_guide.StyleGuide(options, args, listener_trie, formatter) + guide.handle_error('E111', 'stdin', 1, 1, 'faketext') diff --git a/flake8/options/aggregator.py b/flake8/options/aggregator.py index 5c02730..b344cf0 100644 --- a/flake8/options/aggregator.py +++ b/flake8/options/aggregator.py @@ -71,5 +71,4 @@ def aggregate_options(manager, arglist=None, values=None): setattr(default_values, dest_name, value) # Finally parse the command-line options - final_values, args = manager.parse_args(arglist, default_values) - return final_values, args + return manager.parse_args(arglist, default_values) diff --git a/flake8/options/config.py b/flake8/options/config.py index 2484cdd..ecc40f7 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -66,7 +66,7 @@ class ConfigFileFinder(object): LOG.debug('Found cli configuration files: %s', found_files) return config - def generate_possible_local_config_files(self): + def generate_possible_local_files(self): """Find and generate all local config files.""" tail = self.tail parent = self.parent @@ -84,7 +84,7 @@ class ConfigFileFinder(object): """Find all local config files which actually exist. Filter results from - :meth:`~ConfigFileFinder.generate_possible_local_config_files` based + :meth:`~ConfigFileFinder.generate_possible_local_files` based on whether the filename exists or not. :returns: @@ -93,11 +93,12 @@ class ConfigFileFinder(object): :rtype: [str] """ + exists = os.path.exists return [ filename - for filename in self.generate_possible_local_config_files() + for filename in self.generate_possible_local_files() if os.path.exists(filename) - ] + list(filter(os.path.exists, self.extra_config_files)) + ] + [f for f in self.extra_config_files if exists(f)] def local_configs(self): """Parse all local config files into one config object.""" diff --git a/flake8/options/manager.py b/flake8/options/manager.py index 9faaf7c..cb4c831 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -1,6 +1,6 @@ """Option handling and Option management logic.""" import logging -import optparse +import optparse # pylint: disable=deprecated-module from flake8 import utils @@ -18,8 +18,7 @@ class Option(object): metavar=None, # Options below here are specific to Flake8 parse_from_config=False, comma_separated_list=False, - normalize_paths=False, - ): + normalize_paths=False): """Initialize an Option instance wrapping optparse.Option. The following are all passed directly through to optparse. @@ -69,12 +68,17 @@ class Option(object): """ self.short_option_name = short_option_name self.long_option_name = long_option_name - self.option_args = filter(None, (short_option_name, long_option_name)) + self.option_args = [ + x for x in (short_option_name, long_option_name) if x is not None + ] self.option_kwargs = { 'action': action, 'default': default, 'type': type, 'dest': self._make_dest(dest), + 'nargs': nargs, + 'const': const, + 'choices': choices, 'callback': callback, 'callback_args': callback_args, 'callback_kwargs': callback_kwargs, @@ -97,6 +101,8 @@ class Option(object): 'a long_option_name must also be specified.') self.config_name = long_option_name[2:].replace('-', '_') + self._opt = None + def __repr__(self): """Simple representation of an Option class.""" return ( @@ -129,7 +135,7 @@ class Option(object): def to_optparse(self): """Convert a Flake8 Option to an optparse Option.""" - if not hasattr(self, '_opt'): + if self._opt is None: self._opt = optparse.Option(*self.option_args, **self.option_kwargs) return self._opt diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 3a20b91..21171ef 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -57,7 +57,7 @@ class Plugin(object): def execute(self, *args, **kwargs): r"""Call the plugin with \*args and \*\*kwargs.""" - return self.plugin(*args, **kwargs) + return self.plugin(*args, **kwargs) # pylint: disable=not-callable def _load(self, verify_requirements): # Avoid relying on hasattr() here. @@ -134,7 +134,7 @@ class Plugin(object): ) -class PluginManager(object): +class PluginManager(object): # pylint: disable=too-few-public-methods """Find and manage plugins consistently.""" def __init__(self, namespace, verify_requirements=False): @@ -188,6 +188,8 @@ class PluginManager(object): class PluginTypeManager(object): """Parent class for most of the specific plugin types.""" + namespace = None + def __init__(self): """Initialize the plugin type's manager.""" self.manager = PluginManager(self.namespace) @@ -232,6 +234,7 @@ class PluginTypeManager(object): @staticmethod def _generate_call_function(method_name, optmanager, *args, **kwargs): def generated_function(plugin): + """Function that attempts to call a specific method on a plugin.""" method = getattr(plugin, method_name, None) if (method is not None and isinstance(method, collections.Callable)): @@ -244,6 +247,7 @@ class PluginTypeManager(object): return def load_plugin(plugin): + """Call each plugin's load_plugin method.""" return plugin.load_plugin() plugins = list(self.manager.map(load_plugin)) @@ -269,7 +273,7 @@ class PluginTypeManager(object): list(self.manager.map(call_provide_options)) -class NotifierBuilder(object): +class NotifierBuilderMixin(object): # pylint: disable=too-few-public-methods """Mixin class that builds a Notifier from a PluginManager.""" def build_notifier(self): @@ -293,7 +297,7 @@ class Checkers(PluginTypeManager): namespace = 'flake8.extension' -class Listeners(PluginTypeManager, NotifierBuilder): +class Listeners(PluginTypeManager, NotifierBuilderMixin): """All of the listeners registered through entry-points.""" namespace = 'flake8.listen' diff --git a/flake8/plugins/pyflakes.py b/flake8/plugins/pyflakes.py index 7f6107a..f512511 100644 --- a/flake8/plugins/pyflakes.py +++ b/flake8/plugins/pyflakes.py @@ -23,7 +23,7 @@ def patch_pyflakes(): 'F402 ImportShadowedByLoopVar', 'F403 ImportStarUsed', 'F404 LateFutureImport', - 'F810 Redefined', # XXX Obsolete? + 'F810 Redefined', 'F811 RedefinedWhileUnused', 'F812 RedefinedInListComp', 'F821 UndefinedName', @@ -48,23 +48,23 @@ class FlakesChecker(pyflakes.checker.Checker): def __init__(self, tree, filename): """Initialize the PyFlakes plugin with an AST tree and filename.""" filename = utils.normalize_paths(filename)[0] - withDoctest = self.withDoctest + with_doctest = self.with_doctest included_by = [include for include in self.include_in_doctest if include != '' and filename.startswith(include)] if included_by: - withDoctest = True + with_doctest = True for exclude in self.exclude_from_doctest: if exclude != '' and filename.startswith(exclude): - withDoctest = False + with_doctest = False overlaped_by = [include for include in included_by if include.startswith(exclude)] if overlaped_by: - withDoctest = True + with_doctest = True super(FlakesChecker, self).__init__(tree, filename, - withDoctest=withDoctest) + withDoctest=with_doctest) @classmethod def add_options(cls, parser): @@ -98,7 +98,7 @@ class FlakesChecker(pyflakes.checker.Checker): """Parse option values from Flake8's OptionManager.""" if options.builtins: cls.builtIns = cls.builtIns.union(options.builtins) - cls.withDoctest = options.doctests + cls.with_doctest = options.doctests included_files = [] for included_file in options.include_in_doctest: @@ -131,6 +131,9 @@ class FlakesChecker(pyflakes.checker.Checker): def run(self): """Run the plugin.""" - for m in self.messages: - col = getattr(m, 'col', 0) - yield m.lineno, col, (m.flake8_msg % m.message_args), m.__class__ + for message in self.messages: + col = getattr(message, 'col', 0) + yield (message.lineno, + col, + (message.flake8_msg % message.message_args), + message.__class__) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 6a05249..00a46fe 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -145,15 +145,15 @@ class StyleGuide(object): code, selected, ignored) if ((selected is Selected.Explicitly or - selected is Selected.Implicitly) and + selected is Selected.Implicitly) and ignored is Selected.Implicitly): decision = Decision.Selected elif (selected is Selected.Explicitly and - ignored is Ignored.Explicitly): + ignored is Ignored.Explicitly): decision = self._decision_for(code) elif (selected is Ignored.Implicitly or - ignored is Ignored.Explicitly): - decision = Decision.Ignored + ignored is Ignored.Explicitly): + decision = Decision.Ignored # pylint: disable=R0204 self._decision_cache[code] = decision LOG.debug('"%s" will be "%s"', code, decision) diff --git a/flake8/utils.py b/flake8/utils.py index 98e5cea..1165470 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -33,7 +33,8 @@ def normalize_paths(paths, parent=os.curdir): :rtype: [str] """ - return [normalize_path(p) for p in parse_comma_separated_list(paths)] + return [normalize_path(p, parent) + for p in parse_comma_separated_list(paths)] def normalize_path(path, parent=os.curdir): @@ -57,8 +58,8 @@ def stdin_get_value(): if cached_value is None: stdin_value = sys.stdin.read() if sys.version_info < (3, 0): - cached_value = io.BytesIO(stdin_value) + cached_type = io.BytesIO else: - cached_value = io.StringIO(stdin_value) - stdin_get_value.cached_stdin = cached_value + cached_type = io.StringIO + stdin_get_value.cached_stdin = cached_type(stdin_value) return cached_value.getvalue() diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 200f4e5..6ac1d16 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -68,11 +68,11 @@ def test_cli_config(): os.path.abspath('tox.ini'), os.path.abspath('.flake8')]), ]) -def test_generate_possible_local_config_files(args, expected): +def test_generate_possible_local_files(args, expected): """Verify generation of all possible config paths.""" finder = config.ConfigFileFinder('flake8', args, []) - assert (list(finder.generate_possible_local_config_files()) == + assert (list(finder.generate_possible_local_files()) == expected) From bda3124ffafa1f6babfb16f14dbdced9ff68e722 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 15:14:10 -0600 Subject: [PATCH 146/364] Increase the confidence level for pylint to report something --- .pylintrc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pylintrc b/.pylintrc index 8b82de9..ed80ac9 100644 --- a/.pylintrc +++ b/.pylintrc @@ -43,7 +43,7 @@ optimize-ast=no # Only show warnings with the listed confidence levels. Leave empty to show # all. Valid levels: HIGH, INFERENCE, INFERENCE_FAILURE, UNDEFINED -confidence= +confidence=INFERENCE_FAILURE # Enable the message, report, category or checker with the given id(s). You can # either give multiple identifier separated by comma (,) or put this option From 0827df3e2ad1c0e6787e3fc85cae0a9b266af416 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 15:30:54 -0600 Subject: [PATCH 147/364] Fix our evaluation formula --- .pylintrc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pylintrc b/.pylintrc index ed80ac9..f778dd4 100644 --- a/.pylintrc +++ b/.pylintrc @@ -82,7 +82,7 @@ reports=no # respectively contain the number of errors / warnings messages and the total # number of statements analyzed. This is used by the global evaluation report # (RP0004). -evaluation=100.0 - ((float(5 * error + warning + refactor + convention) / statement) * 100) +evaluation=10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10) # Template used to display messages. This is a python new-style format string # used to format the message information. See doc for all details From 6c90e1045b64670f3ecb63289cd67a7be04284eb Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 16:07:28 -0600 Subject: [PATCH 148/364] Fix tests while keeping pylint happy --- flake8/__init__.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index 996f036..8e32938 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -59,13 +59,14 @@ def configure_logging(verbosity, filename=None, if not filename or filename in ('stderr', 'stdout'): fileobj = getattr(sys, filename or 'stderr') - handler = logging.StreamHandler + handler_cls = logging.StreamHandler else: fileobj = filename - handler = logging.FileHandler + handler_cls = logging.FileHandler + handler = handler_cls(fileobj) handler.setFormatter(logging.Formatter(logformat)) - LOG.addHandler(handler(fileobj)) + LOG.addHandler(handler) LOG.setLevel(log_level) LOG.debug('Added a %s logging handler to logger root at %s', filename, __name__) From 6d3d955ba976279fa3f31a805ebe5dab7e6cc010 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 16:10:00 -0600 Subject: [PATCH 149/364] Fix other tests after pylint fixes --- tests/unit/test_option.py | 3 +++ tests/unit/test_plugin_type_manager.py | 4 ++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py index 5facf2a..67e2255 100644 --- a/tests/unit/test_option.py +++ b/tests/unit/test_option.py @@ -33,6 +33,9 @@ def test_to_optparse_creates_an_option_as_we_expect(Option): 'default': None, 'type': None, 'dest': 'test', + 'nargs': None, + 'const': None, + 'choices': None, 'callback': None, 'callback_args': None, 'callback_kwargs': None, diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index 36fb92a..f64e472 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -200,8 +200,8 @@ def test_proxies_getitem_to_managers_plugins_dictionary(PluginManager): assert type_mgr[key] is plugins[key] -class FakePluginTypeManager(manager.NotifierBuilder): - """Provide an easy way to test the NotifierBuilder.""" +class FakePluginTypeManager(manager.NotifierBuilderMixin): + """Provide an easy way to test the NotifierBuilderMixin.""" def __init__(self, manager): """Initialize with our fake manager.""" From 58b2ca69e466ed8669d830dc6a95fc32363448d7 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 16:10:38 -0600 Subject: [PATCH 150/364] Remove arguments from StyleGuide - StyleGuides do not need the arguments passed in - Add a test for is_inline_ignored obeying disable_noqa --- flake8/main/cli.py | 2 +- flake8/style_guide.py | 7 +++++-- tests/unit/test_style_guide.py | 23 ++++++++++++++--------- 3 files changed, 20 insertions(+), 12 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index c9503f1..670e755 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -180,5 +180,5 @@ def main(argv=None): options.format, formatting_plugins['default'] ).execute(options) listener_trie = listening_plugins.build_notifier() - guide = style_guide.StyleGuide(options, args, listener_trie, formatter) + guide = style_guide.StyleGuide(options, listener_trie, formatter) guide.handle_error('E111', 'stdin', 1, 1, 'faketext') diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 00a46fe..5deff7a 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -61,13 +61,12 @@ class StyleGuide(object): re.IGNORECASE ) - def __init__(self, options, arguments, listener_trie, formatter): + def __init__(self, options, listener_trie, formatter): """Initialize our StyleGuide. .. todo:: Add parameter documentation. """ self.options = options - self.arguments = arguments self.listener = listener_trie self.formatter = formatter self._selected = tuple(options.select) @@ -161,6 +160,10 @@ class StyleGuide(object): def is_inline_ignored(self, error): """Determine if an comment has been added to ignore this line.""" + # TODO(sigmavirus24): Determine how to handle stdin with linecache + if self.options.disable_noqa: + return False + physical_line = linecache.getline(error.filename, error.line_number) noqa_match = self.NOQA_INLINE_REGEXP.search(physical_line) if noqa_match is None: diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index 4ae4f5f..03bb341 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -13,6 +13,7 @@ def create_options(**kwargs): """Create and return an instance of optparse.Values.""" kwargs.setdefault('select', []) kwargs.setdefault('ignore', []) + kwargs.setdefault('disable_noqa', False) return optparse.Values(kwargs) @@ -26,7 +27,6 @@ def create_options(**kwargs): def test_is_user_ignored_ignores_errors(ignore_list, error_code): """Verify we detect users explicitly ignoring an error.""" guide = style_guide.StyleGuide(create_options(ignore=ignore_list), - arguments=[], listener_trie=None, formatter=None) @@ -43,7 +43,6 @@ def test_is_user_ignored_ignores_errors(ignore_list, error_code): def test_is_user_ignored_implicitly_selects_errors(ignore_list, error_code): """Verify we detect users does not explicitly ignore an error.""" guide = style_guide.StyleGuide(create_options(ignore=ignore_list), - arguments=[], listener_trie=None, formatter=None) @@ -60,7 +59,6 @@ def test_is_user_ignored_implicitly_selects_errors(ignore_list, error_code): def test_is_user_selected_selects_errors(select_list, error_code): """Verify we detect users explicitly selecting an error.""" guide = style_guide.StyleGuide(create_options(select=select_list), - arguments=[], listener_trie=None, formatter=None) @@ -73,7 +71,6 @@ def test_is_user_selected_implicitly_selects_errors(): select_list = [] error_code = 'E121' guide = style_guide.StyleGuide(create_options(select=select_list), - arguments=[], listener_trie=None, formatter=None) @@ -91,7 +88,6 @@ def test_is_user_selected_implicitly_selects_errors(): def test_is_user_selected_excludes_errors(select_list, error_code): """Verify we detect users implicitly excludes an error.""" guide = style_guide.StyleGuide(create_options(select=select_list), - arguments=[], listener_trie=None, formatter=None) @@ -118,7 +114,6 @@ def test_should_report_error(select_list, ignore_list, error_code, expected): """Verify we decide when to report an error.""" guide = style_guide.StyleGuide(create_options(select=select_list, ignore=ignore_list), - arguments=[], listener_trie=None, formatter=None) @@ -136,7 +131,6 @@ def test_should_report_error(select_list, ignore_list, error_code, expected): def test_is_inline_ignored(error_code, physical_line, expected_result): """Verify that we detect inline usage of ``# noqa``.""" guide = style_guide.StyleGuide(create_options(select=['E', 'W', 'F']), - arguments=[], listener_trie=None, formatter=None) error = style_guide.Error(error_code, 'filename.py', 1, 1, 'error text') @@ -145,6 +139,19 @@ def test_is_inline_ignored(error_code, physical_line, expected_result): assert guide.is_inline_ignored(error) is expected_result +def test_disable_is_inline_ignored(): + """Verify that is_inline_ignored exits immediately if disabling NoQA.""" + guide = style_guide.StyleGuide(create_options(disable_noqa=True), + listener_trie=None, + formatter=None) + error = style_guide.Error('E121', 'filename.py', 1, 1, 'error text') + + with mock.patch('linecache.getline') as getline: + assert guide.is_inline_ignored(error) is False + + assert getline.called is False + + @pytest.mark.parametrize('select_list,ignore_list,error_code', [ (['E111', 'E121'], [], 'E111'), (['E111', 'E121'], [], 'E121'), @@ -158,7 +165,6 @@ def test_handle_error_notifies_listeners(select_list, ignore_list, error_code): formatter = mock.create_autospec(base.BaseFormatter, instance=True) guide = style_guide.StyleGuide(create_options(select=select_list, ignore=ignore_list), - arguments=[], listener_trie=listener_trie, formatter=formatter) @@ -187,7 +193,6 @@ def test_handle_error_does_not_notify_listeners(select_list, ignore_list, formatter = mock.create_autospec(base.BaseFormatter, instance=True) guide = style_guide.StyleGuide(create_options(select=select_list, ignore=ignore_list), - arguments=[], listener_trie=listener_trie, formatter=formatter) From 98357e71db4591ecf9e74321608def35125aa40d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 18:55:01 -0600 Subject: [PATCH 151/364] Fix Notifier.listeners_for If no Trie.find returns None, then node.data will return an AttributeError. --- flake8/plugins/notifier.py | 3 ++- tests/unit/test_notifier.py | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/flake8/plugins/notifier.py b/flake8/plugins/notifier.py index 323ea1d..dc255c4 100644 --- a/flake8/plugins/notifier.py +++ b/flake8/plugins/notifier.py @@ -31,7 +31,8 @@ class Notifier(object): path = error_code while path: node = self.listeners.find(path) - for listener in node.data: + listeners = getattr(node, 'data', []) + for listener in listeners: yield listener path = path[:-1] diff --git a/tests/unit/test_notifier.py b/tests/unit/test_notifier.py index effcc88..6a162cf 100644 --- a/tests/unit/test_notifier.py +++ b/tests/unit/test_notifier.py @@ -38,3 +38,17 @@ class TestNotifier(object): self.notifier.notify('E111', 'extra', 'args') assert self.listener_map['E111'].was_notified is True assert self.listener_map['E1'].was_notified is True + + @pytest.mark.parametrize('code', ['W123', 'W12', 'W1', 'W']) + def test_no_listeners_for(self, code): + """Show that we return an empty list of listeners.""" + assert list(self.notifier.listeners_for(code)) == [] + + @pytest.mark.parametrize('code,expected', [ + ('E101', ['E101', 'E1']), + ('E211', ['E211', 'E2']), + ]) + def test_listeners_for(self, code, expected): + """Verify that we retrieve the correct listeners.""" + assert ([l.error_code for l in self.notifier.listeners_for(code)] == + expected) From 8e2d8ec6f59fdd055a88a40d743782f4feaa8376 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 18:58:42 -0600 Subject: [PATCH 152/364] Update the default log format --- flake8/__init__.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index 8e32938..416c951 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -37,9 +37,10 @@ _VERBOSITY_TO_LOG_LEVEL = { 2: logging.DEBUG, } +LOG_FORMAT = '[flake8] %(asctime)s %(levelname)s %(message)s' -def configure_logging(verbosity, filename=None, - logformat='%(asctime)s %(levelname)s %(message)s'): + +def configure_logging(verbosity, filename=None, logformat=LOG_FORMAT): """Configure logging for flake8. :param int verbosity: From 8914ccd582d7e664340b90ed789bc2c46e983a75 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 18:59:38 -0600 Subject: [PATCH 153/364] Configure logging based on --verbose --- flake8/main/cli.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 670e755..444c60d 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -172,6 +172,8 @@ def main(argv=None): listening_plugins.register_options(option_manager) formatting_plugins.register_options(option_manager) + preliminary_opts, _ = option_manager.parse_args(argv) + flake8.configure_logging(preliminary_opts.verbose) # Parse out our options from our found config files and user-provided CLI # options options, args = aggregator.aggregate_options(option_manager, argv) From aa6766861edac47562cd19c33fe12f5e298f0f0c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 18:59:48 -0600 Subject: [PATCH 154/364] Be consistent in setup.py --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index f8c3ff2..1f4a6a0 100644 --- a/setup.py +++ b/setup.py @@ -23,7 +23,7 @@ except ImportError: tests_require = ['pytest'] if mock is None: - tests_require += ['mock'] + tests_require.append('mock') requires = [ From 0c968f7a158f60e891d0d4d9ec8b5165a8639eb8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 19 Feb 2016 19:38:27 -0600 Subject: [PATCH 155/364] Add parity with ``python -m flake8`` --- flake8/__main__.py | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 flake8/__main__.py diff --git a/flake8/__main__.py b/flake8/__main__.py new file mode 100644 index 0000000..42bc428 --- /dev/null +++ b/flake8/__main__.py @@ -0,0 +1,4 @@ +"""Module allowing for ``python -m flake8 ...``.""" +from flake8.main import cli + +cli.main() From 1b91c8d66ab6f15c2b5946c5ceba9fe907a3976e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 21 Feb 2016 11:12:56 -0600 Subject: [PATCH 156/364] Start working on the code that runs checks --- flake8/checker.py | 100 ++++++++++++++++++++++++++++++++++++++++++++++ flake8/utils.py | 26 ++++++++++++ 2 files changed, 126 insertions(+) create mode 100644 flake8/checker.py diff --git a/flake8/checker.py b/flake8/checker.py new file mode 100644 index 0000000..2883555 --- /dev/null +++ b/flake8/checker.py @@ -0,0 +1,100 @@ +"""Checker Manager and Checker classes.""" +import logging + +LOG = logging.getLogger(__name__) + +try: + import multiprocessing +except ImportError: + multiprocessing = None + +from flake8 import utils + + +class Manager(object): + """Manage the parallelism and checker instances for each plugin and file. + + This class will be responsible for the following: + + - Determining the parallelism of Flake8, e.g.: + + * Do we use :mod:`multiprocessing` or is it unavailable? + + * Do we automatically decide on the number of jobs to use or did the + user provide that? + + - Falling back to a serial way of processing files if we run into an + OSError related to :mod:`multiprocessing` + + - Organizing the results of each checker so we can group the output + together and make our output deterministic. + """ + + def __init__(self, options, arguments, checker_plugins): + """Initialize our Manager instance. + + :param options: + The options parsed from config files and CLI. + :type options: + optparse.Values + :param list arguments: + The extra arguments parsed from the CLI (if any) + :param checker_plugins: + The plugins representing checks parsed from entry-points. + :type checker_plugins: + flake8.plugins.manager.Checkers + """ + self.arguments = arguments + self.options = options + self.checks = checker_plugins + self.jobs = self._job_count() + + def _job_count(self): + # First we walk through all of our error cases: + # - multiprocessing library is not present + # - we're running on windows in which case we know we have significant + # implemenation issues + # - the user provided stdin and that's not something we can handle + # well + # - we're processing a diff, which again does not work well with + # multiprocessing and which really shouldn't require multiprocessing + # - the user provided some awful input + if not multiprocessing: + LOG.warning('The multiprocessing module is not available. ' + 'Ignoring --jobs arguments.') + return None + + if utils.is_windows(): + LOG.warning('The --jobs option is not available on Windows. ' + 'Ignoring --jobs arguments.') + return None + + if utils.is_using_stdin(self.arguments): + LOG.warning('The --jobs option is not compatible with supplying ' + 'input using - . Ignoring --jobs arguments.') + return None + + if self.options.diff: + LOG.warning('The --diff option was specified with --jobs but ' + 'they are not compatible. Ignoring --jobs arguments.') + return None + + jobs = self.options.jobs + if jobs != 'auto' and not jobs.isdigit(): + LOG.warning('"%s" is not a valid parameter to --jobs. Must be one ' + 'of "auto" or a numerical value, e.g., 4.', jobs) + return None + + # If the value is "auto", we want to let the multiprocessing library + # decide the number based on the number of CPUs. However, if that + # function is not implemented for this particular value of Python we + # default to 1 + if jobs == 'auto': + try: + return multiprocessing.cpu_count() + except NotImplementedError: + return 1 + + # Otherwise, we know jobs should be an integer and we can just convert + # it to an integer + return int(jobs) diff --git a/flake8/utils.py b/flake8/utils.py index 1165470..e642d26 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -63,3 +63,29 @@ def stdin_get_value(): cached_type = io.StringIO stdin_get_value.cached_stdin = cached_type(stdin_value) return cached_value.getvalue() + + +def is_windows(): + # type: () -> bool + """Determine if we're running on Windows. + + :returns: + True if running on Windows, otherwise False + :rtype: + bool + """ + return os.name == 'nt' + + +def is_using_stdin(paths): + # type: (List[str]) -> bool + """Determine if we're going to read from stdin. + + :param list paths: + The paths that we're going to check. + :returns: + True if stdin (-) is in the path, otherwise False + :rtype: + bool + """ + return '-' in paths From 1312e4e0ef4a6bb372178a67a92dcc113356ea49 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 21 Feb 2016 13:09:44 -0600 Subject: [PATCH 157/364] Add mypy env There are still kinks to be worked out but this works reasonably well. This will fail until we can figure out how to import things from the typing module for the type: comments only. We do not want to add a dependency on that backport only for this mypy env. --- flake8/style_guide.py | 7 ++++--- tox.ini | 10 ++++++++++ 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 5deff7a..c3b3d5f 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -74,7 +74,7 @@ class StyleGuide(object): self._decision_cache = {} def is_user_selected(self, code): - # type: (Error) -> Union[Selected, Ignored] + # type: (str) -> Union[Selected, Ignored] """Determine if the code has been selected by the user. :param str code: @@ -95,7 +95,7 @@ class StyleGuide(object): return Ignored.Implicitly def is_user_ignored(self, code): - # type: (Error) -> Union[Selected, Ignored] + # type: (str) -> Union[Selected, Ignored] """Determine if the code has been ignored by the user. :param str code: @@ -123,7 +123,7 @@ class StyleGuide(object): return Decision.Ignored def should_report_error(self, code): - # type: (Error) -> Decision + # type: (str) -> Decision """Determine if the error code should be reported or ignored. This method only cares about the select and ignore rules as specified @@ -159,6 +159,7 @@ class StyleGuide(object): return decision def is_inline_ignored(self, error): + # type: (Error) -> bool """Determine if an comment has been added to ignore this line.""" # TODO(sigmavirus24): Determine how to handle stdin with linecache if self.options.disable_noqa: diff --git a/tox.ini b/tox.ini index 9b56c4c..353cb3b 100644 --- a/tox.ini +++ b/tox.ini @@ -35,6 +35,16 @@ deps = commands = pylint flake8 +[testenv:mypy] +basepython = python3 +skipsdist = true +skip_install = true +use_develop = false +deps = + mypy-lang +commands = + mypy flake8 + [testenv:docs] deps = sphinx>=1.3.0 From a21c328870cf177492927687445a55244bfc78ab Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 21 Feb 2016 13:45:56 -0600 Subject: [PATCH 158/364] Refactor our app setup/running a bit --- flake8/main/cli.py | 126 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 100 insertions(+), 26 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 444c60d..72ca829 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -156,31 +156,105 @@ def register_default_options(option_manager): ) +class Application(object): + """Abstract our application into a class.""" + + def __init__(self, program='flake8', version=flake8.__version__): + # type: (str, str) -> NoneType + """Initialize our application. + + :param str program: + The name of the program/application that we're executing. + :param str version: + The version of the program/application we're executing. + """ + self.program = program + self.version = version + self.option_manager = manager.OptionManager( + prog='flake8', version=flake8.__version__ + ) + register_default_options(self.option_manager) + + # Set the verbosity of the program + preliminary_opts, _ = self.option_manager.parse_args() + flake8.configure_logging(preliminary_opts.verbose) + + self.check_plugins = None + self.listening_plugins = None + self.formatting_plugigns = None + self.formatter = None + self.listener_trie = None + self.guide = None + + self.options = None + self.args = None + + def find_plugins(self): + # type: () -> NoneType + """Find and load the plugins for this application.""" + if self.check_plugins is None: + self.check_plugins = plugin_manager.Checkers() + + if self.listening_plugins is None: + self.listening_plugins = plugin_manager.Listeners() + + if self.formatting_plugins is None: + self.formatting_plugins = plugin_manager.ReportFormatters() + + def register_plugin_options(self): + # type: () -> NoneType + """Register options provided by plugins to our option manager.""" + self.check_plugins.register_options(self.option_manager) + self.listening_plugins.register_options(self.option_manager) + self.formatting_plugins.register_options(self.option_manager) + + def parse_configuration_and_cli(self, argv=None): + # type: (Union[NoneType, List[str]]) -> NoneType + """Parse configuration files and the CLI options. + + :param list argv: + Command-line arguments passed in directly. + """ + if self.options is None and self.args is None: + self.options, self.args = aggregator.aggregate_options( + self.option_manager, argv + ) + + def make_formatter(self): + # type: () -> NoneType + """Initialize a formatter based on the parsed options.""" + if self.formatter is None: + self.formatter = self.formatting_plugins.get( + self.options.format, self.formatting_plugins['default'] + ).execute(self.options) + + def make_notifier(self): + # type: () -> NoneType + """Initialize our listener Notifier.""" + if self.listener_trie is None: + self.listener_trie = self.listening_plugins.build_notifier() + + def make_guide(self): + # type: () -> NoneType + """Initialize our StyleGuide.""" + if self.guide is None: + self.guide = style_guide.StyleGuide( + self.options, self.listener_trie, self.formatter + ) + + def run(self, argv=None): + # type: (Union[NoneType, List[str]]) -> NoneType + """Run our application.""" + self.find_plugins() + self.register_plugin_options() + self.parse_configuration_and_cli(argv) + self.make_formatter() + self.make_notifier() + self.make_guide() + + def main(argv=None): + # type: (Union[NoneType, List[str]]) -> NoneType """Main entry-point for the flake8 command-line tool.""" - option_manager = manager.OptionManager( - prog='flake8', version=flake8.__version__ - ) - # Load our plugins - check_plugins = plugin_manager.Checkers() - listening_plugins = plugin_manager.Listeners() - formatting_plugins = plugin_manager.ReportFormatters() - - # Register all command-line and config-file options - register_default_options(option_manager) - check_plugins.register_options(option_manager) - listening_plugins.register_options(option_manager) - formatting_plugins.register_options(option_manager) - - preliminary_opts, _ = option_manager.parse_args(argv) - flake8.configure_logging(preliminary_opts.verbose) - # Parse out our options from our found config files and user-provided CLI - # options - options, args = aggregator.aggregate_options(option_manager, argv) - - formatter = formatting_plugins.get( - options.format, formatting_plugins['default'] - ).execute(options) - listener_trie = listening_plugins.build_notifier() - guide = style_guide.StyleGuide(options, listener_trie, formatter) - guide.handle_error('E111', 'stdin', 1, 1, 'faketext') + app = Application() + app.run(argv) From 8792c30872e7e49c2cf2076c5765238b2e92c729 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 22 Feb 2016 21:47:43 -0600 Subject: [PATCH 159/364] Add utility functions around filename matching We add utils.fnmatch and utils.filenames_for in anticipation of our checker manager creating file checkers for each file. We also include tests for these functions and a couple previously untested utility functions. --- flake8/utils.py | 55 ++++++++++++++++++++++++++++++++++++++++ tests/unit/test_utils.py | 46 +++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+) diff --git a/flake8/utils.py b/flake8/utils.py index e642d26..54c1990 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -1,4 +1,5 @@ """Utility methods for flake8.""" +import fnmatch as _fnmatch import io import os import sys @@ -89,3 +90,57 @@ def is_using_stdin(paths): bool """ return '-' in paths + + +def _default_predicate(*args): + return False + + +def filenames_from(arg, predicate=None): + # type: (str) -> Generator + """Generate filenames from an argument. + + :param str arg: + Parameter from the command-line. + :param callable predicate: + Predicate to use to filter out filenames. If the predicate + returns ``True`` we will exclude the filename, otherwise we + will yield it. + :returns: + Generator of paths + """ + if predicate is None: + predicate = _default_predicate + if os.path.isdir(arg): + for root, sub_directories, files in os.walk(arg): + for filename in files: + joined = os.path.join(root, filename) + if predicate(filename) or predicate(joined): + continue + yield joined + # NOTE(sigmavirus24): os.walk() will skip a directory if you + # remove it from the list of sub-directories. + for directory in sub_directories: + if predicate(directory): + sub_directories.remove(directory) + else: + yield arg + + +def fnmatch(filename, patterns, default=True): + # type: (str, List[str], bool) -> bool + """Wrap :func:`fnmatch.fnmatch` to add some functionality. + + :param str filename: + Name of the file we're trying to match. + :param list patterns: + Patterns we're using to try to match the filename. + :param bool default: + The default value if patterns is empty + :returns: + True if a pattern matches the filename, False if it doesn't. + ``default`` if patterns is empty. + """ + if not patterns: + return default + return any(_fnmatch.fnmatch(filename, pattern) for pattern in patterns) diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index a6096d4..34b4909 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -1,5 +1,6 @@ """Tests for flake8's utils module.""" import os +import mock import pytest @@ -40,3 +41,48 @@ def test_normalize_path(value, expected): def test_normalize_paths(value, expected): """Verify we normalize comma-separated paths provided to the tool.""" assert utils.normalize_paths(value) == expected + + +def test_is_windows_checks_for_nt(): + """Verify that we correctly detect Windows.""" + with mock.patch.object(os, 'name', 'nt'): + assert utils.is_windows() is True + + with mock.patch.object(os, 'name', 'posix'): + assert utils.is_windows() is False + + +@pytest.mark.parametrize('filename,patterns,expected', [ + ('foo.py', [], True), + ('foo.py', ['*.pyc'], False), + ('foo.pyc', ['*.pyc'], True), + ('foo.pyc', ['*.swp', '*.pyc', '*.py'], True), +]) +def test_fnmatch(filename, patterns, expected): + """Verify that our fnmatch wrapper works as expected.""" + assert utils.fnmatch(filename, patterns) is expected + + +def test_filenames_from_a_directory(): + """Verify that filenames_from walks a directory.""" + filenames = list(utils.filenames_from('flake8/')) + assert len(filenames) > 2 + assert 'flake8/__init__.py' in filenames + + +def test_filenames_from_a_directory_with_a_predicate(): + """Verify that predicates filter filenames_from.""" + filenames = list(utils.filenames_from( + arg='flake8/', + predicate=lambda filename: filename == 'flake8/__init__.py', + )) + assert len(filenames) > 2 + assert 'flake8/__init__.py' not in filenames + + +def test_filenames_from_a_single_file(): + """Verify that we simply yield that filename.""" + filenames = list(utils.filenames_from('flake8/__init__.py')) + + assert len(filenames) == 1 + assert ['flake8/__init__.py'] == filenames From d5a480a46401949c42fe42fa2eb50a646e0ec3f6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 22 Feb 2016 21:51:50 -0600 Subject: [PATCH 160/364] Add method to determine if filename is excluded Add a method to make FileCheckers from the list of arguments the user passes. Start to work on FileCheckers --- flake8/checker.py | 79 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 78 insertions(+), 1 deletion(-) diff --git a/flake8/checker.py b/flake8/checker.py index 2883555..0a3a86f 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -1,5 +1,6 @@ """Checker Manager and Checker classes.""" import logging +import os LOG = logging.getLogger(__name__) @@ -48,6 +49,14 @@ class Manager(object): self.options = options self.checks = checker_plugins self.jobs = self._job_count() + self.process_queue = None + self.using_multiprocessing = False + self.processes = [] + self.checkers = [] + + if self.jobs is not None and self.jobs > 1: + self.using_multiprocessing = True + self.process_queue = multiprocessing.Queue() def _job_count(self): # First we walk through all of our error cases: @@ -93,8 +102,76 @@ class Manager(object): try: return multiprocessing.cpu_count() except NotImplementedError: - return 1 + return 0 # Otherwise, we know jobs should be an integer and we can just convert # it to an integer return int(jobs) + + def start(self): + """Start checking files.""" + pass + # for i in range(self.jobs or 0): + # proc = multiprocessing.Process(target=self.process_files) + # proc.daemon = True + # proc.start() + # self.processes.append(proc) + + def make_checkers(self, paths=None): + # type: (List[str]) -> NoneType + """Create checkers for each file.""" + if paths is None: + paths = self.arguments + filename_patterns = self.options.filename + self.checkers = [ + FileChecker(filename, self.checks) + for argument in paths + for filename in utils.filenames_from(argument, + self.is_path_excluded) + if utils.fnmatch(filename, filename_patterns) + ] + + def is_path_excluded(self, path): + # type: (str) -> bool + """Check if a path is excluded. + + :param str path: + Path to check against the exclude patterns. + :returns: + True if there are exclude patterns and the path matches, + otherwise False. + :rtype: + bool + """ + exclude = self.options.exclude + if not exclude: + return False + basename = os.path.basename(path) + if utils.fnmatch(basename, exclude): + LOG.info('"%s" has been excluded', basename) + return True + + absolute_path = os.path.abspath(path) + match = utils.fnmatch(absolute_path, exclude) + LOG.info('"%s" has %sbeen excluded', absolute_path, + '' if match else 'not ') + return match + + +class FileChecker(object): + """Manage running checks for a file and aggregate the results.""" + + def __init__(self, filename, checks): + # type: (str, flake8.plugins.manager.Checkers) -> NoneType + """Initialize our file checker. + + :param str filename: + Name of the file to check. + :param checks: + The plugins registered to check the file. + :type checks: + flake8.plugins.manager.Checkers + """ + self.filename = filename + self.checks = checks + self.results = [] From 7addb72615a53aaf05b4e7e121bc0fbdfdd7d789 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 22 Feb 2016 21:59:59 -0600 Subject: [PATCH 161/364] Add unified linters testenv --- flake8/checker.py | 4 ++-- tox.ini | 17 ++++++++++++++++- 2 files changed, 18 insertions(+), 3 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 0a3a86f..fc2da6b 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -2,8 +2,6 @@ import logging import os -LOG = logging.getLogger(__name__) - try: import multiprocessing except ImportError: @@ -11,6 +9,8 @@ except ImportError: from flake8 import utils +LOG = logging.getLogger(__name__) + class Manager(object): """Manage the parallelism and checker instances for each plugin and file. diff --git a/tox.ini b/tox.ini index 353cb3b..00d6469 100644 --- a/tox.ini +++ b/tox.ini @@ -14,6 +14,7 @@ deps = . commands = {posargs} +# Linters [testenv:flake8] skipsdist = true skip_install = true @@ -30,7 +31,7 @@ skipsdist = true skip_install = true use_develop = false deps = - . + pyflakes pylint commands = pylint flake8 @@ -45,6 +46,19 @@ deps = commands = mypy flake8 +[testenv:linters] +basepython = python3 +skipsdist = true +skip_install = true +use_develop = false +deps = + {[testenv:flake8]deps} + {[testenv:pylint]deps} +commands = + {[testenv:flake8]commands} + {[testenv:pylint]commands} + +# Documentation [testenv:docs] deps = sphinx>=1.3.0 @@ -67,6 +81,7 @@ deps = commands = python setup.py check -r -s +# Flake8 Configuration [flake8] # Ignore some flake8-docstrings errors ignore = D203 From 28f4811cb99418c43dd90a9dc31d0ec1ba06c8b7 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 22 Feb 2016 22:17:37 -0600 Subject: [PATCH 162/364] Read lines from the file in our checker --- flake8/checker.py | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/flake8/checker.py b/flake8/checker.py index fc2da6b..f400f1a 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -1,6 +1,9 @@ """Checker Manager and Checker classes.""" +import io import logging import os +import sys +import tokenize try: import multiprocessing @@ -175,3 +178,36 @@ class FileChecker(object): self.filename = filename self.checks = checks self.results = [] + self.lines = [] + + def read_lines(self): + """Read the lines for this file checker.""" + if self.filename is None or self.filename == '-': + self.filename = 'stdin' + return self.read_lines_from_stdin() + return self.read_lines_from_filename() + + def read_lines_from_stdin(self): + """Read the lines from standard in.""" + return utils.stdin_get_value().splitlines(True) + + def read_lines_from_filename(self): + """Read the lines for a file.""" + if (2, 6) <= sys.version_info < (3, 0): + with open(self.filename, 'rU') as fd: + return fd.readlines() + + elif (3, 0) <= sys.version_info < (4, 0): + try: + with open(self.filename, 'rb') as fd: + (coding, lines) = tokenize.detect_encoding(fd.readline) + textfd = io.TextIOWrapper(fd, coding, line_buffering=True) + return ([l.decode(coding) for l in lines] + + textfd.readlines()) + except (LookupError, SyntaxError, UnicodeError): + with open(self.filename, encoding='latin-1') as fd: + return fd.readlines() + + def run_checks(self): + """Run checks against the file.""" + self.lines = self.read_lines() From 5ee061b810caf03025927f2e8fc7fb15fdcabd58 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 23 Feb 2016 11:17:11 -0600 Subject: [PATCH 163/364] Add line splitting and file reading Add some tests around reading lines and striping UTF BOMs --- flake8/checker.py | 68 +++++++++++++++++++++++++-------- tests/unit/test_file_checker.py | 26 +++++++++++++ 2 files changed, 79 insertions(+), 15 deletions(-) create mode 100644 tests/unit/test_file_checker.py diff --git a/flake8/checker.py b/flake8/checker.py index f400f1a..33e58d4 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -187,27 +187,65 @@ class FileChecker(object): return self.read_lines_from_stdin() return self.read_lines_from_filename() - def read_lines_from_stdin(self): - """Read the lines from standard in.""" - return utils.stdin_get_value().splitlines(True) + def _readlines_py2(self): + with open(self.filename, 'rU') as fd: + return fd.readlines() + + def _readlines_py3(self): + try: + with open(self.filename, 'rb') as fd: + (coding, lines) = tokenize.detect_encoding(fd.readline) + textfd = io.TextIOWrapper(fd, coding, line_buffering=True) + return ([l.decode(coding) for l in lines] + + textfd.readlines()) + except (LookupError, SyntaxError, UnicodeError): + # If we can't detect the codec with tokenize.detect_encoding, or + # the detected encoding is incorrect, just fallback to latin-1. + with open(self.filename, encoding='latin-1') as fd: + return fd.readlines() def read_lines_from_filename(self): """Read the lines for a file.""" if (2, 6) <= sys.version_info < (3, 0): - with open(self.filename, 'rU') as fd: - return fd.readlines() - + readlines = self._readlines_py2 elif (3, 0) <= sys.version_info < (4, 0): - try: - with open(self.filename, 'rb') as fd: - (coding, lines) = tokenize.detect_encoding(fd.readline) - textfd = io.TextIOWrapper(fd, coding, line_buffering=True) - return ([l.decode(coding) for l in lines] + - textfd.readlines()) - except (LookupError, SyntaxError, UnicodeError): - with open(self.filename, encoding='latin-1') as fd: - return fd.readlines() + readlines = self._readlines_py3 + + try: + return readlines() + except IOError: + # If we can not read the file due to an IOError (e.g., the file + # does not exist or we do not have the permissions to open it) + # then we need to format that exception for the user. + # NOTE(sigmavirus24): Historically, pep8 has always reported this + # as an E902. We probably *want* a better error code for this + # going forward. + (exc_type, exception) = sys.exc_info()[:2] + message = '{0}: {1}'.format(exc_type.__name__, exception) + self.results.append('E902', self.filename, 0, 0, message) + return [] + + def read_lines_from_stdin(self): + """Read the lines from standard in.""" + return utils.stdin_get_value().splitlines(True) def run_checks(self): """Run checks against the file.""" self.lines = self.read_lines() + self.strip_utf_bom() + + def strip_utf_bom(self): + """Strip the UTF bom from the lines of the file.""" + if not self.lines: + # If we have nothing to analyze quit early + return + + first_byte = ord(self.lines[0][0]) + if first_byte not in (0xEF, 0xFEFF): + return + + # If the first byte of the file is a UTF-8 BOM, strip it + if first_byte == 0xFEFF: + self.lines[0] = self.lines[0][1:] + elif self.lines[0][:3] == '\xEF\xBB\xBF': + self.lines[0] = self.lines[0][3:] diff --git a/tests/unit/test_file_checker.py b/tests/unit/test_file_checker.py new file mode 100644 index 0000000..e26bc83 --- /dev/null +++ b/tests/unit/test_file_checker.py @@ -0,0 +1,26 @@ +"""Tests for the FileChecker class.""" +from flake8 import checker + +import pytest + + +def test_read_lines_splits_lines(): + """Verify that read_lines splits the lines of the file.""" + file_checker = checker.FileChecker(__file__, []) + lines = file_checker.read_lines() + assert len(lines) > 5 + assert '"""Tests for the FileChecker class."""\n' in lines + + +@pytest.mark.parametrize('first_line', [ + '\xEF\xBB\xBF"""Module docstring."""\n', + '\uFEFF"""Module docstring."""\n', +]) +def test_strip_utf_bom(first_line): + r"""Verify that we strip '\xEF\xBB\xBF' from the first line.""" + lines = [first_line] + file_checker = checker.FileChecker('stdin', []) + file_checker.lines = lines[:] + file_checker.strip_utf_bom() + assert file_checker.lines != lines + assert file_checker.lines[0] == '"""Module docstring."""\n' From 54ad972e5601436e6a89b68b1d23376edbc3a3ff Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 23 Feb 2016 14:42:09 -0600 Subject: [PATCH 164/364] Add doc8 linting --- docs/source/dev/registering_plugins.rst | 2 +- tox.ini | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/source/dev/registering_plugins.rst b/docs/source/dev/registering_plugins.rst index 48e39fc..0cc18d9 100644 --- a/docs/source/dev/registering_plugins.rst +++ b/docs/source/dev/registering_plugins.rst @@ -79,7 +79,7 @@ Note specifically these lines: # snip ... ) -We tell setuptools to register our entry point "X" inside the specific +We tell setuptools to register our entry point "X" inside the specific grouping of entry-points that flake8 should look in. Flake8 presently looks at three groups: diff --git a/tox.ini b/tox.ini index 00d6469..ba6c1af 100644 --- a/tox.ini +++ b/tox.ini @@ -36,6 +36,17 @@ deps = commands = pylint flake8 +[testenv:doc8] +basepython = python3 +skipsdist = true +skip_install = true +use_develop = false +deps = + sphinx + doc8 +commands = + doc8 docs/source/ + [testenv:mypy] basepython = python3 skipsdist = true From 08fd403e345f383124c00e9c5352b207284567a3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 23 Feb 2016 14:42:50 -0600 Subject: [PATCH 165/364] Add doc8 to linters --- tox.ini | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tox.ini b/tox.ini index ba6c1af..e4424a3 100644 --- a/tox.ini +++ b/tox.ini @@ -65,9 +65,11 @@ use_develop = false deps = {[testenv:flake8]deps} {[testenv:pylint]deps} + {[testenv:doc8]deps} commands = {[testenv:flake8]commands} {[testenv:pylint]commands} + {[testenv:doc8]commands} # Documentation [testenv:docs] From 51d15295df1d81a0fa38068a46236e99d9fdb2c5 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 23 Feb 2016 17:08:54 -0600 Subject: [PATCH 166/364] Set the maximum complexity for mccabe --- tox.ini | 1 + 1 file changed, 1 insertion(+) diff --git a/tox.ini b/tox.ini index e4424a3..9ea64cf 100644 --- a/tox.ini +++ b/tox.ini @@ -102,3 +102,4 @@ ignore = D203 # across multiple lines. Presently it cannot be specified across multiple lines. # :-( exclude = .git,__pycache__,docs/source/conf.py +max-complexity = 10 From 1cd5fea73066f96753bbc69cbbfa7f96e8d50e4a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 23 Feb 2016 21:09:19 -0600 Subject: [PATCH 167/364] Add type annotations --- flake8/checker.py | 15 ++++++++++++++- flake8/utils.py | 2 +- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 33e58d4..07c90f4 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -62,6 +62,7 @@ class Manager(object): self.process_queue = multiprocessing.Queue() def _job_count(self): + # type: () -> Union[int, NoneType] # First we walk through all of our error cases: # - multiprocessing library is not present # - we're running on windows in which case we know we have significant @@ -181,6 +182,7 @@ class FileChecker(object): self.lines = [] def read_lines(self): + # type: () -> List[str] """Read the lines for this file checker.""" if self.filename is None or self.filename == '-': self.filename = 'stdin' @@ -188,10 +190,12 @@ class FileChecker(object): return self.read_lines_from_filename() def _readlines_py2(self): + # type: () -> List[str] with open(self.filename, 'rU') as fd: return fd.readlines() def _readlines_py3(self): + # type: () -> List[str] try: with open(self.filename, 'rb') as fd: (coding, lines) = tokenize.detect_encoding(fd.readline) @@ -205,6 +209,7 @@ class FileChecker(object): return fd.readlines() def read_lines_from_filename(self): + # type: () -> List[str] """Read the lines for a file.""" if (2, 6) <= sys.version_info < (3, 0): readlines = self._readlines_py2 @@ -222,19 +227,27 @@ class FileChecker(object): # going forward. (exc_type, exception) = sys.exc_info()[:2] message = '{0}: {1}'.format(exc_type.__name__, exception) - self.results.append('E902', self.filename, 0, 0, message) + self.report('E902', 0, 0, message) return [] def read_lines_from_stdin(self): + # type: () -> List[str] """Read the lines from standard in.""" return utils.stdin_get_value().splitlines(True) + def report(self, error_code, line_number, column, text): + # type: (str, int, int, str) -> NoneType + """Report an error by storing it in the results list.""" + error = (error_code, self.filename, line_number, column, text) + self.results.append(error) + def run_checks(self): """Run checks against the file.""" self.lines = self.read_lines() self.strip_utf_bom() def strip_utf_bom(self): + # type: () -> NoneType """Strip the UTF bom from the lines of the file.""" if not self.lines: # If we have nothing to analyze quit early diff --git a/flake8/utils.py b/flake8/utils.py index 54c1990..fbc90bd 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -97,7 +97,7 @@ def _default_predicate(*args): def filenames_from(arg, predicate=None): - # type: (str) -> Generator + # type: (str, callable) -> Generator """Generate filenames from an argument. :param str arg: From 24d2689a0519cb453eb67d6a0ef37f60efc8f0fd Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 23 Feb 2016 23:20:34 -0600 Subject: [PATCH 168/364] Distinguish check types via plugin type manager Flake8 and pep8 has historically supported three types of checks: - Plugins that accept the physical line - Plugins that accept the logical line - Plugins that accept the AST tree The logical place to make this distinction is on the Checkers plugin type manager class. This adds the foundation for finding plugins that fall into each class. --- flake8/plugins/manager.py | 20 ++++++++++++++++++++ flake8/utils.py | 38 ++++++++++++++++++++++++++++++++++++++ tests/unit/test_utils.py | 22 ++++++++++++++++++++++ 3 files changed, 80 insertions(+) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 21171ef..21639f9 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -6,6 +6,7 @@ import pkg_resources from flake8 import exceptions from flake8.plugins import notifier +from flake8 import utils LOG = logging.getLogger(__name__) @@ -296,6 +297,25 @@ class Checkers(PluginTypeManager): namespace = 'flake8.extension' + def checks_expecting(self, argument_name): + """Retrieve checks that expect an argument with the specified name. + + Find all checker plugins that are expecting a specific argument. + """ + for plugin in self.plugins.values(): + parameters = utils.parameters_for(plugin) + if argument_name in parameters: + yield plugin + + @property + def physical_line_plugins(self): + """List of plugins that expect the physical lines.""" + plugins = getattr(self, '_physical_line_plugins', []) + if not plugins: + plugins = list(self.checks_expecting('physical_line')) + self._physical_line_plugins = plugins + return plugins + class Listeners(PluginTypeManager, NotifierBuilderMixin): """All of the listeners registered through entry-points.""" diff --git a/flake8/utils.py b/flake8/utils.py index fbc90bd..cd1cf72 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -1,5 +1,6 @@ """Utility methods for flake8.""" import fnmatch as _fnmatch +import inspect import io import os import sys @@ -144,3 +145,40 @@ def fnmatch(filename, patterns, default=True): if not patterns: return default return any(_fnmatch.fnmatch(filename, pattern) for pattern in patterns) + + +def parameters_for(plugin): + # type: (flake8.plugins.manager.Plugin) -> List[str] + """Return the parameters for the plugin. + + This will inspect the plugin and return either the function parameters + if the plugin is a function or the parameters for ``__init__`` after + ``self`` if the plugin is a class. + + :param plugin: + The internal plugin object. + :type plugin: + flake8.plugins.manager.Plugin + :returns: + Parameters to the plugin. + :rtype: + list(str) + """ + func = plugin.plugin + is_class = not inspect.isfunction(func) + if is_class: # The plugin is a class + func = plugin.plugin.__init__ + + if sys.version_info < (3, 3): + parameters = inspect.getargspec(func)[0] + else: + parameters = [ + parameter.name + for parameter in inspect.signature(func).parameters.values() + if parameter.kind == parameter.POSITIONAL_OR_KEYWORD + ] + + if is_class: + parameters.remove('self') + + return parameters diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 34b4909..615f327 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -4,6 +4,7 @@ import mock import pytest +from flake8.plugins import manager as plugin_manager from flake8 import utils @@ -86,3 +87,24 @@ def test_filenames_from_a_single_file(): assert len(filenames) == 1 assert ['flake8/__init__.py'] == filenames + + +def test_parameters_for_class_plugin(): + """Verify that we can retrieve the parameters for a class plugin.""" + class FakeCheck(object): + def __init__(self, tree): + pass + + plugin = plugin_manager.Plugin('plugin-name', object()) + plugin._plugin = FakeCheck + assert utils.parameters_for(plugin) == ['tree'] + + +def test_parameters_for_function_plugin(): + """Verify that we retrieve the parameters for a function plugin.""" + def fake_plugin(physical_line, self, tree): + pass + + plugin = plugin_manager.Plugin('plugin-name', object()) + plugin._plugin = fake_plugin + assert utils.parameters_for(plugin) == ['physical_line', 'self', 'tree'] From cd18b9f175a3a73b03f58d4db7fd789c48c671bb Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 23 Feb 2016 23:25:12 -0600 Subject: [PATCH 169/364] Constrain our search for plugin type Flake8 has previously only ever relied on the first member of the parameters list to determine what kind of check it is using. As such, we constrain ourselves to just that parameter when checking and add properties for ast and logical line checks. --- flake8/plugins/manager.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 21639f9..05e5e2b 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -304,9 +304,27 @@ class Checkers(PluginTypeManager): """ for plugin in self.plugins.values(): parameters = utils.parameters_for(plugin) - if argument_name in parameters: + if argument_name == parameters[0]: yield plugin + @property + def ast_plugins(self): + """List of plugins that expect the AST tree.""" + plugins = getattr(self, '_ast_plugins', []) + if not plugins: + plugins = list(self.checks_expecting('tree')) + self._ast_plugins = plugins + return plugins + + @property + def logical_line_plugins(self): + """List of plugins that expect the logical lines.""" + plugins = getattr(self, '_logical_line_plugins', []) + if not plugins: + plugins = list(self.checks_expecting('logical_line')) + self._logical_line_plugins = plugins + return plugins + @property def physical_line_plugins(self): """List of plugins that expect the physical lines.""" From a4e984dbd258faf125a115bae1561db3fbd8e934 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 25 Feb 2016 09:06:45 -0600 Subject: [PATCH 170/364] Add and fix documentation - Add more documentation around utils functions - Fix documentation about default formatting plugins - Add extra documentation of filenames_from predicate parameter - Add test for the default parameter of flake8.utils.fnmatch --- docs/source/internal/formatters.rst | 8 ++-- docs/source/internal/plugin_handling.rst | 2 + docs/source/internal/utils.rst | 53 ++++++++++++++++++++++++ flake8/utils.py | 3 +- tests/unit/test_utils.py | 6 +++ 5 files changed, 67 insertions(+), 5 deletions(-) diff --git a/docs/source/internal/formatters.rst b/docs/source/internal/formatters.rst index caa5718..d54cf87 100644 --- a/docs/source/internal/formatters.rst +++ b/docs/source/internal/formatters.rst @@ -30,7 +30,7 @@ The former allows us to inspect the value provided to ``--format`` by the user and alter our own format based on that value. The second simply uses that format string to format the error. -.. autoclass:: flake8.formatters.default.Default +.. autoclass:: flake8.formatting.default.Default :members: Pylint Formatter @@ -39,9 +39,9 @@ Pylint Formatter The |PylintFormatter| simply defines the default Pylint format string from pep8: ``'%(path)s:%(row)d: [%(code)s] %(text)s'``. -.. autoclass:: flake8.formatters.default.Pylint +.. autoclass:: flake8.formatting.default.Pylint :members: -.. |DefaultFormatter| replace:: :class:`~flake8.formatters.default.Default` -.. |PylintFormatter| replace:: :class:`~flake8.formatters.default.Pylint` +.. |DefaultFormatter| replace:: :class:`~flake8.formatting.default.Default` +.. |PylintFormatter| replace:: :class:`~flake8.formatting.default.Pylint` diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst index e430bfb..b3b9b0e 100644 --- a/docs/source/internal/plugin_handling.rst +++ b/docs/source/internal/plugin_handling.rst @@ -98,8 +98,10 @@ API Documentation :members: .. autoclass:: flake8.plugins.manager.Checkers + :members: .. autoclass:: flake8.plugins.manager.Listeners + :members: build_notifier .. autoclass:: flake8.plugins.manager.ReportFormatters diff --git a/docs/source/internal/utils.rst b/docs/source/internal/utils.rst index 69dee45..d8adeac 100644 --- a/docs/source/internal/utils.rst +++ b/docs/source/internal/utils.rst @@ -45,3 +45,56 @@ strings that should be paths. This function retrieves and caches the value provided on ``sys.stdin``. This allows plugins to use this to retrieve ``stdin`` if necessary. + +.. autofunction:: flake8.utils.is_windows + +This provides a convenient and explicitly named function that checks if we are +currently running on a Windows (or ``nt``) operating system. + +.. autofunction:: flake8.utils.is_using_stdin + +Another helpful function that is named only to be explicit given it is a very +trivial check, this checks if the user specified ``-`` in their arguments to +Flake8 to indicate we should read from stdin. + +.. autofunction:: flake8.utils.filenames_from + +When provided an argument to Flake8, we need to be able to traverse +directories in a convenient manner. For example, if someone runs + +.. code:: + + $ flake8 flake8/ + +Then they want us to check all of the files in the directory ``flake8/``. This +function will handle that while also handling the case where they specify a +file like: + +.. code:: + + $ flake8 flake8/__init__.py + + +.. autofunction:: flake8.utils.fnmatch + +The standard library's :func:`fnmatch.fnmatch` is excellent at deciding if a +filename matches a single pattern. In our use case, however, we typically have +a list of patterns and want to know if the filename matches any of them. This +function abstracts that logic away with a little extra logic. + +.. autofunction:: flake8.utils.parameters_for + +Flake8 analyzes the parameters to plugins to determine what input they are +expecting. Plugins may expect one of the following: + +- ``physical_line`` to receive the line as it appears in the file + +- ``logical_line`` to receive the logical line (not as it appears in the file) + +- ``tree`` to receive the abstract syntax tree (AST) for the file + +We also analyze the rest of the parameters to provide more detail to the +plugin. This function will return the parameters in a consistent way across +versions of Python and will handle both classes and functions that are used as +plugins. Further, if the plugin is a class, it will strip the ``self`` +argument so we can check the parameters of the plugin consistently. diff --git a/flake8/utils.py b/flake8/utils.py index cd1cf72..7e08e41 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -106,7 +106,8 @@ def filenames_from(arg, predicate=None): :param callable predicate: Predicate to use to filter out filenames. If the predicate returns ``True`` we will exclude the filename, otherwise we - will yield it. + will yield it. By default, we include every filename + generated. :returns: Generator of paths """ diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 615f327..28b8e02 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -64,6 +64,12 @@ def test_fnmatch(filename, patterns, expected): assert utils.fnmatch(filename, patterns) is expected +def test_fnmatch_returns_the_default_with_empty_default(): + """The default parameter should be returned when no patterns are given.""" + sentinel = object() + assert utils.fnmatch('file.py', [], default=sentinel) is sentinel + + def test_filenames_from_a_directory(): """Verify that filenames_from walks a directory.""" filenames = list(utils.filenames_from('flake8/')) From 8d36355611e1e5a7c1a81bbbf8560c01d9ead979 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 25 Feb 2016 11:14:41 -0600 Subject: [PATCH 171/364] Re-order project imports based on flake8-import-order --- flake8/main/cli.py | 2 +- flake8/options/aggregator.py | 2 +- flake8/plugins/manager.py | 6 +++--- flake8/plugins/pyflakes.py | 5 +++-- flake8/style_guide.py | 3 +-- setup.py | 6 ++++-- tox.ini | 1 + 7 files changed, 14 insertions(+), 11 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 72ca829..15eea8e 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -1,10 +1,10 @@ """Command-line implementation of flake8.""" import flake8 from flake8 import defaults +from flake8 import style_guide from flake8.options import aggregator from flake8.options import manager from flake8.plugins import manager as plugin_manager -from flake8 import style_guide def register_default_options(option_manager): diff --git a/flake8/options/aggregator.py b/flake8/options/aggregator.py index b344cf0..99d0cfe 100644 --- a/flake8/options/aggregator.py +++ b/flake8/options/aggregator.py @@ -5,8 +5,8 @@ applies the user-specified command-line configuration on top of it. """ import logging -from flake8.options import config from flake8 import utils +from flake8.options import config LOG = logging.getLogger(__name__) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 05e5e2b..aba5f16 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -2,11 +2,11 @@ import collections import logging -import pkg_resources - from flake8 import exceptions -from flake8.plugins import notifier from flake8 import utils +from flake8.plugins import notifier + +import pkg_resources LOG = logging.getLogger(__name__) diff --git a/flake8/plugins/pyflakes.py b/flake8/plugins/pyflakes.py index f512511..7025baa 100644 --- a/flake8/plugins/pyflakes.py +++ b/flake8/plugins/pyflakes.py @@ -1,6 +1,7 @@ """Plugin built-in to Flake8 to treat pyflakes as a plugin.""" # -*- coding: utf-8 -*- from __future__ import absolute_import + try: # The 'demandimport' breaks pyflakes and flake8.plugins.pyflakes from mercurial import demandimport @@ -10,11 +11,11 @@ else: demandimport.disable() import os +from flake8 import utils + import pyflakes import pyflakes.checker -from flake8 import utils - def patch_pyflakes(): """Add error codes to Pyflakes messages.""" diff --git a/flake8/style_guide.py b/flake8/style_guide.py index c3b3d5f..bbfe658 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -1,11 +1,10 @@ """Implementation of the StyleGuide used by Flake8.""" import collections +import enum import linecache import logging import re -import enum - from flake8 import utils __all__ = ( diff --git a/setup.py b/setup.py index 1f4a6a0..ec40eb6 100644 --- a/setup.py +++ b/setup.py @@ -1,10 +1,12 @@ """Packaging logic for Flake8.""" # -*- coding: utf-8 -*- from __future__ import with_statement -import setuptools + import sys -import flake8 +import setuptools + +import flake8 # noqa try: # Work around a traceback with Nose on Python 2.6 diff --git a/tox.ini b/tox.ini index 9ea64cf..d5e07cc 100644 --- a/tox.ini +++ b/tox.ini @@ -22,6 +22,7 @@ use_develop = false deps = flake8 flake8-docstrings + flake8-import-order commands = flake8 From de9f56addfc770ba511550546a7f607c26472e5a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 25 Feb 2016 14:41:37 -0600 Subject: [PATCH 172/364] Fix import ordering in test files --- tests/integration/test_aggregator.py | 4 ++-- tests/unit/test_config_file_finder.py | 7 ++++--- tests/unit/test_merged_config_parser.py | 7 ++++--- tests/unit/test_notifier.py | 4 ++-- tests/unit/test_option.py | 7 ++++--- tests/unit/test_option_manager.py | 4 ++-- tests/unit/test_plugin.py | 7 ++++--- tests/unit/test_plugin_manager.py | 4 ++-- tests/unit/test_plugin_type_manager.py | 7 ++++--- tests/unit/test_style_guide.py | 3 ++- tests/unit/test_utils.py | 7 ++++--- 11 files changed, 34 insertions(+), 27 deletions(-) diff --git a/tests/integration/test_aggregator.py b/tests/integration/test_aggregator.py index 3f64fa3..2186c35 100644 --- a/tests/integration/test_aggregator.py +++ b/tests/integration/test_aggregator.py @@ -1,12 +1,12 @@ """Test aggregation of config files and command-line options.""" import os -import pytest - from flake8.main import cli from flake8.options import aggregator from flake8.options import manager +import pytest + CLI_SPECIFIED_CONFIG = 'tests/fixtures/config_files/cli-specified.ini' diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 6ac1d16..58009f7 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -7,11 +7,12 @@ except ImportError: import os import sys -import mock -import pytest - from flake8.options import config +import mock + +import pytest + CLI_SPECIFIED_FILEPATH = 'tests/fixtures/config_files/cli-specified.ini' diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index 2c4c5fd..c64cae6 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -1,12 +1,13 @@ """Unit tests for flake8.options.config.MergedConfigParser.""" import os -import mock -import pytest - from flake8.options import config from flake8.options import manager +import mock + +import pytest + @pytest.fixture def optmanager(): diff --git a/tests/unit/test_notifier.py b/tests/unit/test_notifier.py index 6a162cf..8c001da 100644 --- a/tests/unit/test_notifier.py +++ b/tests/unit/test_notifier.py @@ -1,8 +1,8 @@ """Unit tests for the Notifier object.""" -import pytest - from flake8.plugins import notifier +import pytest + class _Listener(object): def __init__(self, error_code): diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py index 67e2255..45f9be7 100644 --- a/tests/unit/test_option.py +++ b/tests/unit/test_option.py @@ -1,9 +1,10 @@ """Unit tests for flake8.options.manager.Option.""" -import mock -import pytest - from flake8.options import manager +import mock + +import pytest + def test_to_optparse(): """Test conversion to an optparse.Option class.""" diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 99c5d18..1ba5442 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -2,10 +2,10 @@ import optparse import os -import pytest - from flake8.options import manager +import pytest + TEST_VERSION = '3.0.0b1' diff --git a/tests/unit/test_plugin.py b/tests/unit/test_plugin.py index 01454a1..0f6eec1 100644 --- a/tests/unit/test_plugin.py +++ b/tests/unit/test_plugin.py @@ -1,10 +1,11 @@ """Tests for flake8.plugins.manager.Plugin.""" -import mock -import pytest - from flake8 import exceptions from flake8.plugins import manager +import mock + +import pytest + def test_load_plugin_fallsback_on_old_setuptools(): """Verify we fallback gracefully to on old versions of setuptools.""" diff --git a/tests/unit/test_plugin_manager.py b/tests/unit/test_plugin_manager.py index 8991b96..5a50386 100644 --- a/tests/unit/test_plugin_manager.py +++ b/tests/unit/test_plugin_manager.py @@ -1,8 +1,8 @@ """Tests for flake8.plugins.manager.PluginManager.""" -import mock - from flake8.plugins import manager +import mock + def create_entry_point_mock(name): """Create a mocked EntryPoint.""" diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index f64e472..271ebc0 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -1,12 +1,13 @@ """Tests for flake8.plugins.manager.PluginTypeManager.""" import collections -import mock -import pytest - from flake8 import exceptions from flake8.plugins import manager +import mock + +import pytest + TEST_NAMESPACE = "testing.plugin-type-manager" diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index 03bb341..973281b 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -1,11 +1,12 @@ """Tests for the flake8.style_guide.StyleGuide class.""" import optparse +from flake8 import style_guide from flake8.formatting import base from flake8.plugins import notifier -from flake8 import style_guide import mock + import pytest diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 28b8e02..d69d939 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -1,12 +1,13 @@ """Tests for flake8's utils module.""" import os + +from flake8 import utils +from flake8.plugins import manager as plugin_manager + import mock import pytest -from flake8.plugins import manager as plugin_manager -from flake8 import utils - RELATIVE_PATHS = ["flake8", "pep8", "pyflakes", "mccabe"] From 62a78f4a97ef82b3a28d1478635fc4f501762657 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 25 Feb 2016 14:43:52 -0600 Subject: [PATCH 173/364] Add note to tox.ini for others --- tox.ini | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tox.ini b/tox.ini index d5e07cc..946d532 100644 --- a/tox.ini +++ b/tox.ini @@ -98,6 +98,10 @@ commands = # Flake8 Configuration [flake8] # Ignore some flake8-docstrings errors +# NOTE(sigmavirus24): While we're still using flake8 2.x, this ignore line +# defaults to selecting all other errors so we do not need select=E,F,W,I,D +# Once Flake8 3.0 is released and in a good state, we can use both and it will +# work well \o/ ignore = D203 # NOTE(sigmavirus24): Once we release 3.0.0 this exclude option can be specified # across multiple lines. Presently it cannot be specified across multiple lines. From 6a15bd00b52d44830cda9eb73411423210344fc3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 26 Feb 2016 08:21:09 -0600 Subject: [PATCH 174/364] Store plugin parameters on the plugin itself This allows us to access these from the checker module as well. --- flake8/plugins/manager.py | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index aba5f16..c453dbe 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -35,6 +35,7 @@ class Plugin(object): self.name = name self.entry_point = entry_point self._plugin = None + self._parameters = None def __repr__(self): """Provide an easy to read description of the current plugin.""" @@ -42,6 +43,13 @@ class Plugin(object): self.name, self.entry_point ) + @property + def parameters(self): + """List of arguments that need to be passed to the plugin.""" + if self._parameters is None: + self._parameters = utils.parameters_for(self) + return self._parameters + @property def plugin(self): """The loaded (and cached) plugin associated with the entry-point. @@ -303,8 +311,7 @@ class Checkers(PluginTypeManager): Find all checker plugins that are expecting a specific argument. """ for plugin in self.plugins.values(): - parameters = utils.parameters_for(plugin) - if argument_name == parameters[0]: + if argument_name == plugin.parameters[0]: yield plugin @property From b12f531da42df816f90f43197aa01d34aefb3d4f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 26 Feb 2016 08:52:20 -0600 Subject: [PATCH 175/364] Separate the check runner from file processor This separates concerns so that the check runner can rely on the file processor to store state and such. It introduces two logical collaborators and will allow us to keep feature parity with flake8 2's plugin design (where it could request any attribute from pep8.Checker). --- flake8/checker.py | 79 ++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 67 insertions(+), 12 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 07c90f4..bb13b13 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -179,7 +179,73 @@ class FileChecker(object): self.filename = filename self.checks = checks self.results = [] - self.lines = [] + self.processor = FileProcessor(filename) + + def report(self, error_code, line_number, column, text): + # type: (str, int, int, str) -> NoneType + """Report an error by storing it in the results list.""" + error = (error_code, self.filename, line_number, column, text) + self.results.append(error) + + def run_check(self, plugin): + """Run the check in a single plugin.""" + arguments = {} + for parameter in plugin.parameters: + arguments[parameter] = self.attributes[parameter] + return plugin.execute(**arguments) + + def run_checks(self): + """Run checks against the file.""" + self.run_ast_checks() + self.run_physical_checks() + self.run_logical_checks() + + def run_ast_checks(self): + """Run checks that require an abstract syntax tree.""" + pass + + def run_physical_checks(self): + """Run checks that require the physical line.""" + pass + + def run_logical_checks(self): + """Run checks that require the logical line from a file.""" + pass + + +class FileProcessor(object): + """Processes a file and holdes state. + + This processes a file by generating tokens, logical and physical lines, + and AST trees. This also provides a way of passing state about the file + to checks expecting that state. Any public attribute on this object can + be requested by a plugin. The known public attributes are: + + - multiline + - max_line_length + - tokens + - indent_level + - indect_char + - noqa + - verbose + - line_number + - total_lines + - previous_logical + - logical_line + - previous_indent_level + - blank_before + - blank_lines + """ + + def __init__(self, filename): + """Initialice our file processor. + + :param str filename: + Name of the file to process + """ + self.filename = filename + self.lines = self.read_lines() + self.strip_utf_bom() def read_lines(self): # type: () -> List[str] @@ -235,17 +301,6 @@ class FileChecker(object): """Read the lines from standard in.""" return utils.stdin_get_value().splitlines(True) - def report(self, error_code, line_number, column, text): - # type: (str, int, int, str) -> NoneType - """Report an error by storing it in the results list.""" - error = (error_code, self.filename, line_number, column, text) - self.results.append(error) - - def run_checks(self): - """Run checks against the file.""" - self.lines = self.read_lines() - self.strip_utf_bom() - def strip_utf_bom(self): # type: () -> NoneType """Strip the UTF bom from the lines of the file.""" From 12e71b037218904b2fdf9ff51c7a0982adde020c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 26 Feb 2016 23:16:25 -0600 Subject: [PATCH 176/364] Incorporate more parsing logic from pycodestyle Presently we're working on having two singly-responsible classes that will be easy to test and ideally easier to reason about. --- flake8/checker.py | 150 ++++++++++++++++++++++++++++++------------- flake8/defaults.py | 3 + flake8/exceptions.py | 18 ++++++ 3 files changed, 125 insertions(+), 46 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index bb13b13..bf72362 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -10,6 +10,8 @@ try: except ImportError: multiprocessing = None +from flake8 import defaults +from flake8 import exceptions from flake8 import utils LOG = logging.getLogger(__name__) @@ -128,7 +130,7 @@ class Manager(object): paths = self.arguments filename_patterns = self.options.filename self.checkers = [ - FileChecker(filename, self.checks) + FileChecker(filename, self.checks, self.options) for argument in paths for filename in utils.filenames_from(argument, self.is_path_excluded) @@ -165,8 +167,7 @@ class Manager(object): class FileChecker(object): """Manage running checks for a file and aggregate the results.""" - def __init__(self, filename, checks): - # type: (str, flake8.plugins.manager.Checkers) -> NoneType + def __init__(self, filename, checks, options): """Initialize our file checker. :param str filename: @@ -178,8 +179,24 @@ class FileChecker(object): """ self.filename = filename self.checks = checks + self.options = options self.results = [] - self.processor = FileProcessor(filename) + self.processor = self._make_processor() + + def _make_processor(self): + try: + return FileProcessor(self.filename, self.options) + except IOError: + # If we can not read the file due to an IOError (e.g., the file + # does not exist or we do not have the permissions to open it) + # then we need to format that exception for the user. + # NOTE(sigmavirus24): Historically, pep8 has always reported this + # as an E902. We probably *want* a better error code for this + # going forward. + (exc_type, exception) = sys.exc_info()[:2] + message = '{0}: {1}'.format(exc_type.__name__, exception) + self.report('E902', 0, 0, message) + return None def report(self, error_code, line_number, column, text): # type: (str, int, int, str) -> NoneType @@ -196,21 +213,12 @@ class FileChecker(object): def run_checks(self): """Run checks against the file.""" - self.run_ast_checks() - self.run_physical_checks() - self.run_logical_checks() - - def run_ast_checks(self): - """Run checks that require an abstract syntax tree.""" - pass - - def run_physical_checks(self): - """Run checks that require the physical line.""" - pass - - def run_logical_checks(self): - """Run checks that require the logical line from a file.""" - pass + try: + for token in self.processor.generate_tokens(): + pass + except exceptions.InvalidSyntax as exc: + self.report(exc.error_code, exc.line_number, exc.column_number, + exc.error_message) class FileProcessor(object): @@ -221,23 +229,23 @@ class FileProcessor(object): to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are: - - multiline - - max_line_length - - tokens - - indent_level - - indect_char - - noqa - - verbose - - line_number - - total_lines - - previous_logical - - logical_line - - previous_indent_level - blank_before - blank_lines + - indect_char + - indent_level + - line_number + - logical_line + - max_line_length + - multiline + - noqa + - previous_indent_level + - previous_logical + - tokens + - total_lines + - verbose """ - def __init__(self, filename): + def __init__(self, filename, options): """Initialice our file processor. :param str filename: @@ -246,6 +254,69 @@ class FileProcessor(object): self.filename = filename self.lines = self.read_lines() self.strip_utf_bom() + self.options = options + + # Defaults for public attributes + #: Number of preceding blank lines + self.blank_before = 0 + #: Number of blank lines + self.blank_lines = 0 + #: Character used for indentation + self.indent_char = None + #: Current level of indentation + self.indent_level = 0 + #: Line number in the file + self.line_number = 0 + #: Current logical line + self.logical_line = '' + #: Maximum line length as configured by the user + self.max_line_length = options.max_line_length + #: Whether the current physical line is multiline + self.multiline = False + #: Whether or not we're observing NoQA + self.noqa = False + #: Previous level of indentation + self.previous_indent_level = 0 + #: Previous logical line + self.previous_logical = '' + #: Current set of tokens + self.tokens = [] + #: Total number of lines in the file + self.total_lines = len(self.lines) + #: Verbosity level of Flake8 + self.verbosity = options.verbosity + + def generate_tokens(self): + """Tokenize the file and yield the tokens. + + :raises flake8.exceptions.InvalidSyntax: + If a :class:`tokenize.TokenError` is raised while generating + tokens. + """ + try: + for token in tokenize.generate_tokens(self.next_line): + if token[2][0] > self.total_lines: + break + self.tokens.append(token) + yield token + # NOTE(sigmavirus24): pycodestyle was catching both a SyntaxError + # and a tokenize.TokenError. In looking a the source on Python 2 and + # Python 3, the SyntaxError should never arise from generate_tokens. + # If we were using tokenize.tokenize, we would have to catch that. Of + # course, I'm going to be unsurprised to be proven wrong at a later + # date. + except tokenize.TokenError as exc: + raise exceptions.InvalidSyntax(exc.message, exception=exc) + + def next_line(self): + """Get the next line from the list.""" + if self.line_number >= self.total_lines: + return '' + line = self.lines[self.line_number] + self.line_number += 1 + if self.indent_char is None and line[:1] in defaults.WHITESPACE: + self.indent_char = line[0] + return line def read_lines(self): # type: () -> List[str] @@ -281,20 +352,7 @@ class FileProcessor(object): readlines = self._readlines_py2 elif (3, 0) <= sys.version_info < (4, 0): readlines = self._readlines_py3 - - try: - return readlines() - except IOError: - # If we can not read the file due to an IOError (e.g., the file - # does not exist or we do not have the permissions to open it) - # then we need to format that exception for the user. - # NOTE(sigmavirus24): Historically, pep8 has always reported this - # as an E902. We probably *want* a better error code for this - # going forward. - (exc_type, exception) = sys.exc_info()[:2] - message = '{0}: {1}'.format(exc_type.__name__, exception) - self.report('E902', 0, 0, message) - return [] + return readlines() def read_lines_from_stdin(self): # type: () -> List[str] diff --git a/flake8/defaults.py b/flake8/defaults.py index ac798b5..62939b5 100644 --- a/flake8/defaults.py +++ b/flake8/defaults.py @@ -3,3 +3,6 @@ EXCLUDE = '.svn,CVS,.bzr,.hg,.git,__pycache__,.tox' IGNORE = 'E121,E123,E126,E226,E24,E704' MAX_LINE_LENGTH = 79 + +# Other consants +WHITESPACE = frozenset(' \t') diff --git a/flake8/exceptions.py b/flake8/exceptions.py index ac78769..18ee90c 100644 --- a/flake8/exceptions.py +++ b/flake8/exceptions.py @@ -23,3 +23,21 @@ class FailedToLoadPlugin(Flake8Exception): """Return a nice string for our exception.""" return self.FORMAT % {'name': self.ep_name, 'exc': self.original_exception} + + +class InvalidSyntax(Flake8Exception): + """Exception raised when tokenizing a file fails.""" + + def __init__(self, *args, **kwargs): + """Initialize our InvalidSyntax exception.""" + self.original_exception = kwargs.pop('exception') + self.error_code = 'E902' + self.line_number = 1 + self.column_number = 0 + try: + self.error_message = self.original_exception.message + except AttributeError: + # On Python 3, the IOError is an OSError which has a + # strerror attribute instead of a message attribute + self.error_message = self.original_exception.strerror + super(InvalidSyntax, self).__init__(*args, **kwargs) From 074739de276283fea9a574e08db82e404ddf4fc4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 27 Feb 2016 00:06:19 -0600 Subject: [PATCH 177/364] Accept the StyleGuide instead of options pep8's checker has the noqa logic which we've correctly placed on the StyleGuide object. By passing the StyleGuide in to our checkers we can have the checkers pass the physical line to the StyleGuide so we can eliminate our usage of linecache. --- flake8/checker.py | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index bf72362..9f2154d 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -36,13 +36,13 @@ class Manager(object): together and make our output deterministic. """ - def __init__(self, options, arguments, checker_plugins): + def __init__(self, style_guide, arguments, checker_plugins): """Initialize our Manager instance. - :param options: - The options parsed from config files and CLI. - :type options: - optparse.Values + :param style_guide: + The instantiated style guide for this instance of Flake8. + :type style_guide: + flake8.style_guide.StyleGuide :param list arguments: The extra arguments parsed from the CLI (if any) :param checker_plugins: @@ -51,7 +51,8 @@ class Manager(object): flake8.plugins.manager.Checkers """ self.arguments = arguments - self.options = options + self.style_guide = style_guide + self.options = style_guide.options self.checks = checker_plugins self.jobs = self._job_count() self.process_queue = None @@ -130,7 +131,7 @@ class Manager(object): paths = self.arguments filename_patterns = self.options.filename self.checkers = [ - FileChecker(filename, self.checks, self.options) + FileChecker(filename, self.checks, self.style_guide) for argument in paths for filename in utils.filenames_from(argument, self.is_path_excluded) @@ -167,7 +168,7 @@ class Manager(object): class FileChecker(object): """Manage running checks for a file and aggregate the results.""" - def __init__(self, filename, checks, options): + def __init__(self, filename, checks, style_guide): """Initialize our file checker. :param str filename: @@ -179,13 +180,13 @@ class FileChecker(object): """ self.filename = filename self.checks = checks - self.options = options + self.style_guide = style_guide self.results = [] self.processor = self._make_processor() def _make_processor(self): try: - return FileProcessor(self.filename, self.options) + return FileProcessor(self.filename, self.style_guide.options) except IOError: # If we can not read the file due to an IOError (e.g., the file # does not exist or we do not have the permissions to open it) From 6ac955dfd4db7632018211c8d55bac678081426d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 28 Feb 2016 00:55:47 -0600 Subject: [PATCH 178/364] Continue porting more logic from pep8 --- flake8/checker.py | 77 ++++++++++++++++++++++++++++++++++++++++--- flake8/style_guide.py | 6 ++-- flake8/utils.py | 22 +++++++++++++ 3 files changed, 98 insertions(+), 7 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 9f2154d..38a4634 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -1,4 +1,5 @@ """Checker Manager and Checker classes.""" +import contextlib import io import logging import os @@ -205,22 +206,57 @@ class FileChecker(object): error = (error_code, self.filename, line_number, column, text) self.results.append(error) - def run_check(self, plugin): + def run_check(self, plugin, **arguments): """Run the check in a single plugin.""" - arguments = {} - for parameter in plugin.parameters: - arguments[parameter] = self.attributes[parameter] + self.processor.keyword_arguments_for(plugin.parameters, arguments) return plugin.execute(**arguments) + def run_physical_checks(self, physical_line): + for plugin in self.checks.physical_line_plugins: + result = self.run_check(plugin, physical_line=physical_line) + if result is not None: + column_offset, text = result + error_code, error_text = text.split(' ', 1) + self.report( + error_code=error_code, + line_number=self.processor.line_number, + column=column_offset, + text=error_text, + ) + + self.processor.check_physical_error(error_code, physical_line) + def run_checks(self): """Run checks against the file.""" try: for token in self.processor.generate_tokens(): - pass + self.check_physical_eol(token) except exceptions.InvalidSyntax as exc: self.report(exc.error_code, exc.line_number, exc.column_number, exc.error_message) + def check_physical_eol(self, token): + """Run physical checks if and only if it is at the end of the line.""" + if utils.is_eol_token(token): + # Obviously, a newline token ends a single physical line. + self.run_physical_checks(token[4]) + elif utils.is_multiline_string(token): + # Less obviously, a string that contains newlines is a + # multiline string, either triple-quoted or with internal + # newlines backslash-escaped. Check every physical line in the + # string *except* for the last one: its newline is outside of + # the multiline string, so we consider it a regular physical + # line, and will check it like any other physical line. + # + # Subtleties: + # - have to wind self.line_number back because initially it + # points to the last line of the string, and we want + # check_physical() to give accurate feedback + line_no = token[2][0] + with self.processor.inside_multiline(line_number=line_no): + for line in self.processor.split_line(token): + self.run_physical_checks(line + '\n') + class FileProcessor(object): """Processes a file and holdes state. @@ -287,6 +323,37 @@ class FileProcessor(object): #: Verbosity level of Flake8 self.verbosity = options.verbosity + @contextlib.contextmanager + def inside_multiline(self, line_number): + """Context-manager to toggle the multiline attribute.""" + self.line_number = line_number + self.multiline = True + yield + self.multiline = False + + def split_line(self, token): + """Split a physical line's line based on new-lines. + + This also auto-increments the line number for the caller. + """ + for line in token[1].split('\n')[:-1]: + yield line + self.line_number += 1 + + def keyword_arguments_for(self, parameters, arguments=None): + """Generate the keyword arguments for a list of parameters.""" + if arguments is None: + arguments = {} + for param in parameters: + if param not in arguments: + arguments[param] = getattr(self, param) + return arguments + + def check_physical_error(self, error_code, line): + """Update attributes based on error code and line.""" + if error_code == 'E101': + self.indent_char = line[0] + def generate_tokens(self): """Tokenize the file and yield the tokens. diff --git a/flake8/style_guide.py b/flake8/style_guide.py index bbfe658..51c3a8e 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -157,14 +157,16 @@ class StyleGuide(object): LOG.debug('"%s" will be "%s"', code, decision) return decision - def is_inline_ignored(self, error): + def is_inline_ignored(self, error, physical_line=None): # type: (Error) -> bool """Determine if an comment has been added to ignore this line.""" # TODO(sigmavirus24): Determine how to handle stdin with linecache if self.options.disable_noqa: return False - physical_line = linecache.getline(error.filename, error.line_number) + if physical_line is None: + physical_line = linecache.getline(error.filename, + error.line_number) noqa_match = self.NOQA_INLINE_REGEXP.search(physical_line) if noqa_match is None: LOG.debug('%r is not inline ignored', error) diff --git a/flake8/utils.py b/flake8/utils.py index 7e08e41..a3be0ea 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -4,6 +4,7 @@ import inspect import io import os import sys +import tokenize def parse_comma_separated_list(value): @@ -183,3 +184,24 @@ def parameters_for(plugin): parameters.remove('self') return parameters + +NEWLINE = frozenset([tokenize.NL, tokenize.NEWLINE]) +# Work around Python < 2.6 behaviour, which does not generate NL after +# a comment which is on a line by itself. +COMMENT_WITH_NL = tokenize.generate_tokens(['#\n'].pop).send(None)[1] == '#\n' + + +def is_eol_token(token): + """Check if the token is an end-of-line token.""" + return token[0] in NEWLINE or token[4][token[3][1]:].lstrip() == '\\\n' + +if COMMENT_WITH_NL: # If on Python 2.6 + def is_eol_token(token, _is_eol_token=is_eol_token): + """Check if the token is an end-of-line token.""" + return (_is_eol_token(token) or + (token[0] == tokenize.COMMENT and token[1] == token[4])) + + +def is_multiline_string(token): + """Check if this is a multiline string.""" + return token[0] == tokenize.STRING and '\n' in token[1] From 0c894cc8bfb066b8e45daa7b2c6ebde480b1eccf Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 1 Mar 2016 21:27:36 -0600 Subject: [PATCH 179/364] Pull more logic out of pep8 --- flake8/checker.py | 98 +++++++++++++++++++++++++++++++++++++++++++---- flake8/utils.py | 18 +++++++++ 2 files changed, 109 insertions(+), 7 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 38a4634..a6c14ec 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -201,40 +201,112 @@ class FileChecker(object): return None def report(self, error_code, line_number, column, text): - # type: (str, int, int, str) -> NoneType + # type: (str, int, int, str) -> str """Report an error by storing it in the results list.""" + if error_code is None: + error_code, text = text.split(' ', 1) error = (error_code, self.filename, line_number, column, text) self.results.append(error) + return error_code def run_check(self, plugin, **arguments): """Run the check in a single plugin.""" self.processor.keyword_arguments_for(plugin.parameters, arguments) return plugin.execute(**arguments) + def run_logical_checks(self): + """Run all checks expecting a logical line.""" + for plugin in self.checks.logical_line_plugins: + result = self.run_check(plugin) # , logical_line=logical_line) + if result is not None: + column_offset, text = result + self.report( + error_code=None, + line_number=self.processor.line_number, + column=column_offset, + text=text, + ) + def run_physical_checks(self, physical_line): + """Run all checks for a given physical line.""" for plugin in self.checks.physical_line_plugins: result = self.run_check(plugin, physical_line=physical_line) if result is not None: column_offset, text = result - error_code, error_text = text.split(' ', 1) - self.report( - error_code=error_code, + error_code = self.report( + error_code=None, line_number=self.processor.line_number, column=column_offset, - text=error_text, + text=text, ) self.processor.check_physical_error(error_code, physical_line) + def _log_token(self, token): + if token[2][0] == token[3][0]: + pos = '[%s:%s]' % (token[2][1] or '', token[3][1]) + else: + pos = 'l.%s' % token[3][0] + LOG.debug('l.%s\t%s\t%s\t%r' % + (token[2][0], pos, tokenize.tok_name[token[0]], + token[1])) + + def process_tokens(self): + """Process tokens and trigger checks. + + This can raise a :class:`flake8.exceptions.InvalidSyntax` exception. + Instead of using this directly, you should use + :meth:`flake8.checker.FileChecker.run_checks`. + """ + parens = 0 + processor = self.processor + for token in processor.generate_tokens(): + self.check_physical_eol(token) + token_type, text = token[0:2] + self._log_token(token) + if token_type == tokenize.OP: + parens = utils.count_parentheses(parens, text) + elif parens == 0: + if utils.token_is_newline(token): + self.handle_newline(token_type) + elif (utils.token_is_comment(token) and + len(processor.tokens) == 1): + self.handle_comment(token, text) + + if processor.tokens: + # If any tokens are left over, process them + self.run_physical_checks(processor.lines[-1]) + self.run_logical_checks() + def run_checks(self): """Run checks against the file.""" try: - for token in self.processor.generate_tokens(): - self.check_physical_eol(token) + self.process_tokens() except exceptions.InvalidSyntax as exc: self.report(exc.error_code, exc.line_number, exc.column_number, exc.error_message) + def handle_comment(self, token, token_text): + """Handle the logic when encountering a comment token.""" + # The comment also ends a physical line + token = list(token) + token[1] = token_text.rstrip('\r\n') + token[3] = (token[2][0], token[2][1] + len(token[1])) + self.processor.tokens = [tuple(token)] + self.run_logical_checks() + + def handle_newline(self, token_type): + """Handle the logic when encountering a newline token.""" + if token_type == tokenize.NEWLINE: + self.check_logical() + self.processor.reset_blank_before() + elif len(self.processor.tokens) == 1: + # The physical line contains only this token. + self.processor.visited_new_blank_line() + self.processor.delete_first_token() + else: + self.run_logical_checks() + def check_physical_eol(self, token): """Run physical checks if and only if it is at the end of the line.""" if utils.is_eol_token(token): @@ -331,6 +403,18 @@ class FileProcessor(object): yield self.multiline = False + def reset_blank_before(self): + """Reset the blank_before attribute to zero.""" + self.blank_before = 0 + + def delete_first_token(self): + """Delete the first token in the list of tokens.""" + del self.tokens[0] + + def visited_new_blank_line(self): + """Note that we visited a new blank line.""" + self.blank_lines += 1 + def split_line(self, token): """Split a physical line's line based on new-lines. diff --git a/flake8/utils.py b/flake8/utils.py index a3be0ea..97b766a 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -205,3 +205,21 @@ if COMMENT_WITH_NL: # If on Python 2.6 def is_multiline_string(token): """Check if this is a multiline string.""" return token[0] == tokenize.STRING and '\n' in token[1] + + +def token_is_newline(token): + """Check if the token type is a newline token type.""" + return token[0] in NEWLINE + + +def token_is_comment(token): + """Check if the token type is a comment.""" + return COMMENT_WITH_NL and token[0] == tokenize.COMMENT + + +def count_parentheses(current_parentheses_count, token_text): + """Count the number of parentheses.""" + if token_text in '([{': + return current_parentheses_count + 1 + elif token_text in '}])': + return current_parentheses_count - 1 From 23c9091b1ab5119643f1dd742687f2ae452a6f4c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 2 Mar 2016 23:28:24 -0600 Subject: [PATCH 180/364] Slowly working through pep8.Checker.check_logical --- flake8/checker.py | 51 ++++++++++++++++++++++++++++++++++++++++++++++- flake8/utils.py | 20 +++++++++++++++++++ 2 files changed, 70 insertions(+), 1 deletion(-) diff --git a/flake8/checker.py b/flake8/checker.py index a6c14ec..82f9e79 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -17,6 +17,9 @@ from flake8 import utils LOG = logging.getLogger(__name__) +SKIP_TOKENS = frozenset([tokenize.NL, tokenize.NEWLINE, tokenize.INDENT, + tokenize.DEDENT]) + class Manager(object): """Manage the parallelism and checker instances for each plugin and file. @@ -216,8 +219,15 @@ class FileChecker(object): def run_logical_checks(self): """Run all checks expecting a logical line.""" + comments, logical_line, mapping = self.processor.build_logical_line() + if not mapping: + return + self.processor.update_state(mapping) + + LOG.debug('Logical line: "%s"', logical_line.rstrip()) + for plugin in self.checks.logical_line_plugins: - result = self.run_check(plugin) # , logical_line=logical_line) + result = self.run_check(plugin, logical_line=logical_line) if result is not None: column_offset, text = result self.report( @@ -415,6 +425,45 @@ class FileProcessor(object): """Note that we visited a new blank line.""" self.blank_lines += 1 + def build_logical_line_tokens(self): + """Build the mapping, comments, and logical line lists.""" + logical = [] + comments = [] + length = 0 + previous_row = previous_column = mapping = None + for token_type, text, start, end, line in self.tokens: + if token_type in SKIP_TOKENS: + continue + if not mapping: + mapping = [(0, start)] + if token_type == tokenize.COMMENT: + comments.append(text) + continue + if token_type == tokenize.STRING: + text = utils.mutate_string(text) + if previous_row: + (start_row, start_column) = start + if previous_row != start_row: + row_index = previous_row - 1 + column_index = previous_column - 1 + previous_text = self.lines[row_index][column_index] + if (previous_text == ',' or + (previous_text not in '{[(' and + text not in '}])')): + text = ' ' + text + elif previous_column != start_column: + text = line[previous_column:start_column] + text + logical.append(text) + length += len(text) + mapping.append((length, end)) + (previous_row, previous_column) = end + return comments, logical, mapping + + def build_logical_line(self): + """Build a logical line from the current tokens list.""" + comments, logical, mapping_list = self.build_logical_line_tokens() + return ''.join(comments), ''.join(logical), mapping_list + def split_line(self, token): """Split a physical line's line based on new-lines. diff --git a/flake8/utils.py b/flake8/utils.py index 97b766a..fe024d1 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -223,3 +223,23 @@ def count_parentheses(current_parentheses_count, token_text): return current_parentheses_count + 1 elif token_text in '}])': return current_parentheses_count - 1 + + +def mutate_string(text): + """Replace contents with 'xxx' to prevent syntax matching. + + >>> mute_string('"abc"') + '"xxx"' + >>> mute_string("'''abc'''") + "'''xxx'''" + >>> mute_string("r'abc'") + "r'xxx'" + """ + # String modifiers (e.g. u or r) + start = text.index(text[-1]) + 1 + end = len(text) - 1 + # Triple quotes + if text[-3:] in ('"""', "'''"): + start += 2 + end -= 2 + return text[:start] + 'x' * (end - start) + text[end:] From f7a8b7ade769f93b6d4a9e0f976042a4e45b304b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 4 Mar 2016 20:03:05 -0600 Subject: [PATCH 181/364] Move processor to its own module --- flake8/checker.py | 279 +++------------------------------------- flake8/processor.py | 300 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 314 insertions(+), 265 deletions(-) create mode 100644 flake8/processor.py diff --git a/flake8/checker.py b/flake8/checker.py index 82f9e79..69bab17 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -1,6 +1,4 @@ """Checker Manager and Checker classes.""" -import contextlib -import io import logging import os import sys @@ -11,15 +9,12 @@ try: except ImportError: multiprocessing = None -from flake8 import defaults from flake8 import exceptions +from flake8 import processor from flake8 import utils LOG = logging.getLogger(__name__) -SKIP_TOKENS = frozenset([tokenize.NL, tokenize.NEWLINE, tokenize.INDENT, - tokenize.DEDENT]) - class Manager(object): """Manage the parallelism and checker instances for each plugin and file. @@ -190,7 +185,8 @@ class FileChecker(object): def _make_processor(self): try: - return FileProcessor(self.filename, self.style_guide.options) + return processor.FileProcessor(self.filename, + self.style_guide.options) except IOError: # If we can not read the file due to an IOError (e.g., the file # does not exist or we do not have the permissions to open it) @@ -252,15 +248,6 @@ class FileChecker(object): self.processor.check_physical_error(error_code, physical_line) - def _log_token(self, token): - if token[2][0] == token[3][0]: - pos = '[%s:%s]' % (token[2][1] or '', token[3][1]) - else: - pos = 'l.%s' % token[3][0] - LOG.debug('l.%s\t%s\t%s\t%r' % - (token[2][0], pos, tokenize.tok_name[token[0]], - token[1])) - def process_tokens(self): """Process tokens and trigger checks. @@ -269,23 +256,23 @@ class FileChecker(object): :meth:`flake8.checker.FileChecker.run_checks`. """ parens = 0 - processor = self.processor - for token in processor.generate_tokens(): + file_processor = self.processor + for token in file_processor.generate_tokens(): self.check_physical_eol(token) token_type, text = token[0:2] - self._log_token(token) + processor.log_token(token) if token_type == tokenize.OP: - parens = utils.count_parentheses(parens, text) + parens = processor.count_parentheses(parens, text) elif parens == 0: - if utils.token_is_newline(token): + if processor.token_is_newline(token): self.handle_newline(token_type) - elif (utils.token_is_comment(token) and - len(processor.tokens) == 1): + elif (processor.token_is_comment(token) and + len(file_processor.tokens) == 1): self.handle_comment(token, text) - if processor.tokens: + if file_processor.tokens: # If any tokens are left over, process them - self.run_physical_checks(processor.lines[-1]) + self.run_physical_checks(file_processor.lines[-1]) self.run_logical_checks() def run_checks(self): @@ -319,10 +306,10 @@ class FileChecker(object): def check_physical_eol(self, token): """Run physical checks if and only if it is at the end of the line.""" - if utils.is_eol_token(token): + if processor.is_eol_token(token): # Obviously, a newline token ends a single physical line. self.run_physical_checks(token[4]) - elif utils.is_multiline_string(token): + elif processor.is_multiline_string(token): # Less obviously, a string that contains newlines is a # multiline string, either triple-quoted or with internal # newlines backslash-escaped. Check every physical line in the @@ -338,241 +325,3 @@ class FileChecker(object): with self.processor.inside_multiline(line_number=line_no): for line in self.processor.split_line(token): self.run_physical_checks(line + '\n') - - -class FileProcessor(object): - """Processes a file and holdes state. - - This processes a file by generating tokens, logical and physical lines, - and AST trees. This also provides a way of passing state about the file - to checks expecting that state. Any public attribute on this object can - be requested by a plugin. The known public attributes are: - - - blank_before - - blank_lines - - indect_char - - indent_level - - line_number - - logical_line - - max_line_length - - multiline - - noqa - - previous_indent_level - - previous_logical - - tokens - - total_lines - - verbose - """ - - def __init__(self, filename, options): - """Initialice our file processor. - - :param str filename: - Name of the file to process - """ - self.filename = filename - self.lines = self.read_lines() - self.strip_utf_bom() - self.options = options - - # Defaults for public attributes - #: Number of preceding blank lines - self.blank_before = 0 - #: Number of blank lines - self.blank_lines = 0 - #: Character used for indentation - self.indent_char = None - #: Current level of indentation - self.indent_level = 0 - #: Line number in the file - self.line_number = 0 - #: Current logical line - self.logical_line = '' - #: Maximum line length as configured by the user - self.max_line_length = options.max_line_length - #: Whether the current physical line is multiline - self.multiline = False - #: Whether or not we're observing NoQA - self.noqa = False - #: Previous level of indentation - self.previous_indent_level = 0 - #: Previous logical line - self.previous_logical = '' - #: Current set of tokens - self.tokens = [] - #: Total number of lines in the file - self.total_lines = len(self.lines) - #: Verbosity level of Flake8 - self.verbosity = options.verbosity - - @contextlib.contextmanager - def inside_multiline(self, line_number): - """Context-manager to toggle the multiline attribute.""" - self.line_number = line_number - self.multiline = True - yield - self.multiline = False - - def reset_blank_before(self): - """Reset the blank_before attribute to zero.""" - self.blank_before = 0 - - def delete_first_token(self): - """Delete the first token in the list of tokens.""" - del self.tokens[0] - - def visited_new_blank_line(self): - """Note that we visited a new blank line.""" - self.blank_lines += 1 - - def build_logical_line_tokens(self): - """Build the mapping, comments, and logical line lists.""" - logical = [] - comments = [] - length = 0 - previous_row = previous_column = mapping = None - for token_type, text, start, end, line in self.tokens: - if token_type in SKIP_TOKENS: - continue - if not mapping: - mapping = [(0, start)] - if token_type == tokenize.COMMENT: - comments.append(text) - continue - if token_type == tokenize.STRING: - text = utils.mutate_string(text) - if previous_row: - (start_row, start_column) = start - if previous_row != start_row: - row_index = previous_row - 1 - column_index = previous_column - 1 - previous_text = self.lines[row_index][column_index] - if (previous_text == ',' or - (previous_text not in '{[(' and - text not in '}])')): - text = ' ' + text - elif previous_column != start_column: - text = line[previous_column:start_column] + text - logical.append(text) - length += len(text) - mapping.append((length, end)) - (previous_row, previous_column) = end - return comments, logical, mapping - - def build_logical_line(self): - """Build a logical line from the current tokens list.""" - comments, logical, mapping_list = self.build_logical_line_tokens() - return ''.join(comments), ''.join(logical), mapping_list - - def split_line(self, token): - """Split a physical line's line based on new-lines. - - This also auto-increments the line number for the caller. - """ - for line in token[1].split('\n')[:-1]: - yield line - self.line_number += 1 - - def keyword_arguments_for(self, parameters, arguments=None): - """Generate the keyword arguments for a list of parameters.""" - if arguments is None: - arguments = {} - for param in parameters: - if param not in arguments: - arguments[param] = getattr(self, param) - return arguments - - def check_physical_error(self, error_code, line): - """Update attributes based on error code and line.""" - if error_code == 'E101': - self.indent_char = line[0] - - def generate_tokens(self): - """Tokenize the file and yield the tokens. - - :raises flake8.exceptions.InvalidSyntax: - If a :class:`tokenize.TokenError` is raised while generating - tokens. - """ - try: - for token in tokenize.generate_tokens(self.next_line): - if token[2][0] > self.total_lines: - break - self.tokens.append(token) - yield token - # NOTE(sigmavirus24): pycodestyle was catching both a SyntaxError - # and a tokenize.TokenError. In looking a the source on Python 2 and - # Python 3, the SyntaxError should never arise from generate_tokens. - # If we were using tokenize.tokenize, we would have to catch that. Of - # course, I'm going to be unsurprised to be proven wrong at a later - # date. - except tokenize.TokenError as exc: - raise exceptions.InvalidSyntax(exc.message, exception=exc) - - def next_line(self): - """Get the next line from the list.""" - if self.line_number >= self.total_lines: - return '' - line = self.lines[self.line_number] - self.line_number += 1 - if self.indent_char is None and line[:1] in defaults.WHITESPACE: - self.indent_char = line[0] - return line - - def read_lines(self): - # type: () -> List[str] - """Read the lines for this file checker.""" - if self.filename is None or self.filename == '-': - self.filename = 'stdin' - return self.read_lines_from_stdin() - return self.read_lines_from_filename() - - def _readlines_py2(self): - # type: () -> List[str] - with open(self.filename, 'rU') as fd: - return fd.readlines() - - def _readlines_py3(self): - # type: () -> List[str] - try: - with open(self.filename, 'rb') as fd: - (coding, lines) = tokenize.detect_encoding(fd.readline) - textfd = io.TextIOWrapper(fd, coding, line_buffering=True) - return ([l.decode(coding) for l in lines] + - textfd.readlines()) - except (LookupError, SyntaxError, UnicodeError): - # If we can't detect the codec with tokenize.detect_encoding, or - # the detected encoding is incorrect, just fallback to latin-1. - with open(self.filename, encoding='latin-1') as fd: - return fd.readlines() - - def read_lines_from_filename(self): - # type: () -> List[str] - """Read the lines for a file.""" - if (2, 6) <= sys.version_info < (3, 0): - readlines = self._readlines_py2 - elif (3, 0) <= sys.version_info < (4, 0): - readlines = self._readlines_py3 - return readlines() - - def read_lines_from_stdin(self): - # type: () -> List[str] - """Read the lines from standard in.""" - return utils.stdin_get_value().splitlines(True) - - def strip_utf_bom(self): - # type: () -> NoneType - """Strip the UTF bom from the lines of the file.""" - if not self.lines: - # If we have nothing to analyze quit early - return - - first_byte = ord(self.lines[0][0]) - if first_byte not in (0xEF, 0xFEFF): - return - - # If the first byte of the file is a UTF-8 BOM, strip it - if first_byte == 0xFEFF: - self.lines[0] = self.lines[0][1:] - elif self.lines[0][:3] == '\xEF\xBB\xBF': - self.lines[0] = self.lines[0][3:] diff --git a/flake8/processor.py b/flake8/processor.py new file mode 100644 index 0000000..e681c03 --- /dev/null +++ b/flake8/processor.py @@ -0,0 +1,300 @@ +"""Module containing our file processor that tokenizes a file for checks.""" +import contextlib +import io +import sys +import tokenize + +from flake8 import defaults +from flake8 import exceptions +from flake8 import utils + +NEWLINE = frozenset([tokenize.NL, tokenize.NEWLINE]) +# Work around Python < 2.6 behaviour, which does not generate NL after +# a comment which is on a line by itself. +COMMENT_WITH_NL = tokenize.generate_tokens(['#\n'].pop).send(None)[1] == '#\n' + +SKIP_TOKENS = frozenset([tokenize.NL, tokenize.NEWLINE, tokenize.INDENT, + tokenize.DEDENT]) + + +class FileProcessor(object): + """Processes a file and holdes state. + + This processes a file by generating tokens, logical and physical lines, + and AST trees. This also provides a way of passing state about the file + to checks expecting that state. Any public attribute on this object can + be requested by a plugin. The known public attributes are: + + - blank_before + - blank_lines + - indect_char + - indent_level + - line_number + - logical_line + - max_line_length + - multiline + - noqa + - previous_indent_level + - previous_logical + - tokens + - total_lines + - verbose + """ + + def __init__(self, filename, options): + """Initialice our file processor. + + :param str filename: + Name of the file to process + """ + self.filename = filename + self.lines = self.read_lines() + self.strip_utf_bom() + self.options = options + + # Defaults for public attributes + #: Number of preceding blank lines + self.blank_before = 0 + #: Number of blank lines + self.blank_lines = 0 + #: Character used for indentation + self.indent_char = None + #: Current level of indentation + self.indent_level = 0 + #: Line number in the file + self.line_number = 0 + #: Current logical line + self.logical_line = '' + #: Maximum line length as configured by the user + self.max_line_length = options.max_line_length + #: Whether the current physical line is multiline + self.multiline = False + #: Whether or not we're observing NoQA + self.noqa = False + #: Previous level of indentation + self.previous_indent_level = 0 + #: Previous logical line + self.previous_logical = '' + #: Current set of tokens + self.tokens = [] + #: Total number of lines in the file + self.total_lines = len(self.lines) + #: Verbosity level of Flake8 + self.verbosity = options.verbosity + + @contextlib.contextmanager + def inside_multiline(self, line_number): + """Context-manager to toggle the multiline attribute.""" + self.line_number = line_number + self.multiline = True + yield + self.multiline = False + + def reset_blank_before(self): + """Reset the blank_before attribute to zero.""" + self.blank_before = 0 + + def delete_first_token(self): + """Delete the first token in the list of tokens.""" + del self.tokens[0] + + def visited_new_blank_line(self): + """Note that we visited a new blank line.""" + self.blank_lines += 1 + + def build_logical_line_tokens(self): + """Build the mapping, comments, and logical line lists.""" + logical = [] + comments = [] + length = 0 + previous_row = previous_column = mapping = None + for token_type, text, start, end, line in self.tokens: + if token_type in SKIP_TOKENS: + continue + if not mapping: + mapping = [(0, start)] + if token_type == tokenize.COMMENT: + comments.append(text) + continue + if token_type == tokenize.STRING: + text = utils.mutate_string(text) + if previous_row: + (start_row, start_column) = start + if previous_row != start_row: + row_index = previous_row - 1 + column_index = previous_column - 1 + previous_text = self.lines[row_index][column_index] + if (previous_text == ',' or + (previous_text not in '{[(' and + text not in '}])')): + text = ' ' + text + elif previous_column != start_column: + text = line[previous_column:start_column] + text + logical.append(text) + length += len(text) + mapping.append((length, end)) + (previous_row, previous_column) = end + return comments, logical, mapping + + def build_logical_line(self): + """Build a logical line from the current tokens list.""" + comments, logical, mapping_list = self.build_logical_line_tokens() + return ''.join(comments), ''.join(logical), mapping_list + + def split_line(self, token): + """Split a physical line's line based on new-lines. + + This also auto-increments the line number for the caller. + """ + for line in token[1].split('\n')[:-1]: + yield line + self.line_number += 1 + + def keyword_arguments_for(self, parameters, arguments=None): + """Generate the keyword arguments for a list of parameters.""" + if arguments is None: + arguments = {} + for param in parameters: + if param not in arguments: + arguments[param] = getattr(self, param) + return arguments + + def check_physical_error(self, error_code, line): + """Update attributes based on error code and line.""" + if error_code == 'E101': + self.indent_char = line[0] + + def generate_tokens(self): + """Tokenize the file and yield the tokens. + + :raises flake8.exceptions.InvalidSyntax: + If a :class:`tokenize.TokenError` is raised while generating + tokens. + """ + try: + for token in tokenize.generate_tokens(self.next_line): + if token[2][0] > self.total_lines: + break + self.tokens.append(token) + yield token + # NOTE(sigmavirus24): pycodestyle was catching both a SyntaxError + # and a tokenize.TokenError. In looking a the source on Python 2 and + # Python 3, the SyntaxError should never arise from generate_tokens. + # If we were using tokenize.tokenize, we would have to catch that. Of + # course, I'm going to be unsurprised to be proven wrong at a later + # date. + except tokenize.TokenError as exc: + raise exceptions.InvalidSyntax(exc.message, exception=exc) + + def next_line(self): + """Get the next line from the list.""" + if self.line_number >= self.total_lines: + return '' + line = self.lines[self.line_number] + self.line_number += 1 + if self.indent_char is None and line[:1] in defaults.WHITESPACE: + self.indent_char = line[0] + return line + + def read_lines(self): + # type: () -> List[str] + """Read the lines for this file checker.""" + if self.filename is None or self.filename == '-': + self.filename = 'stdin' + return self.read_lines_from_stdin() + return self.read_lines_from_filename() + + def _readlines_py2(self): + # type: () -> List[str] + with open(self.filename, 'rU') as fd: + return fd.readlines() + + def _readlines_py3(self): + # type: () -> List[str] + try: + with open(self.filename, 'rb') as fd: + (coding, lines) = tokenize.detect_encoding(fd.readline) + textfd = io.TextIOWrapper(fd, coding, line_buffering=True) + return ([l.decode(coding) for l in lines] + + textfd.readlines()) + except (LookupError, SyntaxError, UnicodeError): + # If we can't detect the codec with tokenize.detect_encoding, or + # the detected encoding is incorrect, just fallback to latin-1. + with open(self.filename, encoding='latin-1') as fd: + return fd.readlines() + + def read_lines_from_filename(self): + # type: () -> List[str] + """Read the lines for a file.""" + if (2, 6) <= sys.version_info < (3, 0): + readlines = self._readlines_py2 + elif (3, 0) <= sys.version_info < (4, 0): + readlines = self._readlines_py3 + return readlines() + + def read_lines_from_stdin(self): + # type: () -> List[str] + """Read the lines from standard in.""" + return utils.stdin_get_value().splitlines(True) + + def strip_utf_bom(self): + # type: () -> NoneType + """Strip the UTF bom from the lines of the file.""" + if not self.lines: + # If we have nothing to analyze quit early + return + + first_byte = ord(self.lines[0][0]) + if first_byte not in (0xEF, 0xFEFF): + return + + # If the first byte of the file is a UTF-8 BOM, strip it + if first_byte == 0xFEFF: + self.lines[0] = self.lines[0][1:] + elif self.lines[0][:3] == '\xEF\xBB\xBF': + self.lines[0] = self.lines[0][3:] + + +def is_eol_token(token): + """Check if the token is an end-of-line token.""" + return token[0] in NEWLINE or token[4][token[3][1]:].lstrip() == '\\\n' + +if COMMENT_WITH_NL: # If on Python 2.6 + def is_eol_token(token, _is_eol_token=is_eol_token): + """Check if the token is an end-of-line token.""" + return (_is_eol_token(token) or + (token[0] == tokenize.COMMENT and token[1] == token[4])) + + +def is_multiline_string(token): + """Check if this is a multiline string.""" + return token[0] == tokenize.STRING and '\n' in token[1] + + +def token_is_newline(token): + """Check if the token type is a newline token type.""" + return token[0] in NEWLINE + + +def token_is_comment(token): + """Check if the token type is a comment.""" + return COMMENT_WITH_NL and token[0] == tokenize.COMMENT + + +def count_parentheses(current_parentheses_count, token_text): + """Count the number of parentheses.""" + if token_text in '([{': + return current_parentheses_count + 1 + elif token_text in '}])': + return current_parentheses_count - 1 + + +def log_token(log, token): + """Log a token to a provided logging object.""" + if token[2][0] == token[3][0]: + pos = '[%s:%s]' % (token[2][1] or '', token[3][1]) + else: + pos = 'l.%s' % token[3][0] + log.debug('l.%s\t%s\t%s\t%r' % + (token[2][0], pos, tokenize.tok_name[token[0]], + token[1])) From 36fb688f97d69d1ba0e1ea7bd6502ec7fe8b3141 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 4 Mar 2016 22:51:22 -0600 Subject: [PATCH 182/364] Refactor processor and file checker some more --- flake8/checker.py | 22 ++++++++++++--- flake8/processor.py | 68 ++++++++++++++++++++++++++++++++++++++++++++- flake8/utils.py | 60 --------------------------------------- 3 files changed, 85 insertions(+), 65 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 69bab17..c058db5 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -223,16 +223,19 @@ class FileChecker(object): LOG.debug('Logical line: "%s"', logical_line.rstrip()) for plugin in self.checks.logical_line_plugins: - result = self.run_check(plugin, logical_line=logical_line) - if result is not None: - column_offset, text = result + results = self.run_check(plugin, logical_line=logical_line) or () + for offset, text in results: + offset = find_offset(offset, mapping) + line_number, column_offset = offset self.report( error_code=None, - line_number=self.processor.line_number, + line_number=line_number, column=column_offset, text=text, ) + self.processor.next_logical_line() + def run_physical_checks(self, physical_line): """Run all checks for a given physical line.""" for plugin in self.checks.physical_line_plugins: @@ -325,3 +328,14 @@ class FileChecker(object): with self.processor.inside_multiline(line_number=line_no): for line in self.processor.split_line(token): self.run_physical_checks(line + '\n') + + +def find_offset(offset, mapping): + """Find the offset tuple for a single offset.""" + if isinstance(offset, tuple): + return offset + + for token_offset, position in mapping: + if offset <= token_offset: + break + return (position[0], position[1] + offset - token_offset) diff --git a/flake8/processor.py b/flake8/processor.py index e681c03..062006d 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -102,6 +102,25 @@ class FileProcessor(object): """Note that we visited a new blank line.""" self.blank_lines += 1 + def update_state(self, mapping): + """Update the indent level based on the logical line mapping.""" + (start_row, start_col) = mapping[0][1] + start_line = self.lines[start_row - 1] + self.indent_level = expand_indent(start_line[:start_col]) + if self.blank_before < self.blank_lines: + self.blank_before = self.blank_lines + + def next_logical_line(self): + """Record the previous logical line. + + This also resets the tokens list and the blank_lines count. + """ + if self.logical_line: + self.previous_indent_level = self.indent_level + self.previous_logical = self.logical_line + self.blank_lines = 0 + self.tokens = [] + def build_logical_line_tokens(self): """Build the mapping, comments, and logical line lists.""" logical = [] @@ -117,7 +136,7 @@ class FileProcessor(object): comments.append(text) continue if token_type == tokenize.STRING: - text = utils.mutate_string(text) + text = mutate_string(text) if previous_row: (start_row, start_column) = start if previous_row != start_row: @@ -298,3 +317,50 @@ def log_token(log, token): log.debug('l.%s\t%s\t%s\t%r' % (token[2][0], pos, tokenize.tok_name[token[0]], token[1])) + + +def expand_indent(line): + r"""Return the amount of indentation. + + Tabs are expanded to the next multiple of 8. + + >>> expand_indent(' ') + 4 + >>> expand_indent('\t') + 8 + >>> expand_indent(' \t') + 8 + >>> expand_indent(' \t') + 16 + """ + if '\t' not in line: + return len(line) - len(line.lstrip()) + result = 0 + for char in line: + if char == '\t': + result = result // 8 * 8 + 8 + elif char == ' ': + result += 1 + else: + break + return result + + +def mutate_string(text): + """Replace contents with 'xxx' to prevent syntax matching. + + >>> mute_string('"abc"') + '"xxx"' + >>> mute_string("'''abc'''") + "'''xxx'''" + >>> mute_string("r'abc'") + "r'xxx'" + """ + # String modifiers (e.g. u or r) + start = text.index(text[-1]) + 1 + end = len(text) - 1 + # Triple quotes + if text[-3:] in ('"""', "'''"): + start += 2 + end -= 2 + return text[:start] + 'x' * (end - start) + text[end:] diff --git a/flake8/utils.py b/flake8/utils.py index fe024d1..7e08e41 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -4,7 +4,6 @@ import inspect import io import os import sys -import tokenize def parse_comma_separated_list(value): @@ -184,62 +183,3 @@ def parameters_for(plugin): parameters.remove('self') return parameters - -NEWLINE = frozenset([tokenize.NL, tokenize.NEWLINE]) -# Work around Python < 2.6 behaviour, which does not generate NL after -# a comment which is on a line by itself. -COMMENT_WITH_NL = tokenize.generate_tokens(['#\n'].pop).send(None)[1] == '#\n' - - -def is_eol_token(token): - """Check if the token is an end-of-line token.""" - return token[0] in NEWLINE or token[4][token[3][1]:].lstrip() == '\\\n' - -if COMMENT_WITH_NL: # If on Python 2.6 - def is_eol_token(token, _is_eol_token=is_eol_token): - """Check if the token is an end-of-line token.""" - return (_is_eol_token(token) or - (token[0] == tokenize.COMMENT and token[1] == token[4])) - - -def is_multiline_string(token): - """Check if this is a multiline string.""" - return token[0] == tokenize.STRING and '\n' in token[1] - - -def token_is_newline(token): - """Check if the token type is a newline token type.""" - return token[0] in NEWLINE - - -def token_is_comment(token): - """Check if the token type is a comment.""" - return COMMENT_WITH_NL and token[0] == tokenize.COMMENT - - -def count_parentheses(current_parentheses_count, token_text): - """Count the number of parentheses.""" - if token_text in '([{': - return current_parentheses_count + 1 - elif token_text in '}])': - return current_parentheses_count - 1 - - -def mutate_string(text): - """Replace contents with 'xxx' to prevent syntax matching. - - >>> mute_string('"abc"') - '"xxx"' - >>> mute_string("'''abc'''") - "'''xxx'''" - >>> mute_string("r'abc'") - "r'xxx'" - """ - # String modifiers (e.g. u or r) - start = text.index(text[-1]) + 1 - end = len(text) - 1 - # Triple quotes - if text[-3:] in ('"""', "'''"): - start += 2 - end -= 2 - return text[:start] + 'x' * (end - start) + text[end:] From da2182151773e5e37faa87ad607308694e6c1997 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 10 Mar 2016 19:00:07 -0600 Subject: [PATCH 183/364] Make flake8 actually work --- flake8/checker.py | 13 +++++++++++-- flake8/main/cli.py | 17 ++++++++++++++++- flake8/processor.py | 4 +++- setup.py | 1 + 4 files changed, 31 insertions(+), 4 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index c058db5..675ad0c 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -137,6 +137,14 @@ class Manager(object): if utils.fnmatch(filename, filename_patterns) ] + def run(self): + """Run checks. + + TODO(sigmavirus24): Get rid of this + """ + for checker in self.checkers: + checker.run_checks() + def is_path_excluded(self, path): # type: (str) -> bool """Check if a path is excluded. @@ -210,6 +218,7 @@ class FileChecker(object): def run_check(self, plugin, **arguments): """Run the check in a single plugin.""" + LOG.debug('Running %r with %r', plugin, arguments) self.processor.keyword_arguments_for(plugin.parameters, arguments) return plugin.execute(**arguments) @@ -263,7 +272,7 @@ class FileChecker(object): for token in file_processor.generate_tokens(): self.check_physical_eol(token) token_type, text = token[0:2] - processor.log_token(token) + processor.log_token(LOG, token) if token_type == tokenize.OP: parens = processor.count_parentheses(parens, text) elif parens == 0: @@ -298,7 +307,7 @@ class FileChecker(object): def handle_newline(self, token_type): """Handle the logic when encountering a newline token.""" if token_type == tokenize.NEWLINE: - self.check_logical() + self.run_logical_checks() self.processor.reset_blank_before() elif len(self.processor.tokens) == 1: # The physical line contains only this token. diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 15eea8e..a2be97f 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -1,5 +1,6 @@ """Command-line implementation of flake8.""" import flake8 +from flake8 import checker from flake8 import defaults from flake8 import style_guide from flake8.options import aggregator @@ -181,10 +182,11 @@ class Application(object): self.check_plugins = None self.listening_plugins = None - self.formatting_plugigns = None + self.formatting_plugins = None self.formatter = None self.listener_trie = None self.guide = None + self.file_checker_manager = None self.options = None self.args = None @@ -242,6 +244,17 @@ class Application(object): self.options, self.listener_trie, self.formatter ) + def make_file_checker_manager(self): + # type: () -> NoneType + """Initialize our FileChecker Manager.""" + if self.file_checker_manager is None: + self.file_checker_manager = checker.Manager( + style_guide=self.guide, + arguments=self.args, + checker_plugins=self.check_plugins, + ) + self.file_checker_manager.make_checkers() + def run(self, argv=None): # type: (Union[NoneType, List[str]]) -> NoneType """Run our application.""" @@ -251,6 +264,8 @@ class Application(object): self.make_formatter() self.make_notifier() self.make_guide() + self.make_file_checker_manager() + self.file_checker_manager.run() def main(argv=None): diff --git a/flake8/processor.py b/flake8/processor.py index 062006d..88514e5 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -80,7 +80,7 @@ class FileProcessor(object): #: Total number of lines in the file self.total_lines = len(self.lines) #: Verbosity level of Flake8 - self.verbosity = options.verbosity + self.verbosity = options.verbose @contextlib.contextmanager def inside_multiline(self, line_number): @@ -302,10 +302,12 @@ def token_is_comment(token): def count_parentheses(current_parentheses_count, token_text): """Count the number of parentheses.""" + current_parentheses_count = current_parentheses_count or 0 if token_text in '([{': return current_parentheses_count + 1 elif token_text in '}])': return current_parentheses_count - 1 + return current_parentheses_count def log_token(log, token): diff --git a/setup.py b/setup.py index ec40eb6..9b66694 100644 --- a/setup.py +++ b/setup.py @@ -71,6 +71,7 @@ setuptools.setup( 'console_scripts': ['flake8 = flake8.main.cli:main'], 'flake8.extension': [ 'F = flake8.plugins.pyflakes:FlakesChecker', + 'pep8.tabs_or_spaces = pep8:tabs_or_spaces', ], 'flake8.report': [ 'default = flake8.formatting.default:Default', From f04d8da485751bcc5fbdfccdfa6cb369019572af Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 10 Mar 2016 19:41:19 -0600 Subject: [PATCH 184/364] Add pep8 checks to plugins list --- setup.py | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/setup.py b/setup.py index 9b66694..f0b9b83 100644 --- a/setup.py +++ b/setup.py @@ -71,7 +71,43 @@ setuptools.setup( 'console_scripts': ['flake8 = flake8.main.cli:main'], 'flake8.extension': [ 'F = flake8.plugins.pyflakes:FlakesChecker', + # PEP-0008 checks provied by PyCQA/pycodestyle 'pep8.tabs_or_spaces = pep8:tabs_or_spaces', + 'pep8.tabs_obsolete = pep8:tabs_obsolete', + 'pep8.trailing_whitespace = pep8:trailing_whitespace', + 'pep8.trailing_blank_lines = pep8:trailing_blank_lines', + 'pep8.maximum_line_length = pep8:maximum_line_length', + 'pep8.blank_lines = pep8:blank_lines', + 'pep8.extraneous_whitespace = pep8:extraneous_whitespace', + ('pep8.whitespace_around_keywords = ' + 'pep8:whitespace_around_keywords'), + 'pep8.missing_whitespace = pep8:missing_whitespace', + 'pep8.indentation = pep8:indentation', + 'pep8.continued_indentation = pep8:continued_indentation', + ('pep8.whitespace_before_parameters = ' + 'pep8:whitespace_before_parameters'), + ('pep8.whitespace_around_operator = ' + 'pep8:whitespace_around_operator'), + ('pep8.missing_whitespace_around_operator = ' + 'pep8:missing_whitespace_around_operator'), + 'pep8.whitespace_around_comma = pep8:whitespace_around_comma', + ('pep8.whitespace_around_named_parameter_equals = ' + 'pep8:whitespace_around_named_parameter_equals'), + 'pep8.whitespace_before_comment = pep8:whitespace_before_comment', + 'pep8.imports_on_separate_lines = pep8:imports_on_separate_lines', + ('pep8.module_imports_on_top_of_file = ' + 'pep8:module_imports_on_top_of_file'), + 'pep8.compound_statements = pep8:compound_statements', + 'pep8.explicit_line_join = pep8:explicit_line_join', + ('pep8.break_around_binary_operator = ' + 'pep8:break_around_binary_operator'), + 'pep8.comparison_to_singleton = pep8:comparison_to_singleton', + 'pep8.comparison_negative = pep8:comparison_negative', + 'pep8.comparison_type = pep8:comparison_type', + 'pep8.python_3000_has_key = pep8:python_3000_has_key', + 'pep8.python_3000_raise_comma = pep8:python_3000_raise_comma', + 'pep8.python_3000_not_equal = pep8:python_3000_not_equal', + 'pep8.python_3000_backticks = pep8:python_3000_backticks', ], 'flake8.report': [ 'default = flake8.formatting.default:Default', From 960f4d6af7cae10fefd5f67f9b0886b4e73021ed Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 10 Mar 2016 19:54:53 -0600 Subject: [PATCH 185/364] Fix missing attributes for pep8 plugins --- flake8/checker.py | 2 ++ flake8/processor.py | 15 ++++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/flake8/checker.py b/flake8/checker.py index 675ad0c..9fcaa3c 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -232,6 +232,7 @@ class FileChecker(object): LOG.debug('Logical line: "%s"', logical_line.rstrip()) for plugin in self.checks.logical_line_plugins: + self.processor.update_checker_state_for(plugin) results = self.run_check(plugin, logical_line=logical_line) or () for offset, text in results: offset = find_offset(offset, mapping) @@ -248,6 +249,7 @@ class FileChecker(object): def run_physical_checks(self, physical_line): """Run all checks for a given physical line.""" for plugin in self.checks.physical_line_plugins: + self.processor.update_checker_state_for(plugin) result = self.run_check(plugin, physical_line=physical_line) if result is not None: column_offset, text = result diff --git a/flake8/processor.py b/flake8/processor.py index 88514e5..52d7ab1 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -57,6 +57,12 @@ class FileProcessor(object): self.blank_before = 0 #: Number of blank lines self.blank_lines = 0 + #: Checker states for each plugin? + self._checker_states = {} + #: Current checker state + self.checker_state = None + #: User provided option for hang closing + self.hang_closing = options.hang_closing #: Character used for indentation self.indent_char = None #: Current level of indentation @@ -80,7 +86,7 @@ class FileProcessor(object): #: Total number of lines in the file self.total_lines = len(self.lines) #: Verbosity level of Flake8 - self.verbosity = options.verbose + self.verbose = options.verbose @contextlib.contextmanager def inside_multiline(self, line_number): @@ -110,6 +116,13 @@ class FileProcessor(object): if self.blank_before < self.blank_lines: self.blank_before = self.blank_lines + def update_checker_state_for(self, plugin): + """Update the checker_state attribute for the plugin.""" + if 'checker_state' in plugin.parameters: + self.checker_state = self._checker_states.setdefault( + plugin.name, {} + ) + def next_logical_line(self): """Record the previous logical line. From d331558868e7262948d169d322f044117eb871fd Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 11 Mar 2016 21:07:40 -0600 Subject: [PATCH 186/364] Send logging output to --output-file --- flake8/main/cli.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index a2be97f..aa4f9a7 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -178,7 +178,8 @@ class Application(object): # Set the verbosity of the program preliminary_opts, _ = self.option_manager.parse_args() - flake8.configure_logging(preliminary_opts.verbose) + flake8.configure_logging(preliminary_opts.verbose, + preliminary_opts.output_file) self.check_plugins = None self.listening_plugins = None From 4c797c9ff7300b39601943e841d98dcb3cb691e3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 11 Mar 2016 21:08:41 -0600 Subject: [PATCH 187/364] Simplify how we check if a file is excluded Our typical usage always passes is_path_excluded which checks the basename and the full path --- flake8/utils.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/utils.py b/flake8/utils.py index 7e08e41..1ceb76f 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -117,7 +117,7 @@ def filenames_from(arg, predicate=None): for root, sub_directories, files in os.walk(arg): for filename in files: joined = os.path.join(root, filename) - if predicate(filename) or predicate(joined): + if predicate(joined): continue yield joined # NOTE(sigmavirus24): os.walk() will skip a directory if you From 2c17b4342f8a6f24d0e034b322cefec4f7f5d5df Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 11 Mar 2016 21:26:21 -0600 Subject: [PATCH 188/364] Better log format --- flake8/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index 416c951..53e56f8 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -37,7 +37,8 @@ _VERBOSITY_TO_LOG_LEVEL = { 2: logging.DEBUG, } -LOG_FORMAT = '[flake8] %(asctime)s %(levelname)s %(message)s' +LOG_FORMAT = ('[%(name)-25s]:%(threadName)s %(relativeCreated)6d ' + '%(levelname)-8s %(message)s') def configure_logging(verbosity, filename=None, logformat=LOG_FORMAT): From daf5c4d80d576b23d4b43feee356b736a216573d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 14 Mar 2016 20:23:03 -0500 Subject: [PATCH 189/364] Add ability to check if a file is ignored inline Check for ``# flake8: noqa`` lines inside a file. --- flake8/checker.py | 3 +++ flake8/processor.py | 16 ++++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/flake8/checker.py b/flake8/checker.py index 9fcaa3c..7422427 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -291,6 +291,9 @@ class FileChecker(object): def run_checks(self): """Run checks against the file.""" + if self.processor.should_ignore_file(): + return + try: self.process_tokens() except exceptions.InvalidSyntax as exc: diff --git a/flake8/processor.py b/flake8/processor.py index 52d7ab1..f2f5b00 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -1,6 +1,7 @@ """Module containing our file processor that tokenizes a file for checks.""" import contextlib import io +import re import sys import tokenize @@ -41,6 +42,8 @@ class FileProcessor(object): - verbose """ + NOQA_FILE = re.compile(r'\s*# flake8[:=]\s*noqa', re.I) + def __init__(self, filename, options): """Initialice our file processor. @@ -269,6 +272,19 @@ class FileProcessor(object): """Read the lines from standard in.""" return utils.stdin_get_value().splitlines(True) + def should_ignore_file(self): + # type: () -> bool + """Check if ``# flake8: noqa`` is in the file to be ignored. + + :returns: + True if a line matches :attr:`FileProcessor.NOQA_FILE`, + otherwise False + :rtype: + bool + """ + ignore_file = self.NOQA_FILE.search + return any(ignore_file(line) for line in self.lines) + def strip_utf_bom(self): # type: () -> NoneType """Strip the UTF bom from the lines of the file.""" From 447a6d4fccf92e20f003473f1a32cf118d1c320d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 14 Mar 2016 20:23:20 -0500 Subject: [PATCH 190/364] Fix indentation causing incorrect logical lines --- flake8/processor.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flake8/processor.py b/flake8/processor.py index f2f5b00..30c0e96 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -163,8 +163,8 @@ class FileProcessor(object): (previous_text not in '{[(' and text not in '}])')): text = ' ' + text - elif previous_column != start_column: - text = line[previous_column:start_column] + text + elif previous_column != start_column: + text = line[previous_column:start_column] + text logical.append(text) length += len(text) mapping.append((length, end)) From e2314c38ed464812e04be5fba3f7e2091cbc5500 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 14 Mar 2016 21:37:32 -0500 Subject: [PATCH 191/364] Add a lower level for extra verbosity --- flake8/__init__.py | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index 53e56f8..40336ad 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -30,14 +30,21 @@ del NullHandler __version__ = '3.0.0a1' +# There is nothing lower than logging.DEBUG (10) in the logging library, +# but we want an extra level to avoid being too verbose when using -vv. +_EXTRA_VERBOSE = 5 +logging.addLevelName(_EXTRA_VERBOSE, 'VERBOSE') + _VERBOSITY_TO_LOG_LEVEL = { # output more than warnings but not debugging info - 1: logging.INFO, - # output debugging information and everything else - 2: logging.DEBUG, + 1: logging.INFO, # INFO is a numerical level of 20 + # output debugging information + 2: logging.DEBUG, # DEBUG is a numerical level of 10 + # output extra verbose debugging information + 3: _EXTRA_VERBOSE, } -LOG_FORMAT = ('[%(name)-25s]:%(threadName)s %(relativeCreated)6d ' +LOG_FORMAT = ('%(name)-25s %(processName)-11s %(relativeCreated)6d ' '%(levelname)-8s %(message)s') @@ -54,8 +61,8 @@ def configure_logging(verbosity, filename=None, logformat=LOG_FORMAT): """ if verbosity <= 0: return - if verbosity > 2: - verbosity = 2 + if verbosity > 3: + verbosity = 3 log_level = _VERBOSITY_TO_LOG_LEVEL[verbosity] From 0d3506b45753a3115ecb14f804464c2cd58af29a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 14 Mar 2016 21:37:50 -0500 Subject: [PATCH 192/364] Log tokens at a lower level than debug --- flake8/processor.py | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/flake8/processor.py b/flake8/processor.py index 30c0e96..9ad7751 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -5,6 +5,7 @@ import re import sys import tokenize +import flake8 from flake8 import defaults from flake8 import exceptions from flake8 import utils @@ -345,9 +346,9 @@ def log_token(log, token): pos = '[%s:%s]' % (token[2][1] or '', token[3][1]) else: pos = 'l.%s' % token[3][0] - log.debug('l.%s\t%s\t%s\t%r' % - (token[2][0], pos, tokenize.tok_name[token[0]], - token[1])) + log.log(flake8._EXTRA_VERBOSE, 'l.%s\t%s\t%s\t%r' % + (token[2][0], pos, tokenize.tok_name[token[0]], + token[1])) def expand_indent(line): From d222fcb9e118bec3b051355d12da0f6aa9cce163 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 14 Mar 2016 21:38:08 -0500 Subject: [PATCH 193/364] Correct log levels around loading plugins --- flake8/plugins/manager.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index c453dbe..872a4f4 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -97,7 +97,7 @@ class Plugin(object): Nothing """ if self._plugin is None: - LOG.debug('Loading plugin "%s" from entry-point.', self.name) + LOG.info('Loading plugin "%s" from entry-point.', self.name) try: self._load(verify_requirements) except Exception as load_exception: @@ -162,12 +162,12 @@ class PluginManager(object): # pylint: disable=too-few-public-methods self._load_all_plugins() def _load_all_plugins(self): - LOG.debug('Loading entry-points for "%s".', self.namespace) + LOG.info('Loading entry-points for "%s".', self.namespace) for entry_point in pkg_resources.iter_entry_points(self.namespace): name = entry_point.name self.plugins[name] = Plugin(name, entry_point) self.names.append(name) - LOG.info('Loaded %r for plugin "%s".', self.plugins[name], name) + LOG.debug('Loaded %r for plugin "%s".', self.plugins[name], name) def map(self, func, *args, **kwargs): r"""Call ``func`` with the plugin and \*args and \**kwargs after. From 07b9ffbeb9a9e68ce9376f56595ece772ee628cf Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 14 Mar 2016 21:38:56 -0500 Subject: [PATCH 194/364] Add naive multiprocessing support --- flake8/checker.py | 56 ++++++++++++++++++++++++++++++++++++---------- flake8/main/cli.py | 3 ++- 2 files changed, 46 insertions(+), 13 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 7422427..0398187 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -116,12 +116,27 @@ class Manager(object): def start(self): """Start checking files.""" - pass - # for i in range(self.jobs or 0): - # proc = multiprocessing.Process(target=self.process_files) - # proc.daemon = True - # proc.start() - # self.processes.append(proc) + LOG.info('Making checkers') + self.make_checkers() + if not self.using_multiprocessing: + return + + LOG.info('Populating process queue') + for checker in self.checkers: + self.process_queue.put(checker) + + def stop(self): + """Stop checking files.""" + if not self.using_multiprocessing: + return + + LOG.info('Notifying process workers of completion') + for i in range(self.jobs or 0): + self.process_queue.put('DONE') + + LOG.info('Joining process workers') + for process in self.processes: + process.join() def make_checkers(self, paths=None): # type: (List[str]) -> NoneType @@ -137,14 +152,31 @@ class Manager(object): if utils.fnmatch(filename, filename_patterns) ] - def run(self): - """Run checks. - - TODO(sigmavirus24): Get rid of this - """ - for checker in self.checkers: + def _run_checks_from_queue(self): + LOG.info('Running checks in parallel') + for checker in iter(self.process_queue.get, 'DONE'): + LOG.debug('Running checker for file "%s"', checker.filename) checker.run_checks() + def run(self): + """Run all the checkers. + + This handles starting the process workers or just simply running all + of the checks in serial. + """ + if self.using_multiprocessing: + LOG.info('Starting process workers') + for i in range(self.jobs or 0): + proc = multiprocessing.Process( + target=self._run_checks_from_queue + ) + proc.daemon = True + proc.start() + self.processes.append(proc) + else: + for checker in self.checkers: + checker.run_checks() + def is_path_excluded(self, path): # type: (str) -> bool """Check if a path is excluded. diff --git a/flake8/main/cli.py b/flake8/main/cli.py index aa4f9a7..0dfd19a 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -254,7 +254,6 @@ class Application(object): arguments=self.args, checker_plugins=self.check_plugins, ) - self.file_checker_manager.make_checkers() def run(self, argv=None): # type: (Union[NoneType, List[str]]) -> NoneType @@ -266,7 +265,9 @@ class Application(object): self.make_notifier() self.make_guide() self.make_file_checker_manager() + self.file_checker_manager.start() self.file_checker_manager.run() + self.file_checker_manager.stop() def main(argv=None): From 3b830366b6c96ec65071eb316d69dee463d91390 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 08:40:43 -0500 Subject: [PATCH 195/364] Add a results queue --- flake8/checker.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/flake8/checker.py b/flake8/checker.py index 0398187..274dc23 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -55,6 +55,7 @@ class Manager(object): self.checks = checker_plugins self.jobs = self._job_count() self.process_queue = None + self.results_queue = None self.using_multiprocessing = False self.processes = [] self.checkers = [] @@ -62,6 +63,7 @@ class Manager(object): if self.jobs is not None and self.jobs > 1: self.using_multiprocessing = True self.process_queue = multiprocessing.Queue() + self.results_queue = multiprocessing.Queue() def _job_count(self): # type: () -> Union[int, NoneType] From 189faf68bac204625d6e0b3f3d3ff74ba924dc84 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 08:42:13 -0500 Subject: [PATCH 196/364] Reorganize methods alphabetically Add methods to report errors to the style guide A single file works fine now but not a directory --- flake8/checker.py | 157 +++++++++++++++++++++++++++++----------------- 1 file changed, 100 insertions(+), 57 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 274dc23..7cd6d53 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -116,68 +116,38 @@ class Manager(object): # it to an integer return int(jobs) - def start(self): - """Start checking files.""" - LOG.info('Making checkers') - self.make_checkers() - if not self.using_multiprocessing: - return + def _report_after_parallel(self): + style_guide = self.style_guide + for (filename, results) in iter(self.results_queue.get, 'DONE'): + results = sorted(results, key=lambda tup: (tup[2], tup[3])) + for (error_code, line_number, column, text) in results: + style_guide.handle_error( + code=error_code, + filename=filename, + line_number=line_number, + column_number=column, + text=text + ) - LOG.info('Populating process queue') + def _report_after_serial(self): + style_guide = self.style_guide for checker in self.checkers: - self.process_queue.put(checker) - - def stop(self): - """Stop checking files.""" - if not self.using_multiprocessing: - return - - LOG.info('Notifying process workers of completion') - for i in range(self.jobs or 0): - self.process_queue.put('DONE') - - LOG.info('Joining process workers') - for process in self.processes: - process.join() - - def make_checkers(self, paths=None): - # type: (List[str]) -> NoneType - """Create checkers for each file.""" - if paths is None: - paths = self.arguments - filename_patterns = self.options.filename - self.checkers = [ - FileChecker(filename, self.checks, self.style_guide) - for argument in paths - for filename in utils.filenames_from(argument, - self.is_path_excluded) - if utils.fnmatch(filename, filename_patterns) - ] + results = sorted(checker.results, key=lambda tup: (tup[2], tup[3])) + filename = checker.filename + for (error_code, line_number, column, text) in results: + style_guide.handle_error( + code=error_code, + filename=filename, + line_number=line_number, + column_number=column, + text=text + ) def _run_checks_from_queue(self): LOG.info('Running checks in parallel') for checker in iter(self.process_queue.get, 'DONE'): LOG.debug('Running checker for file "%s"', checker.filename) - checker.run_checks() - - def run(self): - """Run all the checkers. - - This handles starting the process workers or just simply running all - of the checks in serial. - """ - if self.using_multiprocessing: - LOG.info('Starting process workers') - for i in range(self.jobs or 0): - proc = multiprocessing.Process( - target=self._run_checks_from_queue - ) - proc.daemon = True - proc.start() - self.processes.append(proc) - else: - for checker in self.checkers: - checker.run_checks() + checker.run_checks(self.results_queue) def is_path_excluded(self, path): # type: (str) -> bool @@ -205,6 +175,76 @@ class Manager(object): '' if match else 'not ') return match + def make_checkers(self, paths=None): + # type: (List[str]) -> NoneType + """Create checkers for each file.""" + if paths is None: + paths = self.arguments + filename_patterns = self.options.filename + self.checkers = [ + FileChecker(filename, self.checks, self.style_guide) + for argument in paths + for filename in utils.filenames_from(argument, + self.is_path_excluded) + if utils.fnmatch(filename, filename_patterns) + ] + + def report(self): + """Report all of the errors found in the managed file checkers. + + This iterates over each of the checkers and reports the errors sorted + by line number. + """ + if self.using_multiprocessing: + self._report_after_parallel() + else: + self._report_after_serial() + + def run(self): + """Run all the checkers. + + This handles starting the process workers or just simply running all + of the checks in serial. + """ + if self.using_multiprocessing: + LOG.info('Starting process workers') + for i in range(self.jobs or 0): + proc = multiprocessing.Process( + target=self._run_checks_from_queue + ) + proc.daemon = True + proc.start() + self.processes.append(proc) + else: + for checker in self.checkers: + checker.run_checks() + + def start(self): + """Start checking files.""" + LOG.info('Making checkers') + self.make_checkers() + if not self.using_multiprocessing: + return + + LOG.info('Populating process queue') + for checker in self.checkers: + self.process_queue.put(checker) + + def stop(self): + """Stop checking files.""" + if not self.using_multiprocessing: + return + + LOG.info('Notifying process workers of completion') + for i in range(self.jobs or 0): + self.process_queue.put('DONE') + + LOG.info('Joining process workers') + for process in self.processes: + process.join() + LOG.info('Processes joined') + self.results_queue.put('DONE') + class FileChecker(object): """Manage running checks for a file and aggregate the results.""" @@ -246,7 +286,7 @@ class FileChecker(object): """Report an error by storing it in the results list.""" if error_code is None: error_code, text = text.split(' ', 1) - error = (error_code, self.filename, line_number, column, text) + error = (error_code, line_number, column, text) self.results.append(error) return error_code @@ -323,7 +363,7 @@ class FileChecker(object): self.run_physical_checks(file_processor.lines[-1]) self.run_logical_checks() - def run_checks(self): + def run_checks(self, results_queue): """Run checks against the file.""" if self.processor.should_ignore_file(): return @@ -334,6 +374,9 @@ class FileChecker(object): self.report(exc.error_code, exc.line_number, exc.column_number, exc.error_message) + if results_queue is not None: + results_queue.put_nowait((self.filename, self.results)) + def handle_comment(self, token, token_text): """Handle the logic when encountering a comment token.""" # The comment also ends a physical line From 6eb3dee1df547b15a257ba1a666cd04df6928298 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 08:42:30 -0500 Subject: [PATCH 197/364] Reorganize our Application flow --- flake8/main/cli.py | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 0dfd19a..06299ec 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -255,6 +255,18 @@ class Application(object): checker_plugins=self.check_plugins, ) + def run_checks(self): + # type: () -> NoneType + """Run the actual checks with the FileChecker Manager.""" + self.file_checker_manager.start() + self.file_checker_manager.run() + self.file_checker_manager.stop() + + def report_errors(self): + # type: () -> NoneType + """Report all the errors found by flake8 3.0.""" + self.file_checker_manager.report() + def run(self, argv=None): # type: (Union[NoneType, List[str]]) -> NoneType """Run our application.""" @@ -265,9 +277,8 @@ class Application(object): self.make_notifier() self.make_guide() self.make_file_checker_manager() - self.file_checker_manager.start() - self.file_checker_manager.run() - self.file_checker_manager.stop() + self.run_checks() + self.report_errors() def main(argv=None): From 67f9e04335d1f30d8e77c50a96d57b887f9805f3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 11:50:27 -0500 Subject: [PATCH 198/364] Only ever return an integer for job count --- flake8/checker.py | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 7cd6d53..33bbf79 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -60,7 +60,7 @@ class Manager(object): self.processes = [] self.checkers = [] - if self.jobs is not None and self.jobs > 1: + if self.jobs > 1: self.using_multiprocessing = True self.process_queue = multiprocessing.Queue() self.results_queue = multiprocessing.Queue() @@ -79,28 +79,28 @@ class Manager(object): if not multiprocessing: LOG.warning('The multiprocessing module is not available. ' 'Ignoring --jobs arguments.') - return None + return 0 if utils.is_windows(): LOG.warning('The --jobs option is not available on Windows. ' 'Ignoring --jobs arguments.') - return None + return 0 if utils.is_using_stdin(self.arguments): LOG.warning('The --jobs option is not compatible with supplying ' 'input using - . Ignoring --jobs arguments.') - return None + return 0 if self.options.diff: LOG.warning('The --diff option was specified with --jobs but ' 'they are not compatible. Ignoring --jobs arguments.') - return None + return 0 jobs = self.options.jobs if jobs != 'auto' and not jobs.isdigit(): LOG.warning('"%s" is not a valid parameter to --jobs. Must be one ' 'of "auto" or a numerical value, e.g., 4.', jobs) - return None + return 0 # If the value is "auto", we want to let the multiprocessing library # decide the number based on the number of CPUs. However, if that @@ -207,8 +207,8 @@ class Manager(object): of the checks in serial. """ if self.using_multiprocessing: - LOG.info('Starting process workers') - for i in range(self.jobs or 0): + LOG.info('Starting %d process workers', self.jobs) + for i in range(self.jobs): proc = multiprocessing.Process( target=self._run_checks_from_queue ) @@ -232,11 +232,7 @@ class Manager(object): def stop(self): """Stop checking files.""" - if not self.using_multiprocessing: - return - - LOG.info('Notifying process workers of completion') - for i in range(self.jobs or 0): + for i in range(self.jobs): self.process_queue.put('DONE') LOG.info('Joining process workers') From c0659d1a8ce9c4eb6a29a34f4bbf2b0ec2d2f83a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 11:53:12 -0500 Subject: [PATCH 199/364] Catch Keyboard interruptions in our application Add logging to our main application --- flake8/main/cli.py | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 06299ec..ecbdde9 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -1,4 +1,6 @@ """Command-line implementation of flake8.""" +import logging + import flake8 from flake8 import checker from flake8 import defaults @@ -7,6 +9,8 @@ from flake8.options import aggregator from flake8.options import manager from flake8.plugins import manager as plugin_manager +LOG = logging.getLogger(__name__) + def register_default_options(option_manager): """Register the default options on our OptionManager.""" @@ -267,9 +271,7 @@ class Application(object): """Report all the errors found by flake8 3.0.""" self.file_checker_manager.report() - def run(self, argv=None): - # type: (Union[NoneType, List[str]]) -> NoneType - """Run our application.""" + def _run(self, argv): self.find_plugins() self.register_plugin_options() self.parse_configuration_and_cli(argv) @@ -280,6 +282,16 @@ class Application(object): self.run_checks() self.report_errors() + def run(self, argv=None): + # type: (Union[NoneType, List[str]]) -> NoneType + """Run our application.""" + try: + self._run(argv) + except KeyboardInterrupt as exc: + LOG.critical('Caught keyboard interrupt from user') + LOG.exception(exc) + self.file_checker_manager._force_cleanup() + def main(argv=None): # type: (Union[NoneType, List[str]]) -> NoneType From 19062c5e9c948fd7816561b8d6dd2f5c8b60a842 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 12:18:45 -0500 Subject: [PATCH 200/364] Wrap up multiprocessing work --- flake8/checker.py | 55 ++++++++++++++++++++++++++++++++++++---------- flake8/main/cli.py | 2 ++ 2 files changed, 45 insertions(+), 12 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 33bbf79..0d6a386 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -65,6 +65,18 @@ class Manager(object): self.process_queue = multiprocessing.Queue() self.results_queue = multiprocessing.Queue() + @staticmethod + def _cleanup_queue(q): + while not q.empty(): + q.get_nowait() + + def _force_cleanup(self): + if self.using_multiprocessing: + for proc in self.processes: + proc.join(0.2) + self._cleanup_queue(self.process_queue) + self._cleanup_queue(self.results_queue) + def _job_count(self): # type: () -> Union[int, NoneType] # First we walk through all of our error cases: @@ -116,10 +128,29 @@ class Manager(object): # it to an integer return int(jobs) + def _results(self): + seen_done = 0 + while True: + LOG.info('Retrieving results') + result = self.results_queue.get() + if result == 'DONE': + seen_done += 1 + if seen_done >= self.jobs: + break + continue + + yield result + def _report_after_parallel(self): style_guide = self.style_guide - for (filename, results) in iter(self.results_queue.get, 'DONE'): - results = sorted(results, key=lambda tup: (tup[2], tup[3])) + final_results = {} + for (filename, results) in self._results(): + final_results[filename] = results + + for checker in self.checkers: + filename = checker.filename + results = sorted(final_results[filename], + key=lambda tup: (tup[1], tup[2])) for (error_code, line_number, column, text) in results: style_guide.handle_error( code=error_code, @@ -148,6 +179,7 @@ class Manager(object): for checker in iter(self.process_queue.get, 'DONE'): LOG.debug('Running checker for file "%s"', checker.filename) checker.run_checks(self.results_queue) + self.results_queue.put('DONE') def is_path_excluded(self, path): # type: (str) -> bool @@ -195,9 +227,7 @@ class Manager(object): This iterates over each of the checkers and reports the errors sorted by line number. """ - if self.using_multiprocessing: - self._report_after_parallel() - else: + if not self.using_multiprocessing: self._report_after_serial() def run(self): @@ -215,9 +245,13 @@ class Manager(object): proc.daemon = True proc.start() self.processes.append(proc) + proc = multiprocessing.Process(target=self._report_after_parallel) + proc.start() + LOG.info('Started process to report errors') + self.processes.append(proc) else: for checker in self.checkers: - checker.run_checks() + checker.run_checks(self.results_queue) def start(self): """Start checking files.""" @@ -235,11 +269,8 @@ class Manager(object): for i in range(self.jobs): self.process_queue.put('DONE') - LOG.info('Joining process workers') - for process in self.processes: - process.join() - LOG.info('Processes joined') - self.results_queue.put('DONE') + for proc in self.processes: + proc.join() class FileChecker(object): @@ -371,7 +402,7 @@ class FileChecker(object): exc.error_message) if results_queue is not None: - results_queue.put_nowait((self.filename, self.results)) + results_queue.put((self.filename, self.results)) def handle_comment(self, token, token_text): """Handle the logic when encountering a comment token.""" diff --git a/flake8/main/cli.py b/flake8/main/cli.py index ecbdde9..5bf6144 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -264,11 +264,13 @@ class Application(object): """Run the actual checks with the FileChecker Manager.""" self.file_checker_manager.start() self.file_checker_manager.run() + LOG.info('Finished running') self.file_checker_manager.stop() def report_errors(self): # type: () -> NoneType """Report all the errors found by flake8 3.0.""" + LOG.info('Reporting errors') self.file_checker_manager.report() def _run(self, argv): From 666e1c2b06151b7f6f2d290d82aa38b13bafb033 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 12:27:05 -0500 Subject: [PATCH 201/364] Ensure the logical line is tracked on the processor --- flake8/processor.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/flake8/processor.py b/flake8/processor.py index 9ad7751..d75256b 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -175,7 +175,8 @@ class FileProcessor(object): def build_logical_line(self): """Build a logical line from the current tokens list.""" comments, logical, mapping_list = self.build_logical_line_tokens() - return ''.join(comments), ''.join(logical), mapping_list + self.logical_line = ''.join(logical) + return ''.join(comments), self.logical_line, mapping_list def split_line(self, token): """Split a physical line's line based on new-lines. From 576d1f6c85b321f4a9897a386af8815a7ca779d4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 14:10:05 -0500 Subject: [PATCH 202/364] Bypass linecache by recording physical line --- flake8/checker.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 0d6a386..c449a9b 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -151,13 +151,14 @@ class Manager(object): filename = checker.filename results = sorted(final_results[filename], key=lambda tup: (tup[1], tup[2])) - for (error_code, line_number, column, text) in results: + for (error_code, line_number, column, text, line) in results: style_guide.handle_error( code=error_code, filename=filename, line_number=line_number, column_number=column, - text=text + text=text, + physical_line=line, ) def _report_after_serial(self): @@ -165,13 +166,14 @@ class Manager(object): for checker in self.checkers: results = sorted(checker.results, key=lambda tup: (tup[2], tup[3])) filename = checker.filename - for (error_code, line_number, column, text) in results: + for (error_code, line_number, column, text, line) in results: style_guide.handle_error( code=error_code, filename=filename, line_number=line_number, column_number=column, - text=text + text=text, + physical_line=line, ) def _run_checks_from_queue(self): @@ -313,7 +315,8 @@ class FileChecker(object): """Report an error by storing it in the results list.""" if error_code is None: error_code, text = text.split(' ', 1) - error = (error_code, line_number, column, text) + physical_line = self.processor.line_for(line_number) + error = (error_code, line_number, column, text, physical_line) self.results.append(error) return error_code From 0b063a10249558ede919bcd7f67c6aa563ba74ab Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 14:10:20 -0500 Subject: [PATCH 203/364] Run checks expecting an AST --- flake8/checker.py | 33 +++++++++++++++++++++++++++++++++ flake8/main/cli.py | 3 +++ flake8/processor.py | 9 +++++++++ flake8/style_guide.py | 5 +++-- 4 files changed, 48 insertions(+), 2 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index c449a9b..905cade 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -326,6 +326,37 @@ class FileChecker(object): self.processor.keyword_arguments_for(plugin.parameters, arguments) return plugin.execute(**arguments) + def run_ast_checks(self): + """Run all checks expecting an abstract syntax tree.""" + try: + ast = self.processor.build_ast() + except (ValueError, SyntaxError, TypeError): + (exc_type, exception) = sys.exc_info()[:2] + if len(exception.args) > 1: + offset = exception.args[1] + if len(offset) > 2: + offset = offset[1:3] + else: + offset = (1, 0) + + self.report('E999', offset[0], offset[1], '%s: %s' % + (exc_type.__name__, exception.args[0])) + return + + for plugin in self.checks.ast_plugins: + checker = self.run_check(plugin, tree=ast) + # NOTE(sigmavirus24): If we want to allow for AST plugins that are + # not classes exclusively, we can do the following: + # retrieve_results = getattr(checker, 'run', lambda: checker) + # Otherwise, we just call run on the checker + for (line_number, offset, text, check) in checker.run(): + self.report( + error_code=None, + line_number=line_number, + column=offset, + text=text, + ) + def run_logical_checks(self): """Run all checks expecting a logical line.""" comments, logical_line, mapping = self.processor.build_logical_line() @@ -404,6 +435,8 @@ class FileChecker(object): self.report(exc.error_code, exc.line_number, exc.column_number, exc.error_message) + self.run_ast_checks() + if results_queue is not None: results_queue.put((self.filename, self.results)) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 5bf6144..dbd9603 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -227,6 +227,9 @@ class Application(object): self.option_manager, argv ) + self.check_plugins.provide_options(self.option_manager, self.options, + self.args) + def make_formatter(self): # type: () -> NoneType """Initialize a formatter based on the parsed options.""" diff --git a/flake8/processor.py b/flake8/processor.py index d75256b..1dc27a1 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -10,6 +10,7 @@ from flake8 import defaults from flake8 import exceptions from flake8 import utils +PyCF_ONLY_AST = 1024 NEWLINE = frozenset([tokenize.NL, tokenize.NEWLINE]) # Work around Python < 2.6 behaviour, which does not generate NL after # a comment which is on a line by itself. @@ -172,6 +173,10 @@ class FileProcessor(object): (previous_row, previous_column) = end return comments, logical, mapping + def build_ast(self): + """Build an abstract syntax tree from the list of lines.""" + return compile(''.join(self.lines), '', 'exec', PyCF_ONLY_AST) + def build_logical_line(self): """Build a logical line from the current tokens list.""" comments, logical, mapping_list = self.build_logical_line_tokens() @@ -223,6 +228,10 @@ class FileProcessor(object): except tokenize.TokenError as exc: raise exceptions.InvalidSyntax(exc.message, exception=exc) + def line_for(self, line_number): + """Retrieve the physical line at the specified line number.""" + return self.lines[line_number - 1] + def next_line(self): """Get the next line from the list.""" if self.line_number >= self.total_lines: diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 51c3a8e..2c18c9d 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -187,12 +187,13 @@ class StyleGuide(object): error, codes_str) return False - def handle_error(self, code, filename, line_number, column_number, text): + def handle_error(self, code, filename, line_number, column_number, text, + physical_line=None): # type: (str, str, int, int, str) -> NoneType """Handle an error reported by a check.""" error = Error(code, filename, line_number, column_number, text) if (self.should_report_error(error.code) is Decision.Selected and - self.is_inline_ignored(error) is False): + self.is_inline_ignored(error, physical_line) is False): self.formatter.handle(error) self.listener.notify(error.code, error) From ee18ac981e3dbaf3365a817663834c7b547f83cb Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 14:39:43 -0500 Subject: [PATCH 204/364] Handle case where file was ignored --- flake8/checker.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/checker.py b/flake8/checker.py index 905cade..becde97 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -149,7 +149,7 @@ class Manager(object): for checker in self.checkers: filename = checker.filename - results = sorted(final_results[filename], + results = sorted(final_results.get(filename, []), key=lambda tup: (tup[1], tup[2])) for (error_code, line_number, column, text, line) in results: style_guide.handle_error( From 784a70dd0ed74bef5ca2dac636705dea82687a47 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 15 Mar 2016 15:58:24 -0500 Subject: [PATCH 205/364] Move flake8 2 out of the way --- {flake8 => old/flake8}/__init__.py | 0 {flake8 => old/flake8}/__main__.py | 0 {flake8 => old/flake8}/_pyflakes.py | 0 {flake8 => old/flake8}/callbacks.py | 0 {flake8 => old/flake8}/compat.py | 0 {flake8 => old/flake8}/engine.py | 0 {flake8 => old/flake8}/hooks.py | 0 {flake8 => old/flake8}/main.py | 0 {flake8 => old/flake8}/reporter.py | 0 {flake8 => old/flake8}/run.py | 0 {flake8 => old/flake8}/tests/__init__.py | 0 {flake8 => old/flake8}/tests/_test_warnings.py | 0 {flake8 => old/flake8}/tests/test_engine.py | 0 {flake8 => old/flake8}/tests/test_hooks.py | 0 {flake8 => old/flake8}/tests/test_integration.py | 0 {flake8 => old/flake8}/tests/test_main.py | 0 {flake8 => old/flake8}/tests/test_pyflakes.py | 0 {flake8 => old/flake8}/tests/test_reporter.py | 0 {flake8 => old/flake8}/tests/test_util.py | 0 {flake8 => old/flake8}/util.py | 0 setup.py => old/setup.py | 0 tox.ini => old/tox.ini | 0 22 files changed, 0 insertions(+), 0 deletions(-) rename {flake8 => old/flake8}/__init__.py (100%) rename {flake8 => old/flake8}/__main__.py (100%) rename {flake8 => old/flake8}/_pyflakes.py (100%) rename {flake8 => old/flake8}/callbacks.py (100%) rename {flake8 => old/flake8}/compat.py (100%) rename {flake8 => old/flake8}/engine.py (100%) rename {flake8 => old/flake8}/hooks.py (100%) rename {flake8 => old/flake8}/main.py (100%) rename {flake8 => old/flake8}/reporter.py (100%) rename {flake8 => old/flake8}/run.py (100%) rename {flake8 => old/flake8}/tests/__init__.py (100%) rename {flake8 => old/flake8}/tests/_test_warnings.py (100%) rename {flake8 => old/flake8}/tests/test_engine.py (100%) rename {flake8 => old/flake8}/tests/test_hooks.py (100%) rename {flake8 => old/flake8}/tests/test_integration.py (100%) rename {flake8 => old/flake8}/tests/test_main.py (100%) rename {flake8 => old/flake8}/tests/test_pyflakes.py (100%) rename {flake8 => old/flake8}/tests/test_reporter.py (100%) rename {flake8 => old/flake8}/tests/test_util.py (100%) rename {flake8 => old/flake8}/util.py (100%) rename setup.py => old/setup.py (100%) rename tox.ini => old/tox.ini (100%) diff --git a/flake8/__init__.py b/old/flake8/__init__.py similarity index 100% rename from flake8/__init__.py rename to old/flake8/__init__.py diff --git a/flake8/__main__.py b/old/flake8/__main__.py similarity index 100% rename from flake8/__main__.py rename to old/flake8/__main__.py diff --git a/flake8/_pyflakes.py b/old/flake8/_pyflakes.py similarity index 100% rename from flake8/_pyflakes.py rename to old/flake8/_pyflakes.py diff --git a/flake8/callbacks.py b/old/flake8/callbacks.py similarity index 100% rename from flake8/callbacks.py rename to old/flake8/callbacks.py diff --git a/flake8/compat.py b/old/flake8/compat.py similarity index 100% rename from flake8/compat.py rename to old/flake8/compat.py diff --git a/flake8/engine.py b/old/flake8/engine.py similarity index 100% rename from flake8/engine.py rename to old/flake8/engine.py diff --git a/flake8/hooks.py b/old/flake8/hooks.py similarity index 100% rename from flake8/hooks.py rename to old/flake8/hooks.py diff --git a/flake8/main.py b/old/flake8/main.py similarity index 100% rename from flake8/main.py rename to old/flake8/main.py diff --git a/flake8/reporter.py b/old/flake8/reporter.py similarity index 100% rename from flake8/reporter.py rename to old/flake8/reporter.py diff --git a/flake8/run.py b/old/flake8/run.py similarity index 100% rename from flake8/run.py rename to old/flake8/run.py diff --git a/flake8/tests/__init__.py b/old/flake8/tests/__init__.py similarity index 100% rename from flake8/tests/__init__.py rename to old/flake8/tests/__init__.py diff --git a/flake8/tests/_test_warnings.py b/old/flake8/tests/_test_warnings.py similarity index 100% rename from flake8/tests/_test_warnings.py rename to old/flake8/tests/_test_warnings.py diff --git a/flake8/tests/test_engine.py b/old/flake8/tests/test_engine.py similarity index 100% rename from flake8/tests/test_engine.py rename to old/flake8/tests/test_engine.py diff --git a/flake8/tests/test_hooks.py b/old/flake8/tests/test_hooks.py similarity index 100% rename from flake8/tests/test_hooks.py rename to old/flake8/tests/test_hooks.py diff --git a/flake8/tests/test_integration.py b/old/flake8/tests/test_integration.py similarity index 100% rename from flake8/tests/test_integration.py rename to old/flake8/tests/test_integration.py diff --git a/flake8/tests/test_main.py b/old/flake8/tests/test_main.py similarity index 100% rename from flake8/tests/test_main.py rename to old/flake8/tests/test_main.py diff --git a/flake8/tests/test_pyflakes.py b/old/flake8/tests/test_pyflakes.py similarity index 100% rename from flake8/tests/test_pyflakes.py rename to old/flake8/tests/test_pyflakes.py diff --git a/flake8/tests/test_reporter.py b/old/flake8/tests/test_reporter.py similarity index 100% rename from flake8/tests/test_reporter.py rename to old/flake8/tests/test_reporter.py diff --git a/flake8/tests/test_util.py b/old/flake8/tests/test_util.py similarity index 100% rename from flake8/tests/test_util.py rename to old/flake8/tests/test_util.py diff --git a/flake8/util.py b/old/flake8/util.py similarity index 100% rename from flake8/util.py rename to old/flake8/util.py diff --git a/setup.py b/old/setup.py similarity index 100% rename from setup.py rename to old/setup.py diff --git a/tox.ini b/old/tox.ini similarity index 100% rename from tox.ini rename to old/tox.ini From 09000762fb07463eb8ff8a1ace214c7605363879 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 09:15:52 -0500 Subject: [PATCH 206/364] Rename test file to mirror actual class it is testing --- tests/unit/{test_file_checker.py => test_file_processor.py} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename tests/unit/{test_file_checker.py => test_file_processor.py} (100%) diff --git a/tests/unit/test_file_checker.py b/tests/unit/test_file_processor.py similarity index 100% rename from tests/unit/test_file_checker.py rename to tests/unit/test_file_processor.py From fbbb3bece571a10d9a52f387bca271de48930480 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 09:16:22 -0500 Subject: [PATCH 207/364] Ignore directories we do not want to run tests from --- setup.cfg | 3 +++ 1 file changed, 3 insertions(+) diff --git a/setup.cfg b/setup.cfg index e96761b..abde58d 100644 --- a/setup.cfg +++ b/setup.cfg @@ -7,3 +7,6 @@ universal=1 [metadata] requires-dist = enum34; python_version<"3.4" + +[pytest] +norecursedirs = .git .* *.egg* old docs dist build From 3da9fd3cb734f5d5a528036e3eb4f8e5e14fe63c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 14:52:20 -0500 Subject: [PATCH 208/364] Remove outdated git to hg conversion script --- bin/git-patch-to-hg-export.py | 99 ----------------------------------- 1 file changed, 99 deletions(-) delete mode 100755 bin/git-patch-to-hg-export.py diff --git a/bin/git-patch-to-hg-export.py b/bin/git-patch-to-hg-export.py deleted file mode 100755 index 118e94e..0000000 --- a/bin/git-patch-to-hg-export.py +++ /dev/null @@ -1,99 +0,0 @@ -#!/usr/bin/env python -"""\ -Git patch to HG changeset patch converter. - -USAGE: git-patch-to-hg-export.py < git.patch > hg.patch -""" - -from email.utils import parsedate_tz, mktime_tz -import re -import os -import sys - - -def git_patch_to_hg(fin, fout): - fout.write('# HG changeset patch\n') - - subject_re = re.compile(r'^(RE:)?\s*(\[[^]]*\])?\s*', re.I) - - # headers - for line in fin: - if line.startswith('From: '): - fout.write('# User %s' % line[6:]) - elif line.startswith('Date: '): - t = parsedate_tz(line[6:]) - timestamp = mktime_tz(t) - timezone = -t[-1] - fout.write('# Date %d %d\n' % (timestamp, timezone)) - elif line.startswith('Subject: '): - subject = subject_re.sub('', line[9:]) - fout.write(subject + '\n') - elif line == '\n' or line == '\r\n': - break - - # commit message - for line in fin: - if line == '---\n': - break - fout.write(line) - - # skip over the diffstat - for line in fin: - if line.startswith('diff --git'): - fout.write('\n' + line) - break - - # diff - # NOTE: there will still be an index line after each diff --git, but it - # will be ignored - for line in fin: - fout.write(line.encode('utf-8')) - - # NOTE: the --/version will still be at the end, but it will be ignored - - -def open_file(): - if len(sys.argv) > 1: - if re.match('https?://', sys.argv[1]): - import requests - import io - resp = requests.get(sys.argv[1]) - if resp.ok: - return io.StringIO(resp.content.decode('utf-8')) - else: - return io.StringIO('') - elif os.path.exists(sys.argv[1]): - return open(sys.argv[1]) - return sys.stdin - - -if __name__ == "__main__": - patch_fd = open_file() - git_patch_to_hg(patch_fd, sys.stdout) - - -__author__ = "Mark Lodato " - -__license__ = """ -This is the MIT license: http://www.opensource.org/licenses/mit-license.php - -Copyright (c) 2009 Mark Lodato - -Permission is hereby granted, free of charge, to any person obtaining a copy -of this software and associated documentation files (the "Software"), to -deal in the Software without restriction, including without limitation the -rights to use, copy, modify, merge, publish, distribute, sublicense, and/or -sell copies of the Software, and to permit persons to whom the Software is -furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in -all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING -FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS -IN THE SOFTWARE. -""" From 48b995fa62004e0595b4de9edb93f3ea2677a397 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 14:55:57 -0500 Subject: [PATCH 209/364] Remove unnecessary and outdated test runner --- flake8/processor.py | 6 ++++-- run_tests.py | 32 ------------------------------- tests/unit/test_file_processor.py | 29 +++++++++++++++++----------- 3 files changed, 22 insertions(+), 45 deletions(-) delete mode 100755 run_tests.py diff --git a/flake8/processor.py b/flake8/processor.py index 1dc27a1..bcb33cc 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -46,14 +46,16 @@ class FileProcessor(object): NOQA_FILE = re.compile(r'\s*# flake8[:=]\s*noqa', re.I) - def __init__(self, filename, options): + def __init__(self, filename, options, lines=None): """Initialice our file processor. :param str filename: Name of the file to process """ self.filename = filename - self.lines = self.read_lines() + self.lines = lines + if lines is None: + self.lines = self.read_lines() self.strip_utf_bom() self.options = options diff --git a/run_tests.py b/run_tests.py deleted file mode 100755 index 09d6884..0000000 --- a/run_tests.py +++ /dev/null @@ -1,32 +0,0 @@ -#!/usr/bin/env python - -import unittest -import os -import re -import sys -sys.path.insert(0, '.') - -TEST_DIR = 'flake8.tests' - - -def collect_tests(): - # list files in directory tests/ - names = os.listdir(TEST_DIR.replace('.', '/')) - regex = re.compile("(?!_+)\w+\.py$") - join = '.'.join - # Make a list of the names like 'tests.test_name' - names = [join([TEST_DIR, f[:-3]]) for f in names if regex.match(f)] - modules = [__import__(name, fromlist=[TEST_DIR]) for name in names] - load_tests = unittest.defaultTestLoader.loadTestsFromModule - suites = [load_tests(m) for m in modules] - suite = suites.pop() - for s in suites: - suite.addTests(s) - return suite - -if __name__ == "__main__": - suite = collect_tests() - res = unittest.TextTestRunner(verbosity=1).run(suite) - - # If it was successful, we don't want to exit with code 1 - raise SystemExit(not res.wasSuccessful()) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index e26bc83..2970697 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -1,26 +1,33 @@ -"""Tests for the FileChecker class.""" -from flake8 import checker +"""Tests for the FileProcessor class.""" +import optparse + +from flake8 import processor import pytest +def options_from(**kwargs): + kwargs.setdefault('hang_closing', True) + kwargs.setdefault('max_line_length', 79) + kwargs.setdefault('verbose', False) + return optparse.Values(kwargs) + + def test_read_lines_splits_lines(): """Verify that read_lines splits the lines of the file.""" - file_checker = checker.FileChecker(__file__, []) - lines = file_checker.read_lines() + file_processor = processor.FileProcessor(__file__, options_from()) + lines = file_processor.lines assert len(lines) > 5 - assert '"""Tests for the FileChecker class."""\n' in lines + assert '"""Tests for the FileProcessor class."""\n' in lines @pytest.mark.parametrize('first_line', [ '\xEF\xBB\xBF"""Module docstring."""\n', - '\uFEFF"""Module docstring."""\n', + u'\uFEFF"""Module docstring."""\n', ]) def test_strip_utf_bom(first_line): r"""Verify that we strip '\xEF\xBB\xBF' from the first line.""" lines = [first_line] - file_checker = checker.FileChecker('stdin', []) - file_checker.lines = lines[:] - file_checker.strip_utf_bom() - assert file_checker.lines != lines - assert file_checker.lines[0] == '"""Module docstring."""\n' + file_processor = processor.FileProcessor('-', options_from(), lines[:]) + assert file_processor.lines != lines + assert file_processor.lines[0] == '"""Module docstring."""\n' From 0f2a2614eee2f50f66b097d29206c230092c0b05 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 14:56:54 -0500 Subject: [PATCH 210/364] Switch out dev-requirements for tox --- dev-requirements.txt | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/dev-requirements.txt b/dev-requirements.txt index 1908775..053148f 100644 --- a/dev-requirements.txt +++ b/dev-requirements.txt @@ -1,5 +1 @@ -pep8 -pyflakes -mccabe -mock -nose +tox From 55cc996368469bf9c3cfec5789d9c9675bd4e664 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 15:10:35 -0500 Subject: [PATCH 211/364] Move old documentation out of the way --- {docs => old/docs}/Makefile | 0 {docs => old/docs}/api.rst | 0 {docs => old/docs}/buildout.rst | 0 {docs => old/docs}/changes.rst | 0 {docs => old/docs}/conf.py | 0 {docs => old/docs}/config.rst | 0 {docs => old/docs}/extensions.rst | 0 {docs => old/docs}/index.rst | 0 {docs => old/docs}/setuptools.rst | 0 {docs => old/docs}/vcs.rst | 0 {docs => old/docs}/warnings.rst | 0 11 files changed, 0 insertions(+), 0 deletions(-) rename {docs => old/docs}/Makefile (100%) rename {docs => old/docs}/api.rst (100%) rename {docs => old/docs}/buildout.rst (100%) rename {docs => old/docs}/changes.rst (100%) rename {docs => old/docs}/conf.py (100%) rename {docs => old/docs}/config.rst (100%) rename {docs => old/docs}/extensions.rst (100%) rename {docs => old/docs}/index.rst (100%) rename {docs => old/docs}/setuptools.rst (100%) rename {docs => old/docs}/vcs.rst (100%) rename {docs => old/docs}/warnings.rst (100%) diff --git a/docs/Makefile b/old/docs/Makefile similarity index 100% rename from docs/Makefile rename to old/docs/Makefile diff --git a/docs/api.rst b/old/docs/api.rst similarity index 100% rename from docs/api.rst rename to old/docs/api.rst diff --git a/docs/buildout.rst b/old/docs/buildout.rst similarity index 100% rename from docs/buildout.rst rename to old/docs/buildout.rst diff --git a/docs/changes.rst b/old/docs/changes.rst similarity index 100% rename from docs/changes.rst rename to old/docs/changes.rst diff --git a/docs/conf.py b/old/docs/conf.py similarity index 100% rename from docs/conf.py rename to old/docs/conf.py diff --git a/docs/config.rst b/old/docs/config.rst similarity index 100% rename from docs/config.rst rename to old/docs/config.rst diff --git a/docs/extensions.rst b/old/docs/extensions.rst similarity index 100% rename from docs/extensions.rst rename to old/docs/extensions.rst diff --git a/docs/index.rst b/old/docs/index.rst similarity index 100% rename from docs/index.rst rename to old/docs/index.rst diff --git a/docs/setuptools.rst b/old/docs/setuptools.rst similarity index 100% rename from docs/setuptools.rst rename to old/docs/setuptools.rst diff --git a/docs/vcs.rst b/old/docs/vcs.rst similarity index 100% rename from docs/vcs.rst rename to old/docs/vcs.rst diff --git a/docs/warnings.rst b/old/docs/warnings.rst similarity index 100% rename from docs/warnings.rst rename to old/docs/warnings.rst From e2b8fbd3a7da1fdacb3dfc686149f226b3e6387d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 15:10:57 -0500 Subject: [PATCH 212/364] Exclude more directories --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 946d532..cd56e86 100644 --- a/tox.ini +++ b/tox.ini @@ -106,5 +106,5 @@ ignore = D203 # NOTE(sigmavirus24): Once we release 3.0.0 this exclude option can be specified # across multiple lines. Presently it cannot be specified across multiple lines. # :-( -exclude = .git,__pycache__,docs/source/conf.py +exclude = .git,__pycache__,docs/source/conf.py,old,build,dist max-complexity = 10 From 5cde4bebaeec732d21a568a81dfa791804cb1b35 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 15:16:00 -0500 Subject: [PATCH 213/364] Add docstring for our test helper --- tests/unit/test_file_processor.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index 2970697..30a5e30 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -7,6 +7,7 @@ import pytest def options_from(**kwargs): + """Generate a Values instances with our kwargs.""" kwargs.setdefault('hang_closing', True) kwargs.setdefault('max_line_length', 79) kwargs.setdefault('verbose', False) From 0fd276d5eedfc6cdeb40570934037eb9bbb06fbd Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 17 Mar 2016 15:16:15 -0500 Subject: [PATCH 214/364] Add a test for FileProcessor.should_ignore_file --- tests/unit/test_file_processor.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index 30a5e30..6e72a12 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -32,3 +32,17 @@ def test_strip_utf_bom(first_line): file_processor = processor.FileProcessor('-', options_from(), lines[:]) assert file_processor.lines != lines assert file_processor.lines[0] == '"""Module docstring."""\n' + + +@pytest.mark.parametrize('lines, expected', [ + (['\xEF\xBB\xBF"""Module docstring."""\n'], False), + ([u'\uFEFF"""Module docstring."""\n'], False), + (['#!/usr/bin/python', '# flake8 is great', 'a = 1'], False), + (['#!/usr/bin/python', '# flake8: noqa', 'a = 1'], True), + (['# flake8: noqa', '#!/usr/bin/python', 'a = 1'], True), + (['#!/usr/bin/python', 'a = 1', '# flake8: noqa'], True), +]) +def test_should_ignore_file(lines, expected): + """Verify that we ignore a file if told to.""" + file_processor = processor.FileProcessor('-', options_from(), lines) + assert file_processor.should_ignore_file() is expected From a1c1247cba53e8e3160f00b9e923db5b36b3d131 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 18 Mar 2016 13:55:37 -0500 Subject: [PATCH 215/364] Remove 2.6 from testing matrix Explicitly list which versions of Python 2 and 3 we support --- .travis.yml | 2 -- setup.py | 3 +++ 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 78a0db1..46fffdb 100644 --- a/.travis.yml +++ b/.travis.yml @@ -11,8 +11,6 @@ notifications: matrix: include: - - python: 2.6 - env: TOXENV=py26 - python: 2.7 env: TOXENV=py27 - python: 3.3 diff --git a/setup.py b/setup.py index f0b9b83..c090263 100644 --- a/setup.py +++ b/setup.py @@ -120,7 +120,10 @@ setuptools.setup( "License :: OSI Approved :: MIT License", "Programming Language :: Python", "Programming Language :: Python :: 2", + "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.4", + "Programming Language :: Python :: 3.5", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: Software Development :: Quality Assurance", ], From a4cc9d6fa89deec98c586dfb3c8bf9da1d8c9d6e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 20 Mar 2016 14:16:59 -0500 Subject: [PATCH 216/364] Add test for reading from stdin with the file processor --- tests/unit/test_file_processor.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index 6e72a12..c09f203 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -3,6 +3,7 @@ import optparse from flake8 import processor +import mock import pytest @@ -46,3 +47,14 @@ def test_should_ignore_file(lines, expected): """Verify that we ignore a file if told to.""" file_processor = processor.FileProcessor('-', options_from(), lines) assert file_processor.should_ignore_file() is expected + + +@mock.patch('flake8.utils.stdin_get_value') +def test_read_lines_from_stdin(stdin_get_value): + """Verify that we use our own utility function to retrieve stdin.""" + stdin_value = mock.Mock() + stdin_value.splitlines.return_value = [] + stdin_get_value.return_value = stdin_value + file_processor = processor.FileProcessor('-', options_from()) + stdin_get_value.assert_called_once_with() + stdin_value.splitlines.assert_called_once_with(True) From 5554ef78cfea5f11b8aeb2862781c93db9fdf918 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 26 Mar 2016 08:51:21 -0500 Subject: [PATCH 217/364] Silence pytest warnings --- tests/unit/test_plugin_type_manager.py | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index 271ebc0..fb17a37 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -43,7 +43,7 @@ def create_manager_with_plugins(plugins): return manager_mock -class TestType(manager.PluginTypeManager): +class FakeTestType(manager.PluginTypeManager): """Fake PluginTypeManager.""" namespace = TEST_NAMESPACE @@ -52,7 +52,7 @@ class TestType(manager.PluginTypeManager): @mock.patch('flake8.plugins.manager.PluginManager') def test_instantiates_a_manager(PluginManager): """Verify we create a PluginManager on instantiation.""" - TestType() + FakeTestType() PluginManager.assert_called_once_with(TEST_NAMESPACE) @@ -63,7 +63,7 @@ def test_proxies_names_to_manager(PluginManager): PluginManager.return_value = mock.Mock(names=[ 'T100', 'T200', 'T300' ]) - type_mgr = TestType() + type_mgr = FakeTestType() assert type_mgr.names == ['T100', 'T200', 'T300'] @@ -74,7 +74,7 @@ def test_proxies_plugins_to_manager(PluginManager): PluginManager.return_value = mock.Mock(plugins=[ 'T100', 'T200', 'T300' ]) - type_mgr = TestType() + type_mgr = FakeTestType() assert type_mgr.plugins == ['T100', 'T200', 'T300'] @@ -102,7 +102,7 @@ def test_load_plugins(PluginManager): # Return our PluginManager mock PluginManager.return_value = create_mapping_manager_mock(plugins) - type_mgr = TestType() + type_mgr = FakeTestType() # Load the tests (do what we're actually testing) assert len(type_mgr.load_plugins()) == 8 # Assert that our closure does what we think it does @@ -121,7 +121,7 @@ def test_load_plugins_fails(PluginManager): # Return our PluginManager mock PluginManager.return_value = create_mapping_manager_mock(plugins) - type_mgr = TestType() + type_mgr = FakeTestType() with pytest.raises(exceptions.FailedToLoadPlugin): type_mgr.load_plugins() @@ -146,7 +146,7 @@ def test_register_options(PluginManager): PluginManager.return_value = create_mapping_manager_mock(plugins) optmanager = object() - type_mgr = TestType() + type_mgr = FakeTestType() type_mgr.register_options(optmanager) for plugin in plugins: @@ -166,7 +166,7 @@ def test_provide_options(PluginManager): options = object() extra_args = [] - type_mgr = TestType() + type_mgr = FakeTestType() type_mgr.provide_options(optmanager, options, extra_args) for plugin in plugins: @@ -182,7 +182,7 @@ def test_proxy_contains_to_managers_plugins_dict(PluginManager): # Return our PluginManager mock PluginManager.return_value = create_manager_with_plugins(plugins) - type_mgr = TestType() + type_mgr = FakeTestType() for i in range(8): key = 'T10%i' % i assert key in type_mgr @@ -195,7 +195,7 @@ def test_proxies_getitem_to_managers_plugins_dictionary(PluginManager): # Return our PluginManager mock PluginManager.return_value = create_manager_with_plugins(plugins) - type_mgr = TestType() + type_mgr = FakeTestType() for i in range(8): key = 'T10%i' % i assert type_mgr[key] is plugins[key] From da172ec641642f3c741b3360095de9e4588acb9a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 26 Mar 2016 08:51:32 -0500 Subject: [PATCH 218/364] Print pytest warnings immediately --- setup.cfg | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.cfg b/setup.cfg index abde58d..8af25a3 100644 --- a/setup.cfg +++ b/setup.cfg @@ -10,3 +10,4 @@ requires-dist = [pytest] norecursedirs = .git .* *.egg* old docs dist build +addopts = -rw From c11d33d917bc940fd6baeaaae9c1d11d26f48af3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 26 Mar 2016 08:51:50 -0500 Subject: [PATCH 219/364] Log the attribute error for developers --- flake8/processor.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/flake8/processor.py b/flake8/processor.py index bcb33cc..bf96c7a 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -1,6 +1,7 @@ """Module containing our file processor that tokenizes a file for checks.""" import contextlib import io +import logging import re import sys import tokenize @@ -10,6 +11,7 @@ from flake8 import defaults from flake8 import exceptions from flake8 import utils +LOG = logging.getLogger(__name__) PyCF_ONLY_AST = 1024 NEWLINE = frozenset([tokenize.NL, tokenize.NEWLINE]) # Work around Python < 2.6 behaviour, which does not generate NL after @@ -199,8 +201,13 @@ class FileProcessor(object): if arguments is None: arguments = {} for param in parameters: - if param not in arguments: + if param in arguments: + continue + try: arguments[param] = getattr(self, param) + except AttributeError as exc: + LOG.exception(exc) + raise return arguments def check_physical_error(self, error_code, line): From 82dd4f6d25a5ea78b4742633881f6a2553ccd1f8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 26 Mar 2016 08:51:58 -0500 Subject: [PATCH 220/364] Add more FileProcessor tests --- tests/unit/test_file_processor.py | 80 +++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index c09f203..32da2cc 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -58,3 +58,83 @@ def test_read_lines_from_stdin(stdin_get_value): file_processor = processor.FileProcessor('-', options_from()) stdin_get_value.assert_called_once_with() stdin_value.splitlines.assert_called_once_with(True) + + +@mock.patch('flake8.utils.stdin_get_value') +def test_read_lines_sets_filename_attribute(stdin_get_value): + """Verify that we update the filename attribute.""" + stdin_value = mock.Mock() + stdin_value.splitlines.return_value = [] + stdin_get_value.return_value = stdin_value + file_processor = processor.FileProcessor('-', options_from()) + assert file_processor.filename == 'stdin' + + +def test_line_for(): + """Verify we grab the correct line from the cached lines.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'Line 1', + 'Line 2', + 'Line 3', + ]) + + for i in range(1, 4): + assert file_processor.line_for(i) == 'Line {0}'.format(i) + + +def test_next_line(): + """Verify we update the file_processor state for each new line.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'Line 1', + 'Line 2', + 'Line 3', + ]) + + for i in range(1, 4): + assert file_processor.next_line() == 'Line {}'.format(i) + assert file_processor.line_number == i + + +@pytest.mark.parametrize('error_code, line, expected_indent_char', [ + ('E101', '\t\ta = 1', '\t'), + ('E101', ' a = 1', ' '), + ('W101', 'frobulate()', None), + ('F821', 'class FizBuz:', None), +]) +def test_check_physical_error(error_code, line, expected_indent_char): + """Verify we update the indet char for the appropriate error code.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'Line 1', + ]) + + file_processor.check_physical_error(error_code, line) + assert file_processor.indent_char == expected_indent_char + + +@pytest.mark.parametrize('params, args, expected_kwargs', [ + (['blank_before', 'blank_lines'], None, {'blank_before': 0, + 'blank_lines': 0}), + (['noqa', 'fake'], {'fake': 'foo'}, {'noqa': False, 'fake': 'foo'}), + (['blank_before', 'blank_lines', 'noqa'], + {'blank_before': 10, 'blank_lines': 5, 'noqa': True}, + {'blank_before': 10, 'blank_lines': 5, 'noqa': True}), + ([], {'fake': 'foo'}, {'fake': 'foo'}), +]) +def test_keyword_arguments_for(params, args, expected_kwargs): + """Verify the keyword args are generated properly.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'Line 1', + ]) + kwargs_for = file_processor.keyword_arguments_for + + assert kwargs_for(params, args) == expected_kwargs + + +def test_keyword_arguments_for_does_not_handle_attribute_errors(): + """Verify we re-raise AttributeErrors.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'Line 1', + ]) + + with pytest.raises(AttributeError): + file_processor.keyword_arguments_for(['fake']) From 698af805143475d2fb340e97c106ae9b2112afbe Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 27 Mar 2016 14:32:21 -0500 Subject: [PATCH 221/364] Add test for FileProcessor#split_line --- tests/unit/test_file_processor.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index 32da2cc..bfa3e55 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -138,3 +138,21 @@ def test_keyword_arguments_for_does_not_handle_attribute_errors(): with pytest.raises(AttributeError): file_processor.keyword_arguments_for(['fake']) + + +@pytest.mark.parametrize('unsplit_line, expected_lines', [ + ('line', []), + ('line 1\n', ['line 1']), + ('line 1\nline 2\n', ['line 1', 'line 2']), + ('line 1\n\nline 2\n', ['line 1', '', 'line 2']), +]) +def test_split_line(unsplit_line, expected_lines): + """Verify the token line spliting.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'Line 1', + ]) + + actual_lines = list(file_processor.split_line((1, unsplit_line))) + assert expected_lines == actual_lines + + assert len(actual_lines) == file_processor.line_number From 7de8a058dd79cf2be65530402089aec8cca61b2c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 28 Mar 2016 19:43:14 -0500 Subject: [PATCH 222/364] Add unit test for build_ast --- tests/unit/test_file_processor.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index bfa3e55..fba104a 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -1,4 +1,5 @@ """Tests for the FileProcessor class.""" +import ast import optparse from flake8 import processor @@ -156,3 +157,13 @@ def test_split_line(unsplit_line, expected_lines): assert expected_lines == actual_lines assert len(actual_lines) == file_processor.line_number + + +def test_build_ast(): + """Verify the logic for how we build an AST for plugins.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'a = 1\n' + ]) + + module = file_processor.build_ast() + assert isinstance(module, ast.Module) From 4ea161ff9c2160c4eea5d819f96c449c0edb0c10 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 28 Mar 2016 21:04:34 -0500 Subject: [PATCH 223/364] Add a few more unit tests around FileProcessor --- tests/unit/test_file_processor.py | 42 +++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index fba104a..027e74b 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -167,3 +167,45 @@ def test_build_ast(): module = file_processor.build_ast() assert isinstance(module, ast.Module) + + +def test_next_logical_line_updates_the_previous_logical_line(): + """Verify that we update our tracking of the previous logical line.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'a = 1\n' + ]) + + file_processor.indent_level = 1 + file_processor.logical_line = 'a = 1' + assert file_processor.previous_logical == '' + assert file_processor.previous_indent_level is 0 + + file_processor.next_logical_line() + assert file_processor.previous_logical == 'a = 1' + assert file_processor.previous_indent_level == 1 + + +def test_visited_new_blank_line(): + """Verify we update the number of blank lines seen.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'a = 1\n' + ]) + + assert file_processor.blank_lines == 0 + file_processor.visited_new_blank_line() + assert file_processor.blank_lines == 1 + + +def test_inside_multiline(): + """Verify we update the line number and reset multiline.""" + file_processor = processor.FileProcessor('-', options_from(), lines=[ + 'a = 1\n' + ]) + + assert file_processor.multiline is False + assert file_processor.line_number == 0 + with file_processor.inside_multiline(10): + assert file_processor.multiline is True + assert file_processor.line_number == 10 + + assert file_processor.multiline is False From 425f89eee9a84da78a4a113bbdbd12214e6459d9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 29 Mar 2016 08:07:15 -0500 Subject: [PATCH 224/364] Add more tests around the processor module --- flake8/processor.py | 11 +++++++-- tests/unit/test_file_processor.py | 40 +++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+), 2 deletions(-) diff --git a/flake8/processor.py b/flake8/processor.py index bf96c7a..e71784c 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -370,6 +370,8 @@ def log_token(log, token): token[1])) +# NOTE(sigmavirus24): This was taken wholesale from +# https://github.com/PyCQA/pycodestyle def expand_indent(line): r"""Return the amount of indentation. @@ -397,6 +399,9 @@ def expand_indent(line): return result +# NOTE(sigmavirus24): This was taken wholesale from +# https://github.com/PyCQA/pycodestyle. The in-line comments were edited to be +# more descriptive. def mutate_string(text): """Replace contents with 'xxx' to prevent syntax matching. @@ -407,10 +412,12 @@ def mutate_string(text): >>> mute_string("r'abc'") "r'xxx'" """ - # String modifiers (e.g. u or r) + # NOTE(sigmavirus24): If there are string modifiers (e.g., b, u, r) + # use the last "character" to determine if we're using single or double + # quotes and then find the first instance of it start = text.index(text[-1]) + 1 end = len(text) - 1 - # Triple quotes + # Check for triple-quoted strings if text[-3:] in ('"""', "'''"): start += 2 end -= 2 diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index 027e74b..50c5fee 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -209,3 +209,43 @@ def test_inside_multiline(): assert file_processor.line_number == 10 assert file_processor.multiline is False + + +@pytest.mark.parametrize('string, expected', [ + ('""', '""'), + ("''", "''"), + ('"a"', '"x"'), + ("'a'", "'x'"), + ('"x"', '"x"'), + ("'x'", "'x'"), + ('"abcdef"', '"xxxxxx"'), + ("'abcdef'", "'xxxxxx'"), + ('""""""', '""""""'), + ("''''''", "''''''"), + ('"""a"""', '"""x"""'), + ("'''a'''", "'''x'''"), + ('"""x"""', '"""x"""'), + ("'''x'''", "'''x'''"), + ('"""abcdef"""', '"""xxxxxx"""'), + ("'''abcdef'''", "'''xxxxxx'''"), + ('"""xxxxxx"""', '"""xxxxxx"""'), + ("'''xxxxxx'''", "'''xxxxxx'''"), +]) +def test_mutate_string(string, expected): + """Verify we appropriately mutate the string to sanitize it.""" + actual = processor.mutate_string(string) + assert expected == actual + + +@pytest.mark.parametrize('string, expected', [ + (' ', 4), + (' ', 6), + ('\t', 8), + ('\t\t', 16), + (' \t', 8), + (' \t', 16), +]) +def test_expand_indent(string, expected): + """Verify we correctly measure the amount of indentation.""" + actual = processor.expand_indent(string) + assert expected == actual From 81eb3e41cce40e00e2aa5f4653366cf96898e8db Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 2 Apr 2016 11:13:23 -0500 Subject: [PATCH 225/364] Add more processor tests --- tests/unit/test_file_processor.py | 42 +++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/tests/unit/test_file_processor.py b/tests/unit/test_file_processor.py index 50c5fee..db17c2e 100644 --- a/tests/unit/test_file_processor.py +++ b/tests/unit/test_file_processor.py @@ -1,6 +1,7 @@ """Tests for the FileProcessor class.""" import ast import optparse +import tokenize from flake8 import processor @@ -249,3 +250,44 @@ def test_expand_indent(string, expected): """Verify we correctly measure the amount of indentation.""" actual = processor.expand_indent(string) assert expected == actual + + +@pytest.mark.parametrize('token, log_string', [ + [(tokenize.COMMENT, '# this is a comment', + (1, 0), # (start_row, start_column) + (1, 19), # (end_ro, end_column) + '# this is a comment',), + "l.1\t[:19]\tCOMMENT\t'# this is a comment'"], + [(tokenize.COMMENT, '# this is a comment', + (1, 5), # (start_row, start_column) + (1, 19), # (end_ro, end_column) + '# this is a comment',), + "l.1\t[5:19]\tCOMMENT\t'# this is a comment'"], + [(tokenize.COMMENT, '# this is a comment', + (1, 0), # (start_row, start_column) + (2, 19), # (end_ro, end_column) + '# this is a comment',), + "l.1\tl.2\tCOMMENT\t'# this is a comment'"], +]) +def test_log_token(token, log_string): + """Verify we use the log object passed in.""" + LOG = mock.Mock() + processor.log_token(LOG, token) + LOG.log.assert_called_once_with( + 5, # flake8._EXTRA_VERBOSE + log_string, + ) + + +@pytest.mark.parametrize('current_count, token_text, expected', [ + (None, '(', 1), + (None, '[', 1), + (None, '{', 1), + (1, ')', 0), + (1, ']', 0), + (1, '}', 0), + (10, '+', 10), +]) +def test_count_parentheses(current_count, token_text, expected): + """Verify our arithmetic is correct.""" + assert processor.count_parentheses(current_count, token_text) == expected From 09ad1d850eaa3736a4b642cba6211ac3292f7cc6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 2 Apr 2016 11:30:12 -0500 Subject: [PATCH 226/364] Restructure Checker Manager for serial retries --- flake8/checker.py | 78 +++++++++++++++++++++++++++++++++++------------ 1 file changed, 58 insertions(+), 20 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index becde97..8a0603f 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -1,4 +1,5 @@ """Checker Manager and Checker classes.""" +import errno import logging import os import sys @@ -15,6 +16,21 @@ from flake8 import utils LOG = logging.getLogger(__name__) +SERIAL_RETRY_ERRNOS = set([ + # ENOSPC: Added by sigmavirus24 + # > On some operating systems (OSX), multiprocessing may cause an + # > ENOSPC error while trying to trying to create a Semaphore. + # > In those cases, we should replace the customized Queue Report + # > class with pep8's StandardReport class to ensure users don't run + # > into this problem. + # > (See also: https://gitlab.com/pycqa/flake8/issues/74) + errno.ENOSPC, + # NOTE(sigmavirus24): When adding to this list, include the reasoning + # on the lines before the error code and always append your error + # code. Further, please always add a trailing `,` to reduce the visual + # noise in diffs. +]) + class Manager(object): """Manage the parallelism and checker instances for each plugin and file. @@ -62,8 +78,6 @@ class Manager(object): if self.jobs > 1: self.using_multiprocessing = True - self.process_queue = multiprocessing.Queue() - self.results_queue = multiprocessing.Queue() @staticmethod def _cleanup_queue(q): @@ -232,28 +246,52 @@ class Manager(object): if not self.using_multiprocessing: self._report_after_serial() + def run_parallel(self): + """Run the checkers in parallel.""" + # NOTE(sigmavirus24): Initialize Queues here to handle serial retries + # in one place. + self.process_queue = multiprocessing.Queue() + self.results_queue = multiprocessing.Queue() + + LOG.info('Starting %d process workers', self.jobs - 1) + for i in range(self.jobs - 1): + proc = multiprocessing.Process( + target=self._run_checks_from_queue + ) + proc.daemon = True + proc.start() + self.processes.append(proc) + proc = multiprocessing.Process(target=self._report_after_parallel) + proc.start() + LOG.info('Started process to report errors') + self.processes.append(proc) + + def run_serial(self): + """Run the checkers in serial.""" + for checker in self.checkers: + checker.run_checks(self.results_queue) + def run(self): """Run all the checkers. - This handles starting the process workers or just simply running all - of the checks in serial. + This will intelligently decide whether to run the checks in parallel + or whether to run them in serial. + + If running the checks in parallel causes a problem (e.g., + https://gitlab.com/pycqa/flake8/issues/74) this also implements + fallback to serial processing. """ - if self.using_multiprocessing: - LOG.info('Starting %d process workers', self.jobs) - for i in range(self.jobs): - proc = multiprocessing.Process( - target=self._run_checks_from_queue - ) - proc.daemon = True - proc.start() - self.processes.append(proc) - proc = multiprocessing.Process(target=self._report_after_parallel) - proc.start() - LOG.info('Started process to report errors') - self.processes.append(proc) - else: - for checker in self.checkers: - checker.run_checks(self.results_queue) + try: + if self.using_multiprocessing: + self.run_parallel() + else: + self.run_serial() + except OSError as oserr: + if oserr.errno not in SERIAL_RETRY_ERRNOS: + LOG.exception(oserr) + raise + LOG.warning('Running in serial after OS exception, %r', oserr) + self.run_serial() def start(self): """Start checking files.""" From e3707bbe08a21c05b00cee94741e5273808c4859 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 2 Apr 2016 14:02:43 -0500 Subject: [PATCH 227/364] Fix multiprocessing work with retries --- flake8/checker.py | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 8a0603f..17c01cb 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -72,12 +72,17 @@ class Manager(object): self.jobs = self._job_count() self.process_queue = None self.results_queue = None - self.using_multiprocessing = False + self.using_multiprocessing = self.jobs > 1 self.processes = [] self.checkers = [] - if self.jobs > 1: - self.using_multiprocessing = True + try: + self.process_queue = multiprocessing.Queue() + self.results_queue = multiprocessing.Queue() + except OSError as oserr: + if oserr.errno not in SERIAL_RETRY_ERRNOS: + raise + self.using_multiprocessing = False @staticmethod def _cleanup_queue(q): @@ -144,8 +149,8 @@ class Manager(object): def _results(self): seen_done = 0 + LOG.info('Retrieving results') while True: - LOG.info('Retrieving results') result = self.results_queue.get() if result == 'DONE': seen_done += 1 @@ -248,19 +253,15 @@ class Manager(object): def run_parallel(self): """Run the checkers in parallel.""" - # NOTE(sigmavirus24): Initialize Queues here to handle serial retries - # in one place. - self.process_queue = multiprocessing.Queue() - self.results_queue = multiprocessing.Queue() - LOG.info('Starting %d process workers', self.jobs - 1) - for i in range(self.jobs - 1): + for i in range(self.jobs): proc = multiprocessing.Process( target=self._run_checks_from_queue ) proc.daemon = True proc.start() self.processes.append(proc) + proc = multiprocessing.Process(target=self._report_after_parallel) proc.start() LOG.info('Started process to report errors') @@ -310,6 +311,7 @@ class Manager(object): self.process_queue.put('DONE') for proc in self.processes: + LOG.info('Joining %s to the main process', proc.name) proc.join() From eb3d8f57912527d1229b6960509e8e1e7b8d02da Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 3 Apr 2016 14:41:12 -0500 Subject: [PATCH 228/364] Add unit tests around serial retries --- tests/unit/test_checker_manager.py | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 tests/unit/test_checker_manager.py diff --git a/tests/unit/test_checker_manager.py b/tests/unit/test_checker_manager.py new file mode 100644 index 0000000..2d3143e --- /dev/null +++ b/tests/unit/test_checker_manager.py @@ -0,0 +1,27 @@ +"""Tests for the Manager object for FileCheckers.""" +import errno +import mock + +import pytest + +from flake8 import checker + + +def test_oserrors_cause_serial_fall_back(): + """Verify that OSErrors will cause the Manager to fallback to serial.""" + err = OSError(errno.ENOSPC, 'Ominous message about spaceeeeee') + style_guide = mock.Mock() + style_guide.options = mock.Mock(diff=False, jobs='4') + with mock.patch('multiprocessing.Queue', side_effect=err): + manager = checker.Manager(style_guide, [], []) + assert manager.using_multiprocessing is False + + +def test_oserrors_are_reraised(): + """Verify that OSErrors will cause the Manager to fallback to serial.""" + err = OSError(errno.EAGAIN, 'Ominous message') + style_guide = mock.Mock() + style_guide.options = mock.Mock(diff=False, jobs='4') + with mock.patch('multiprocessing.Queue', side_effect=err): + with pytest.raises(OSError): + checker.Manager(style_guide, [], []) From 78b986d5066e229c490c764ce5511f73fe97cc01 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 5 Apr 2016 18:52:35 -0500 Subject: [PATCH 229/364] Refactor checker manager and add tests --- flake8/checker.py | 50 +++++++++++++----------------- tests/unit/test_checker_manager.py | 22 ++++++++++--- 2 files changed, 40 insertions(+), 32 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 17c01cb..58b2eff 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -76,13 +76,14 @@ class Manager(object): self.processes = [] self.checkers = [] - try: - self.process_queue = multiprocessing.Queue() - self.results_queue = multiprocessing.Queue() - except OSError as oserr: - if oserr.errno not in SERIAL_RETRY_ERRNOS: - raise - self.using_multiprocessing = False + if self.using_multiprocessing: + try: + self.process_queue = multiprocessing.Queue() + self.results_queue = multiprocessing.Queue() + except OSError as oserr: + if oserr.errno not in SERIAL_RETRY_ERRNOS: + raise + self.using_multiprocessing = False @staticmethod def _cleanup_queue(q): @@ -160,8 +161,19 @@ class Manager(object): yield result - def _report_after_parallel(self): + def _handle_results(self, filename, results): style_guide = self.style_guide + for (error_code, line_number, column, text, physical_line) in results: + style_guide.handle_error( + code=error_code, + filename=filename, + line_number=line_number, + column_number=column, + text=text, + physical_line=physical_line, + ) + + def _report_after_parallel(self): final_results = {} for (filename, results) in self._results(): final_results[filename] = results @@ -170,30 +182,12 @@ class Manager(object): filename = checker.filename results = sorted(final_results.get(filename, []), key=lambda tup: (tup[1], tup[2])) - for (error_code, line_number, column, text, line) in results: - style_guide.handle_error( - code=error_code, - filename=filename, - line_number=line_number, - column_number=column, - text=text, - physical_line=line, - ) + self._handle_results(filename, results) def _report_after_serial(self): - style_guide = self.style_guide for checker in self.checkers: results = sorted(checker.results, key=lambda tup: (tup[2], tup[3])) - filename = checker.filename - for (error_code, line_number, column, text, line) in results: - style_guide.handle_error( - code=error_code, - filename=filename, - line_number=line_number, - column_number=column, - text=text, - physical_line=line, - ) + self._handle_results(checker.filename, results) def _run_checks_from_queue(self): LOG.info('Running checks in parallel') diff --git a/tests/unit/test_checker_manager.py b/tests/unit/test_checker_manager.py index 2d3143e..7e622bc 100644 --- a/tests/unit/test_checker_manager.py +++ b/tests/unit/test_checker_manager.py @@ -7,11 +7,18 @@ import pytest from flake8 import checker +def style_guide_mock(**kwargs): + kwargs.setdefault('diff', False) + kwargs.setdefault('jobs', '4') + style_guide = mock.Mock() + style_guide.options = mock.Mock(**kwargs) + return style_guide + + def test_oserrors_cause_serial_fall_back(): """Verify that OSErrors will cause the Manager to fallback to serial.""" err = OSError(errno.ENOSPC, 'Ominous message about spaceeeeee') - style_guide = mock.Mock() - style_guide.options = mock.Mock(diff=False, jobs='4') + style_guide = style_guide_mock() with mock.patch('multiprocessing.Queue', side_effect=err): manager = checker.Manager(style_guide, [], []) assert manager.using_multiprocessing is False @@ -20,8 +27,15 @@ def test_oserrors_cause_serial_fall_back(): def test_oserrors_are_reraised(): """Verify that OSErrors will cause the Manager to fallback to serial.""" err = OSError(errno.EAGAIN, 'Ominous message') - style_guide = mock.Mock() - style_guide.options = mock.Mock(diff=False, jobs='4') + style_guide = style_guide_mock() with mock.patch('multiprocessing.Queue', side_effect=err): with pytest.raises(OSError): checker.Manager(style_guide, [], []) + + +def test_multiprocessing_is_disabled(): + """Verify not being able to import multiprocessing forces jobs to 0.""" + style_guide = style_guide_mock() + with mock.patch('flake8.checker.multiprocessing', None): + manager = checker.Manager(style_guide, [], []) + assert manager.jobs == 0 From 102037788dd6f1ddd9be8923a4fc2bc59506059f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 5 Apr 2016 18:54:28 -0500 Subject: [PATCH 230/364] Fix import ordering --- tests/unit/test_checker_manager.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/unit/test_checker_manager.py b/tests/unit/test_checker_manager.py index 7e622bc..63eebce 100644 --- a/tests/unit/test_checker_manager.py +++ b/tests/unit/test_checker_manager.py @@ -1,13 +1,15 @@ """Tests for the Manager object for FileCheckers.""" import errno + +from flake8 import checker + import mock import pytest -from flake8 import checker - def style_guide_mock(**kwargs): + """Create a mock StyleGuide object.""" kwargs.setdefault('diff', False) kwargs.setdefault('jobs', '4') style_guide = mock.Mock() From e32476b6712f0e7b9cf61f7a25c59f14f82bc3dc Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 7 May 2016 19:07:14 -0500 Subject: [PATCH 231/364] Add test for the make_checkers method --- flake8/checker.py | 8 +++++++- tests/unit/test_checker_manager.py | 17 +++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/flake8/checker.py b/flake8/checker.py index 58b2eff..2107fec 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -349,7 +349,13 @@ class FileChecker(object): """Report an error by storing it in the results list.""" if error_code is None: error_code, text = text.split(' ', 1) - physical_line = self.processor.line_for(line_number) + + physical_line = '' + # If we're recovering from a problem in _make_processor, we will not + # have this attribute. + if getattr(self, 'processor', None): + physical_line = self.processor.line_for(line_number) + error = (error_code, line_number, column, text, physical_line) self.results.append(error) return error_code diff --git a/tests/unit/test_checker_manager.py b/tests/unit/test_checker_manager.py index 63eebce..5b91204 100644 --- a/tests/unit/test_checker_manager.py +++ b/tests/unit/test_checker_manager.py @@ -41,3 +41,20 @@ def test_multiprocessing_is_disabled(): with mock.patch('flake8.checker.multiprocessing', None): manager = checker.Manager(style_guide, [], []) assert manager.jobs == 0 + + +def test_make_checkers(): + """Verify that we create a list of FileChecker instances.""" + style_guide = style_guide_mock() + files = ['file1', 'file2'] + with mock.patch('flake8.checker.multiprocessing', None): + manager = checker.Manager(style_guide, files, []) + + with mock.patch('flake8.utils.filenames_from') as filenames_from: + filenames_from.side_effect = [['file1'], ['file2']] + with mock.patch('flake8.utils.fnmatch', return_value=True): + with mock.patch('flake8.processor.FileProcessor'): + manager.make_checkers() + + for file_checker in manager.checkers: + assert file_checker.filename in files From d8635bb92a3b8fe00e744c5a1830283145f0a58a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 7 May 2016 19:28:54 -0500 Subject: [PATCH 232/364] Simplify result handling If we start collecting the results in run_parallel *immediately* after starting the worker processes, we do not need another process to handle the results. This also allows us to store all of the results on a the FileChecker class and process results uniformly. This also means we can count the number of errors and warnings in a run and use that to exit appropriately (using SystemExit). --- flake8/checker.py | 42 +++++++++++++++++------------------------- flake8/main/cli.py | 11 +++++++++-- 2 files changed, 26 insertions(+), 27 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 2107fec..c96d6f6 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -173,22 +173,6 @@ class Manager(object): physical_line=physical_line, ) - def _report_after_parallel(self): - final_results = {} - for (filename, results) in self._results(): - final_results[filename] = results - - for checker in self.checkers: - filename = checker.filename - results = sorted(final_results.get(filename, []), - key=lambda tup: (tup[1], tup[2])) - self._handle_results(filename, results) - - def _report_after_serial(self): - for checker in self.checkers: - results = sorted(checker.results, key=lambda tup: (tup[2], tup[3])) - self._handle_results(checker.filename, results) - def _run_checks_from_queue(self): LOG.info('Running checks in parallel') for checker in iter(self.process_queue.get, 'DONE'): @@ -242,12 +226,16 @@ class Manager(object): This iterates over each of the checkers and reports the errors sorted by line number. """ - if not self.using_multiprocessing: - self._report_after_serial() + results_found = 0 + for checker in self.checkers: + results = sorted(checker.results, key=lambda tup: (tup[2], tup[3])) + self._handle_results(checker.filename, results) + results_found += len(results) + return results_found def run_parallel(self): """Run the checkers in parallel.""" - LOG.info('Starting %d process workers', self.jobs - 1) + LOG.info('Starting %d process workers', self.jobs) for i in range(self.jobs): proc = multiprocessing.Process( target=self._run_checks_from_queue @@ -256,10 +244,14 @@ class Manager(object): proc.start() self.processes.append(proc) - proc = multiprocessing.Process(target=self._report_after_parallel) - proc.start() - LOG.info('Started process to report errors') - self.processes.append(proc) + final_results = {} + for (filename, results) in self._results(): + final_results[filename] = results + + for checker in self.checkers: + filename = checker.filename + checker.results = sorted(final_results.get(filename, []), + key=lambda tup: (tup[1], tup[2])) def run_serial(self): """Run the checkers in serial.""" @@ -299,11 +291,11 @@ class Manager(object): for checker in self.checkers: self.process_queue.put(checker) - def stop(self): - """Stop checking files.""" for i in range(self.jobs): self.process_queue.put('DONE') + def stop(self): + """Stop checking files.""" for proc in self.processes: LOG.info('Joining %s to the main process', proc.name) proc.join() diff --git a/flake8/main/cli.py b/flake8/main/cli.py index dbd9603..9d6e824 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -195,6 +195,7 @@ class Application(object): self.options = None self.args = None + self.result_count = 0 def find_plugins(self): # type: () -> NoneType @@ -274,7 +275,7 @@ class Application(object): # type: () -> NoneType """Report all the errors found by flake8 3.0.""" LOG.info('Reporting errors') - self.file_checker_manager.report() + self.result_count = self.file_checker_manager.report() def _run(self, argv): self.find_plugins() @@ -289,7 +290,12 @@ class Application(object): def run(self, argv=None): # type: (Union[NoneType, List[str]]) -> NoneType - """Run our application.""" + """Run our application. + + This method will also handle KeyboardInterrupt exceptions for the + entirety of the flake8 application. If it sees a KeyboardInterrupt it + will forcibly clean up the :class:`~flake8.checker.Manager`. + """ try: self._run(argv) except KeyboardInterrupt as exc: @@ -303,3 +309,4 @@ def main(argv=None): """Main entry-point for the flake8 command-line tool.""" app = Application() app.run(argv) + raise SystemExit(app.result_count > 0) From 3ad798db61b87a82bcea6e83d387bb228e071870 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 7 May 2016 19:34:27 -0500 Subject: [PATCH 233/364] Implement --exit-zero functionality --- flake8/main/cli.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 9d6e824..22c63de 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -197,6 +197,10 @@ class Application(object): self.args = None self.result_count = 0 + def exit(self): + if not self.options.exit_zero: + raise SystemExit(self.result_count > 0) + def find_plugins(self): # type: () -> NoneType """Find and load the plugins for this application.""" @@ -309,4 +313,4 @@ def main(argv=None): """Main entry-point for the flake8 command-line tool.""" app = Application() app.run(argv) - raise SystemExit(app.result_count > 0) + app.exit() From a82f8cb580da54a5d534314cb6fdbeb2a9acd804 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 7 May 2016 20:13:07 -0500 Subject: [PATCH 234/364] Print the total number of errors before exiting --- flake8/main/cli.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 22c63de..1208bfc 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -1,4 +1,6 @@ """Command-line implementation of flake8.""" +from __future__ import print_function + import logging import flake8 @@ -198,6 +200,15 @@ class Application(object): self.result_count = 0 def exit(self): + # type: () -> NoneType + """Handle finalization and exiting the program. + + This should be the last thing called on the application instance. It + will check certain options and exit appropriately. + """ + if self.options.count: + print(self.result_count) + if not self.options.exit_zero: raise SystemExit(self.result_count > 0) From 414922d08e1cd9a61200d0b0485156136070c8e8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 8 May 2016 14:02:40 -0500 Subject: [PATCH 235/364] Add documentation around the cli --- docs/source/index.rst | 1 + docs/source/internal/cli.rst | 26 ++++++++++++ flake8/main/cli.py | 78 +++++++++++++++++++++++++++++++++--- tox.ini | 1 + 4 files changed, 101 insertions(+), 5 deletions(-) create mode 100644 docs/source/internal/cli.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index c8a4f35..4d35951 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -27,6 +27,7 @@ Developer Guide .. toctree:: :maxdepth: 2 + internal/cli internal/formatters internal/option_handling internal/plugin_handling diff --git a/docs/source/internal/cli.rst b/docs/source/internal/cli.rst new file mode 100644 index 0000000..fc696bc --- /dev/null +++ b/docs/source/internal/cli.rst @@ -0,0 +1,26 @@ +Command Line Interface +====================== + +The command line interface of Flake8 is modeled as an application via +:class:`~flake8.main.cli.Application`. When a user runs ``flake8`` at their +command line, :func:`~flake8.main.cli.main` is run which handles +management of the application. + +User input is parsed *twice* to accomodate logging and verbosity options +passed by the user as early as possible so as much logging can be produced as +possible. + +The default flake8 options are registered by +:func:`~flake8.main.cli.register_default_options`. Trying to register these +options in plugins will result in errors. + + +API Documentation +----------------- + +.. autofunction:: flake8.main.cli.main + +.. autoclass:: flake8.main.cli.Application + :members: + +.. autofunction:: flake8.main.cli.register_default_options diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 1208bfc..7c9c39f 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -15,7 +15,32 @@ LOG = logging.getLogger(__name__) def register_default_options(option_manager): - """Register the default options on our OptionManager.""" + """Register the default options on our OptionManager. + + The default options include: + + - ``-v``/``--verbose`` + - ``-q``/``--quiet`` + - ``--count`` + - ``--diff`` + - ``--exclude`` + - ``--filename`` + - ``--format`` + - ``--hang-closing`` + - ``--ignore`` + - ``--max-line-length`` + - ``--select`` + - ``--disable-noqa`` + - ``--show-source`` + - ``--statistics`` + - ``--enabled-extensions`` + - ``--exit-zero`` + - ``-j``/``--jobs`` + - ``--output-file`` + - ``--append-config`` + - ``--config`` + - ``--isolated`` + """ add_option = option_manager.add_option # pep8 options @@ -175,8 +200,12 @@ class Application(object): :param str version: The version of the program/application we're executing. """ + #: The name of the program being run self.program = program + #: The version of the program being run self.version = version + #: The instance of :class:`flake8.options.manager.OptionManager` used + #: to parse and handle the options and arguments passed by the user self.option_manager = manager.OptionManager( prog='flake8', version=flake8.__version__ ) @@ -187,16 +216,31 @@ class Application(object): flake8.configure_logging(preliminary_opts.verbose, preliminary_opts.output_file) + #: The instance of :class:`flake8.plugins.manager.Checkers` self.check_plugins = None + #: The instance of :class:`flake8.plugins.manager.Listeners` self.listening_plugins = None + #: The instance of :class:`flake8.plugins.manager.ReportFormatters` self.formatting_plugins = None + #: The user-selected formatter from :attr:`formatting_plugins` self.formatter = None + #: The :class:`flake8.plugins.notifier.Notifier` for listening plugins self.listener_trie = None + #: The :class:`flake8.style_guide.StyleGuide` built from the user's + #: options self.guide = None + #: The :class:`flake8.checker.Manager` that will handle running all of + #: the checks selected by the user. self.file_checker_manager = None + #: The user-supplied options parsed into an instance of + #: :class:`optparse.Values` self.options = None + #: The left over arguments that were not parsed by + #: :attr:`option_manager` self.args = None + #: The number of errors, warnings, and other messages after running + #: flake8 self.result_count = 0 def exit(self): @@ -214,7 +258,14 @@ class Application(object): def find_plugins(self): # type: () -> NoneType - """Find and load the plugins for this application.""" + """Find and load the plugins for this application. + + If :attr:`check_plugins`, :attr:`listening_plugins`, or + :attr:`formatting_plugins` are ``None`` then this method will update + them with the appropriate plugin manager instance. Given the expense + of finding plugins (via :mod:`pkg_resources`) we want this to be + idempotent and so only update those attributes if they are ``None``. + """ if self.check_plugins is None: self.check_plugins = plugin_manager.Checkers() @@ -280,7 +331,12 @@ class Application(object): def run_checks(self): # type: () -> NoneType - """Run the actual checks with the FileChecker Manager.""" + """Run the actual checks with the FileChecker Manager. + + This method encapsulates the logic to make a + :class:`~flake8.checker.Manger` instance run the checks it is + managing. + """ self.file_checker_manager.start() self.file_checker_manager.run() LOG.info('Finished running') @@ -288,11 +344,16 @@ class Application(object): def report_errors(self): # type: () -> NoneType - """Report all the errors found by flake8 3.0.""" + """Report all the errors found by flake8 3.0. + + This also updates the :attr:`result_count` attribute with the total + number of errors, warnings, and other messages found. + """ LOG.info('Reporting errors') self.result_count = self.file_checker_manager.report() def _run(self, argv): + # type: (Union[NoneType, List[str]]) -> NoneType self.find_plugins() self.register_plugin_options() self.parse_configuration_and_cli(argv) @@ -321,7 +382,14 @@ class Application(object): def main(argv=None): # type: (Union[NoneType, List[str]]) -> NoneType - """Main entry-point for the flake8 command-line tool.""" + """Main entry-point for the flake8 command-line tool. + + This handles the creation of an instance of :class:`Application`, runs it, + and then exits the application. + + :param list argv: + The arguments to be passed to the application for parsing. + """ app = Application() app.run(argv) app.exit() diff --git a/tox.ini b/tox.ini index cd56e86..d93448f 100644 --- a/tox.ini +++ b/tox.ini @@ -76,6 +76,7 @@ commands = [testenv:docs] deps = sphinx>=1.3.0 + sphinx_rtd_theme commands = sphinx-build -E -W -c docs/source/ -b html docs/source/ docs/build/html From c8a0b6f05eab2b7b5ca50dc72e780a2460466695 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 8 May 2016 14:04:46 -0500 Subject: [PATCH 236/364] Add bandit for security linting --- tox.ini | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tox.ini b/tox.ini index d93448f..e11577a 100644 --- a/tox.ini +++ b/tox.ini @@ -58,6 +58,16 @@ deps = commands = mypy flake8 +[testenv:bandit] +basepython = python3 +skipsdist = true +skip_install = true +use_develop = false +deps = + bandit +commands = + bandit -r flake8/ + [testenv:linters] basepython = python3 skipsdist = true @@ -67,10 +77,12 @@ deps = {[testenv:flake8]deps} {[testenv:pylint]deps} {[testenv:doc8]deps} + {[testenv:bandit]deps} commands = {[testenv:flake8]commands} {[testenv:pylint]commands} {[testenv:doc8]commands} + {[testenv:bandit]commands} # Documentation [testenv:docs] From 44d994dab975f3eb6522e8233de98a52d321f6a2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 14:10:03 -0500 Subject: [PATCH 237/364] Print version information the second time we parse arguments --- flake8/main/cli.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 7c9c39f..d8a95f9 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -2,6 +2,7 @@ from __future__ import print_function import logging +import sys import flake8 from flake8 import checker @@ -212,7 +213,9 @@ class Application(object): register_default_options(self.option_manager) # Set the verbosity of the program - preliminary_opts, _ = self.option_manager.parse_args() + args = sys.argv[:] + args.remove('--version') + preliminary_opts, _ = self.option_manager.parse_args(args) flake8.configure_logging(preliminary_opts.verbose, preliminary_opts.output_file) From abdc957f17afe2a9f160627098deb303e3492f70 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 14:11:47 -0500 Subject: [PATCH 238/364] Handle determining plugin versions and names Since we now treat pep8 checks each as an individual plugin, we need a way to represent pep8 as a single plugin in the version output (like we currently do). As such we need to be a bit wiser in how we tell the OptionManager about plugins and their versions and we only do this for certain plugins. --- flake8/main/cli.py | 1 + flake8/options/manager.py | 12 ++--- flake8/plugins/manager.py | 99 ++++++++++++++++++++++++++++++++++++--- 3 files changed, 97 insertions(+), 15 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index d8a95f9..e168bef 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -282,6 +282,7 @@ class Application(object): # type: () -> NoneType """Register options provided by plugins to our option manager.""" self.check_plugins.register_options(self.option_manager) + self.check_plugins.register_plugin_versions(self.option_manager) self.listening_plugins.register_options(self.option_manager) self.formatting_plugins.register_options(self.option_manager) diff --git a/flake8/options/manager.py b/flake8/options/manager.py index cb4c831..bf144c9 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -167,7 +167,7 @@ class OptionManager(object): @staticmethod def format_plugin(plugin_tuple): """Convert a plugin tuple into a dictionary mapping name to value.""" - return dict(zip(["entry", "name", "version"], plugin_tuple)) + return dict(zip(["name", "version"], plugin_tuple)) def add_option(self, *args, **kwargs): """Create and register a new option. @@ -213,7 +213,7 @@ class OptionManager(object): def generate_epilog(self): """Create an epilog with the version and name of each of plugin.""" - plugin_version_format = '%(name)s(%(entry)s): %(version)s' + plugin_version_format = '%(name)s: %(version)s' self.parser.epilog = 'Installed plugins: ' + self.generate_versions( plugin_version_format ) @@ -229,17 +229,13 @@ class OptionManager(object): return options, xargs - def register_plugin(self, entry_point_name, name, version): + def register_plugin(self, name, version): """Register a plugin relying on the OptionManager. - :param str entry_point_name: - The name of the entry-point loaded with pkg_resources. For - example, if the entry-point looks like: ``C90 = mccabe.Checker`` - then the ``entry_point_name`` would be ``C90``. :param str name: The name of the checker itself. This will be the ``name`` attribute of the class or function loaded from the entry-point. :param str version: The version of the checker that we're using. """ - self.registered_plugins.add((entry_point_name, name, version)) + self.registered_plugins.add((name, version)) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 872a4f4..56ee813 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -18,6 +18,8 @@ __all__ = ( 'ReportFormatters', ) +NO_GROUP_FOUND = object() + class Plugin(object): """Wrap an EntryPoint from setuptools and other logic.""" @@ -36,6 +38,9 @@ class Plugin(object): self.entry_point = entry_point self._plugin = None self._parameters = None + self._group = None + self._plugin_name = None + self._version = None def __repr__(self): """Provide an easy to read description of the current plugin.""" @@ -43,6 +48,28 @@ class Plugin(object): self.name, self.entry_point ) + def is_in_a_group(self): + """Determine if this plugin is in a group. + + :returns: + True if the plugin is in a group, otherwise False. + :rtype: + bool + """ + return self.group() is not None + + def group(self): + """Find and parse the group the plugin is in.""" + if self._group is None: + name = self.name.split('.', 1) + if len(name) > 1: + self._group = name[0] + else: + self._group = NO_GROUP_FOUND + if self._group is NO_GROUP_FOUND: + return None + return self._group + @property def parameters(self): """List of arguments that need to be passed to the plugin.""" @@ -61,8 +88,25 @@ class Plugin(object): @property def version(self): - """Return the version attribute on the plugin.""" - return self.plugin.version + """Return the version of the plugin.""" + if self._version is None: + if self.is_in_a_group(): + self._version = version_for(self) + else: + self._version = self.plugin.version + + return self._version + + @property + def plugin_name(self): + """Return the name of the plugin.""" + if self._plugin_name is None: + if self.is_in_a_group(): + self._plugin_name = self.group() + else: + self._plugin_name = self.plugin.name + + return self._plugin_name def execute(self, *args, **kwargs): r"""Call the plugin with \*args and \*\*kwargs.""" @@ -136,11 +180,6 @@ class Plugin(object): self.name, optmanager ) add_options(optmanager) - optmanager.register_plugin( - entry_point_name=self.name, - name=self.plugin.name, - version=self.plugin.version - ) class PluginManager(object): # pylint: disable=too-few-public-methods @@ -193,6 +232,46 @@ class PluginManager(object): # pylint: disable=too-few-public-methods for name in self.names: yield func(self.plugins[name], *args, **kwargs) + def versions(self): + # () -> (str, str) + """Generate the versions of plugins. + + :returns: + Tuples of the plugin_name and version + :rtype: + tuple + """ + plugins_seen = set() + for entry_point_name in self.names: + plugin = self.plugins[entry_point_name] + plugin_name = plugin.plugin_name + if plugin.plugin_name in plugins_seen: + continue + plugins_seen.add(plugin_name) + yield (plugin_name, plugin.version) + + +def version_for(plugin): + # (Plugin) -> Union[str, NoneType] + """Determine the version of a plugin by it's module. + + :param plugin: + The loaded plugin + :type plugin: + Plugin + :returns: + version string for the module + :rtype: + str + """ + module_name = plugin.plugin.__module__ + try: + module = __import__(module_name) + except ImportError: + return None + + return getattr(module, '__version__', None) + class PluginTypeManager(object): """Parent class for most of the specific plugin types.""" @@ -264,6 +343,12 @@ class PluginTypeManager(object): self.plugins_loaded = True return plugins + def register_plugin_versions(self, optmanager): + """Register the plugins and their versions with the OptionManager.""" + self.load_plugins() + for (plugin_name, version) in self.manager.versions(): + optmanager.register_plugin(name=plugin_name, version=version) + def register_options(self, optmanager): """Register all of the checkers' options to the OptionManager.""" self.load_plugins() From dd0c50dbcea017a5d5483dc784df81e33b644c17 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 14:17:23 -0500 Subject: [PATCH 239/364] Ensure we provide options to all plugins --- flake8/main/cli.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index e168bef..6991fb0 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -300,6 +300,12 @@ class Application(object): self.check_plugins.provide_options(self.option_manager, self.options, self.args) + self.listening_plugins.provide_options(self.option_manager, + self.options, + self.args) + self.formatting_plugins.provide_options(self.option_manager, + self.options, + self.args) def make_formatter(self): # type: () -> NoneType From f20d44565b32a0396be8e281a4ec723172df0655 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 14:23:16 -0500 Subject: [PATCH 240/364] Ignore --help/-h until later as well Explain why we remove --version, --help, and -h --- flake8/main/cli.py | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 6991fb0..ef9b328 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -212,10 +212,30 @@ class Application(object): ) register_default_options(self.option_manager) - # Set the verbosity of the program + # We haven't found or registered our plugins yet, so let's defer + # printing the version until we aggregate options from config files + # and the command-line. First, let's clone our arguments on the CLI, + # then we'll attempt to remove ``--version`` so that we can avoid + # triggering the "version" action in optparse. If it's not there, we + # do not need to worry and we can continue. If it is, we successfully + # defer printing the version until just a little bit later. + # Similarly we have to defer printing the help text until later. args = sys.argv[:] - args.remove('--version') + try: + args.remove('--version') + except ValueError: + pass + try: + args.remove('--help') + except ValueError: + pass + try: + args.remove('-h') + except ValueError: + pass + preliminary_opts, _ = self.option_manager.parse_args(args) + # Set the verbosity of the program flake8.configure_logging(preliminary_opts.verbose, preliminary_opts.output_file) From 559922dbbc86f521bd7e09855779f21e6cf5dd3e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 16:37:45 -0500 Subject: [PATCH 241/364] Update tests after last night's refactor --- tests/unit/test_option_manager.py | 38 +++++++++++++++---------------- tests/unit/test_plugin.py | 6 ----- 2 files changed, 18 insertions(+), 26 deletions(-) diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 1ba5442..0086585 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -119,8 +119,7 @@ def test_parse_args_normalize_paths(optmanager): def test_format_plugin(): """Verify that format_plugin turns a tuple into a dictionary.""" - plugin = manager.OptionManager.format_plugin(('T101', 'Testing', '0.0.0')) - assert plugin['entry'] == 'T101' + plugin = manager.OptionManager.format_plugin(('Testing', '0.0.0')) assert plugin['name'] == 'Testing' assert plugin['version'] == '0.0.0' @@ -128,9 +127,9 @@ def test_format_plugin(): def test_generate_versions(optmanager): """Verify a comma-separated string is generated of registered plugins.""" optmanager.registered_plugins = [ - ('T100', 'Testing 100', '0.0.0'), - ('T101', 'Testing 101', '0.0.0'), - ('T300', 'Testing 300', '0.0.0'), + ('Testing 100', '0.0.0'), + ('Testing 101', '0.0.0'), + ('Testing 300', '0.0.0'), ] assert (optmanager.generate_versions() == 'Testing 100: 0.0.0, Testing 101: 0.0.0, Testing 300: 0.0.0') @@ -138,14 +137,13 @@ def test_generate_versions(optmanager): def test_generate_versions_with_format_string(optmanager): """Verify a comma-separated string is generated of registered plugins.""" - optmanager.registered_plugins = [ - ('T100', 'Testing', '0.0.0'), - ('T101', 'Testing', '0.0.0'), - ('T300', 'Testing', '0.0.0'), - ] + optmanager.registered_plugins.update([ + ('Testing', '0.0.0'), + ('Testing', '0.0.0'), + ('Testing', '0.0.0'), + ]) assert ( - optmanager.generate_versions('%(name)s(%(entry)s): %(version)s') == - 'Testing(T100): 0.0.0, Testing(T101): 0.0.0, Testing(T300): 0.0.0' + optmanager.generate_versions() == 'Testing: 0.0.0' ) @@ -155,9 +153,9 @@ def test_update_version_string(optmanager): assert optmanager.parser.version == TEST_VERSION optmanager.registered_plugins = [ - ('T100', 'Testing 100', '0.0.0'), - ('T101', 'Testing 101', '0.0.0'), - ('T300', 'Testing 300', '0.0.0'), + ('Testing 100', '0.0.0'), + ('Testing 101', '0.0.0'), + ('Testing 300', '0.0.0'), ] optmanager.update_version_string() @@ -172,14 +170,14 @@ def test_generate_epilog(optmanager): assert optmanager.parser.epilog is None optmanager.registered_plugins = [ - ('T100', 'Testing 100', '0.0.0'), - ('T101', 'Testing 101', '0.0.0'), - ('T300', 'Testing 300', '0.0.0'), + ('Testing 100', '0.0.0'), + ('Testing 101', '0.0.0'), + ('Testing 300', '0.0.0'), ] expected_value = ( - 'Installed plugins: Testing 100(T100): 0.0.0, Testing 101(T101): ' - '0.0.0, Testing 300(T300): 0.0.0' + 'Installed plugins: Testing 100: 0.0.0, Testing 101: 0.0.0, Testing' + ' 300: 0.0.0' ) optmanager.generate_epilog() diff --git a/tests/unit/test_plugin.py b/tests/unit/test_plugin.py index 0f6eec1..e995c7b 100644 --- a/tests/unit/test_plugin.py +++ b/tests/unit/test_plugin.py @@ -114,12 +114,6 @@ def test_register_options(): # Assert that we call add_options plugin_obj.add_options.assert_called_once_with(option_manager) - # Assert that we register the plugin - option_manager.register_plugin.assert_called_once_with( - entry_point_name='T000', - version=plugin_obj.version, - name=plugin_obj.name, - ) def test_register_options_checks_plugin_for_method(): From dc919ba4af326fdf31d59ebaf18fc734e60bbfce Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 17:09:31 -0500 Subject: [PATCH 242/364] Add more environments to Travis CI config --- .travis.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 46fffdb..889f406 100644 --- a/.travis.yml +++ b/.travis.yml @@ -22,6 +22,8 @@ matrix: - python: pypy env: TOXENV=pypy - python: 2.7 - env: TOXENV=py27-flake8 + env: TOXENV=readme - python: 3.4 - env: TOXENV=py34-flake8 + env: TOXENV=linters + - python: 3.4 + env: TOXENV=docs From c8ad626525631314decbb5bd544be47102610fd0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 17:09:44 -0500 Subject: [PATCH 243/364] Trim unsupported environments --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index e11577a..cfbc877 100644 --- a/tox.ini +++ b/tox.ini @@ -1,6 +1,6 @@ [tox] minversion=2.3.1 -envlist = py26,py27,py32,py33,py34,py35,flake8 +envlist = py27,py33,py34,py35,flake8 [testenv] deps = From adaa47730ee0ef119a9394bf477764fdba3de3a4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 17:26:11 -0500 Subject: [PATCH 244/364] Comment html_static_path in sphinx config --- docs/source/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 0d10af2..f30410b 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -144,7 +144,7 @@ html_theme = 'sphinx_rtd_theme' # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] +# html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied From ccadc09a0e2489be41e024a2bfa1a100072af041 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 20:12:17 -0500 Subject: [PATCH 245/364] Add documentation about how we handle pep8 as a plugin --- docs/source/internal/plugin_handling.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst index b3b9b0e..d4692ce 100644 --- a/docs/source/internal/plugin_handling.rst +++ b/docs/source/internal/plugin_handling.rst @@ -83,6 +83,21 @@ Finally, Flake8 has always provided its own plugin shim for Pyflakes. As part of that we carry our own shim in-tree and now store that in :mod:`flake8.plugins.pyflakes`. +Flake8 also registers plugins for pep8. Each check in pep8 requires different +parameters and it cannot easily be shimmed together like Pyflakes was. As +such, plugins have a concept of a "group". If you look at our :file:`setup.py` +you will see that we register pep8 checks roughly like so: + +.. code:: + + pep8. = pep8: + +We do this to identify that ``>`` is part of a group. This also +enables us to special-case how we handle reporting those checks. Instead of +reporting each check in the ``--version`` output, we report ``pep8`` and check +``pep8`` the module for a ``__version__`` attribute. We only report it once +to avoid confusing users. + API Documentation ----------------- From d929dd57cb80a31f151535d32f7593d95c4e80ee Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 10 May 2016 20:12:38 -0500 Subject: [PATCH 246/364] Add a property for off-by-default plugins If a plugin is off-by-default use its entry-point name (as we currently do in flake8 2.x) to add it to the default ignore list. --- flake8/plugins/manager.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index 56ee813..fe51391 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -108,6 +108,11 @@ class Plugin(object): return self._plugin_name + @property + def off_by_default(self): + """Return whether the plugin is ignored by default.""" + return getattr(self.plugin, 'off_by_default', False) + def execute(self, *args, **kwargs): r"""Call the plugin with \*args and \*\*kwargs.""" return self.plugin(*args, **kwargs) # pylint: disable=not-callable @@ -181,6 +186,9 @@ class Plugin(object): ) add_options(optmanager) + if self.off_by_default: + optmanager.extend_default_ignore([self.name]) + class PluginManager(object): # pylint: disable=too-few-public-methods """Find and manage plugins consistently.""" From 89762b9dd17b15e60595b843d0f90dfc0d68f75b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 14 May 2016 19:35:58 -0500 Subject: [PATCH 247/364] Document flake8.checker and flake8.processor --- docs/source/index.rst | 1 + docs/source/internal/checker.rst | 70 ++++++++++++++++++++++++++++++++ flake8/processor.py | 28 ++++++------- 3 files changed, 85 insertions(+), 14 deletions(-) create mode 100644 docs/source/internal/checker.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 4d35951..74810ed 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -27,6 +27,7 @@ Developer Guide .. toctree:: :maxdepth: 2 + internal/checker internal/cli internal/formatters internal/option_handling diff --git a/docs/source/internal/checker.rst b/docs/source/internal/checker.rst new file mode 100644 index 0000000..fb5380e --- /dev/null +++ b/docs/source/internal/checker.rst @@ -0,0 +1,70 @@ +=============== + Check Running +=============== + +In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 we have taken +that responsibility upon ourselves. This has allowed us to simplify our +handling of the ``--jobs`` parameter (using :mod:`multiprocessing`) as well as +simplifying our fallback in the event something goes awry with concurency. +At the lowest level we have a |FileChecker|. Instances of |FileChecker| are +created for *each* file to be analyzed by Flake8. Each instance, has a copy of +all of the plugins registered with setuptools in the ``flake8.extension`` +entry-point group. + +The |FileChecker| instances are managed by an instance of |Manager|. The +|Manager| instance is what handles creating sub-processes with +:mod:`multiprocessing` module and falling back to running checks in serial if +an operating system level error arises. When creating |FileChecker| instances, +the |Manager| is responsible for determining if a particular file has been +excluded. + + +Processing Files +---------------- + +Unfortunately, since Flake8 took over check running from pep8/pycodestyle, it +also was required to take over parsing and processing files for the checkers +to use. Since we couldn't reuse pycodestyle's functionality (since it did not +separate cleanly the processing from check running) we isolated that function +into the :class:`~flake8.processor.FileProcessor` class. Further, we moved +several helper functions into the :mod:`flake8.processor` module (see also +:ref:`Processor Utility Functions `). + + +API Reference +------------- + +.. autoclass:: flake8.checker.FileChecker + :members: + +.. autoclass:: flake8.checker.Manager + :members: + +.. autoclass:: flake8.processor.FileProcessor + :members: + + +.. _processor_utility_functions: + +Utility Functions +````````````````` + +.. autofunction:: flake8.processor.count_parentheses + +.. autofunction:: flake8.processor.expand_indent + +.. autofunction:: flake8.processor.is_eol_token + +.. autofunction:: flake8.processor.is_multiline_string + +.. autofunction:: flake8.processor.log_token + +.. autofunction:: flake8.processor.mutate_string + +.. autofunction:: flake8.processor.token_is_comment + +.. autofunction:: flake8.processor.token_is_newline + +.. Substitutions +.. |FileChecker| replace:: :class:`~flake8.checker.FileChecker` +.. |Manager| replace:: :class:`~flake8.checker.Manager` diff --git a/flake8/processor.py b/flake8/processor.py index e71784c..ab99089 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -30,20 +30,20 @@ class FileProcessor(object): to checks expecting that state. Any public attribute on this object can be requested by a plugin. The known public attributes are: - - blank_before - - blank_lines - - indect_char - - indent_level - - line_number - - logical_line - - max_line_length - - multiline - - noqa - - previous_indent_level - - previous_logical - - tokens - - total_lines - - verbose + - :attr:`blank_before` + - :attr:`blank_lines` + - :attr:`indect_char` + - :attr:`indent_level` + - :attr:`line_number` + - :attr:`logical_line` + - :attr:`max_line_length` + - :attr:`multiline` + - :attr:`noqa` + - :attr:`previous_indent_level` + - :attr:`previous_logical` + - :attr:`tokens` + - :attr:`total_lines` + - :attr:`verbose` """ NOQA_FILE = re.compile(r'\s*# flake8[:=]\s*noqa', re.I) From 99030223a420df556500ea1e4aa5fe1d9234fd5d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 16 May 2016 08:17:51 -0500 Subject: [PATCH 248/364] Trim trailing whitespace --- docs/source/internal/checker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/internal/checker.rst b/docs/source/internal/checker.rst index fb5380e..b4f3b72 100644 --- a/docs/source/internal/checker.rst +++ b/docs/source/internal/checker.rst @@ -27,7 +27,7 @@ also was required to take over parsing and processing files for the checkers to use. Since we couldn't reuse pycodestyle's functionality (since it did not separate cleanly the processing from check running) we isolated that function into the :class:`~flake8.processor.FileProcessor` class. Further, we moved -several helper functions into the :mod:`flake8.processor` module (see also +several helper functions into the :mod:`flake8.processor` module (see also :ref:`Processor Utility Functions `). From c68978116683adbad8177ee523be5244c63a4ad8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 16 May 2016 15:58:59 -0500 Subject: [PATCH 249/364] Update title --- docs/source/internal/checker.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/internal/checker.rst b/docs/source/internal/checker.rst index b4f3b72..4c2789f 100644 --- a/docs/source/internal/checker.rst +++ b/docs/source/internal/checker.rst @@ -1,6 +1,6 @@ -=============== - Check Running -=============== +==================== + How Checks are Run +==================== In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 we have taken that responsibility upon ourselves. This has allowed us to simplify our From 72833b629a506923f4df01cc114032c9343f8e4a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 16 May 2016 20:15:14 -0500 Subject: [PATCH 250/364] Start adding documentation about developing plugins --- docs/source/dev/plugin_parameters.rst | 48 +++++++++++++++++++++++++++ docs/source/index.rst | 1 + flake8/processor.py | 1 + 3 files changed, 50 insertions(+) create mode 100644 docs/source/dev/plugin_parameters.rst diff --git a/docs/source/dev/plugin_parameters.rst b/docs/source/dev/plugin_parameters.rst new file mode 100644 index 0000000..a2c8c7f --- /dev/null +++ b/docs/source/dev/plugin_parameters.rst @@ -0,0 +1,48 @@ +==================================== + Receiving Information For A Plugin +==================================== + +Plugins to Flake8 have a great deal of information that they can request from +a :class:`~flake8.processor.FileProcessor` instance. Historically, Flake8 has +supported two types of plugins: + +#. classes that accept parsed abstract syntax trees (ASTs) + +#. functions that accept a range of arguments + +Flake8 now does not distinguish between the two types of plugins. Any plugin +can accept either an AST or a range of arguments. Further, any plugin that has +certain callable attributes can also register options and receive parsed +options. + +Indicating Desired Data +======================= + +Flake8 inspects the plugin's signature to determine what parameters it expects +using :func:`flake8.utils.parameters_for`. +:attr:`flake8.plugins.manager.Plugin.parameters` caches the values so that +each plugin makes that fairly expensive call once per plugin. When processing +a file, a plugin can ask for any of the following: + +- :attr:`~flake8.processor.FileProcessor.blank_before` +- :attr:`~flake8.processor.FileProcessor.blank_lines` +- :attr:`~flake8.processor.FileProcessor.checker_state` +- :attr:`~flake8.processor.FileProcessor.indect_char` +- :attr:`~flake8.processor.FileProcessor.indent_level` +- :attr:`~flake8.processor.FileProcessor.line_number` +- :attr:`~flake8.processor.FileProcessor.logical_line` +- :attr:`~flake8.processor.FileProcessor.max_line_length` +- :attr:`~flake8.processor.FileProcessor.multiline` +- :attr:`~flake8.processor.FileProcessor.noqa` +- :attr:`~flake8.processor.FileProcessor.previous_indent_level` +- :attr:`~flake8.processor.FileProcessor.previous_logical` +- :attr:`~flake8.processor.FileProcessor.tokens` +- :attr:`~flake8.processor.FileProcessor.total_lines` +- :attr:`~flake8.processor.FileProcessor.verbose` + +Alternatively, a plugin can accept ``tree`` and ``filename``. +``tree`` will be a parsed abstract syntax tree that will be used by plugins +like PyFlakes and McCabe. + +Finally, any plugin that has callable attributes ``provide_options`` and +``register_options`` can parse option information and register new options. diff --git a/docs/source/index.rst b/docs/source/index.rst index 74810ed..e1b4165 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -19,6 +19,7 @@ Plugin Developer Guide :maxdepth: 2 dev/formatters + dev/plugin_parameters dev/registering_plugins Developer Guide diff --git a/flake8/processor.py b/flake8/processor.py index ab99089..8e6d897 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -32,6 +32,7 @@ class FileProcessor(object): - :attr:`blank_before` - :attr:`blank_lines` + - :attr:`checker_state` - :attr:`indect_char` - :attr:`indent_level` - :attr:`line_number` From 6b9b4a5aca6569ca2f89a2d19fe3984718aa28e9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 16 May 2016 20:53:04 -0500 Subject: [PATCH 251/364] Add more information about providing information to plugins --- docs/source/dev/plugin_parameters.rst | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/source/dev/plugin_parameters.rst b/docs/source/dev/plugin_parameters.rst index a2c8c7f..d1d6cb0 100644 --- a/docs/source/dev/plugin_parameters.rst +++ b/docs/source/dev/plugin_parameters.rst @@ -15,6 +15,7 @@ can accept either an AST or a range of arguments. Further, any plugin that has certain callable attributes can also register options and receive parsed options. + Indicating Desired Data ======================= @@ -44,5 +45,26 @@ Alternatively, a plugin can accept ``tree`` and ``filename``. ``tree`` will be a parsed abstract syntax tree that will be used by plugins like PyFlakes and McCabe. -Finally, any plugin that has callable attributes ``provide_options`` and + +Registering and Parsing Options +=============================== + +Any plugin that has callable attributes ``provide_options`` and ``register_options`` can parse option information and register new options. + +Your ``register_options`` function should expect to receive an instance of +|OptionManager|. An |OptionManager| instance behaves very similarly to +:class:`optparse.OptionParser`. It, however, uses the layer that Flake8 has +developed on top of :mod:`optparse` to also handle configuration file parsing. +:meth:`~flake8.options.manager.OptionManager.add_option` creates an |Option| +which accepts the same parameters as :mod:`optparse` as well as three extra +boolean parameters: + +- ``parse_from_config`` +- ``comma_separated_list`` +- ``normalize_paths`` + + +.. substitutions +.. |OptionManager| replace:: :class:`~flake8.options.manager.OptionManager` +.. |Option| replace:: :class:`~flake8.options.manager.Option` From 9ada68118dca394441c2ba0a6bd94ce3ab898d45 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 17 May 2016 20:01:49 -0500 Subject: [PATCH 252/364] Finish writing about registering plugin options --- docs/source/dev/plugin_parameters.rst | 78 +++++++++++++++++++++++++-- 1 file changed, 75 insertions(+), 3 deletions(-) diff --git a/docs/source/dev/plugin_parameters.rst b/docs/source/dev/plugin_parameters.rst index d1d6cb0..17a4dd3 100644 --- a/docs/source/dev/plugin_parameters.rst +++ b/docs/source/dev/plugin_parameters.rst @@ -22,7 +22,7 @@ Indicating Desired Data Flake8 inspects the plugin's signature to determine what parameters it expects using :func:`flake8.utils.parameters_for`. :attr:`flake8.plugins.manager.Plugin.parameters` caches the values so that -each plugin makes that fairly expensive call once per plugin. When processing +each plugin makes that fairly expensive call once per plugin. When processing a file, a plugin can ask for any of the following: - :attr:`~flake8.processor.FileProcessor.blank_before` @@ -46,8 +46,8 @@ Alternatively, a plugin can accept ``tree`` and ``filename``. like PyFlakes and McCabe. -Registering and Parsing Options -=============================== +Registering Options +=================== Any plugin that has callable attributes ``provide_options`` and ``register_options`` can parse option information and register new options. @@ -61,9 +61,81 @@ which accepts the same parameters as :mod:`optparse` as well as three extra boolean parameters: - ``parse_from_config`` + + The command-line option should also be parsed from config files discovered + by Flake8. + + .. note:: + + This takes the place of appending strings to a list on the + :class:`optparse.OptionParser`. + - ``comma_separated_list`` + + The value provided to this option is a comma-separated list. After parsing + the value, it should be further broken up into a list. This also allows us + to handle values like: + + .. code:: + + E123,E124, + E125, + E126 + - ``normalize_paths`` + The value provided to this option is a path. It should be normalized to be + an absolute path. This can be combined with ``comma_separated_list`` to + allow a comma-separated list of paths. + +Each of these options works individually or can be combined. Let's look at a +couple examples from Flake8. In each example, we will have ``option_manager`` +which is an instance of |OptionManager|. + +.. code-block:: python + + option_manager.add_option( + '--max-line-length', type='int', metavar='n', + default=defaults.MAX_LINE_LENGTH, parse_from_config=True, + help='Maximum allowed line length for the entirety of this run. ' + '(Default: %default)', + ) + +Here we are adding the ``--max-line-length`` command-line option which is +always an integer and will be parsed from the configuration file. Since we +provide a default, we take advantage of :mod:`optparse`\ 's willingness to +display that in the help text with ``%default``. + +.. code-block:: python + + option_manager.add_option( + '--select', metavar='errors', default='', + parse_from_config=True, comma_separated_list=True, + help='Comma-separated list of errors and warnings to enable.' + ' For example, ``--select=E4,E51,W234``. (Default: %default)', + ) + +In adding the ``--select`` command-line option, we're also indicating to the +|OptionManager| that we want the value parsed from the config files and parsed +as a comma-separated list. + +.. code-block:: python + + option_manager.add_option( + '--exclude', metavar='patterns', default=defaults.EXCLUDE, + comma_separated_list=True, parse_from_config=True, + normalize_paths=True, + help='Comma-separated list of files or directories to exclude.' + '(Default: %default)', + ) + +Finally, we show an option that uses all three extra flags. Values from +``--exclude`` will be parsed from the config, converted from a comma-separated +list, and then each item will be normalized. + +For information about other parameters to +:meth:`~flake8.options.manager.OptionManager.add_option` refer to the +documentation of :mod:`optparse`. .. substitutions .. |OptionManager| replace:: :class:`~flake8.options.manager.OptionManager` From 36bb148ad542b99816de771739f6d2ad25343245 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 17 May 2016 20:19:02 -0500 Subject: [PATCH 253/364] Add documentation for receiving parsed options --- docs/source/dev/plugin_parameters.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/source/dev/plugin_parameters.rst b/docs/source/dev/plugin_parameters.rst index 17a4dd3..39db95e 100644 --- a/docs/source/dev/plugin_parameters.rst +++ b/docs/source/dev/plugin_parameters.rst @@ -137,6 +137,25 @@ For information about other parameters to :meth:`~flake8.options.manager.OptionManager.add_option` refer to the documentation of :mod:`optparse`. + +Accessing Parsed Options +======================== + +When a plugin has a callable ``provide_options`` attribute, Flake8 will call +it and attempt to provide the |OptionManager| instance, the parsed options +which will be an instance of :class:`optparse.Values`, and the extra arguments +that were not parsed by the |OptionManager|. If that fails, we will just pass +the :class:`optparse.Values`. In other words, your ``provide_options`` +callable will have one of the following signatures: + +.. code-block:: python + + def provide_options(option_manager, options, args): + pass + # or + def provide_options(options): + pass + .. substitutions .. |OptionManager| replace:: :class:`~flake8.options.manager.OptionManager` .. |Option| replace:: :class:`~flake8.options.manager.Option` From 1d3b2548144febc638ba4d219717ef0896320959 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 18 May 2016 22:10:42 -0500 Subject: [PATCH 254/364] Nest plugin development docs --- docs/source/dev/formatters.rst | 2 ++ docs/source/dev/index.rst | 17 +++++++++++++++++ docs/source/dev/plugin_parameters.rst | 8 +++++--- docs/source/index.rst | 4 +--- 4 files changed, 25 insertions(+), 6 deletions(-) create mode 100644 docs/source/dev/index.rst diff --git a/docs/source/dev/formatters.rst b/docs/source/dev/formatters.rst index 0b37abd..5307e75 100644 --- a/docs/source/dev/formatters.rst +++ b/docs/source/dev/formatters.rst @@ -1,3 +1,5 @@ +.. _formatting-plugins: + =========================================== Developing a Formatting Plugin for Flake8 =========================================== diff --git a/docs/source/dev/index.rst b/docs/source/dev/index.rst new file mode 100644 index 0000000..43f01c1 --- /dev/null +++ b/docs/source/dev/index.rst @@ -0,0 +1,17 @@ +============================ + Writing Plugins for Flake8 +============================ + +Since Flake8 2.0, the Flake8 tool has allowed for extensions and custom +plugins. In Flake8 3.0, we're expanding that ability to customize and +extend **and** we're attempting to thoroughly document it too. Some of the +documentation in this section will reference third-party documentation +in order to reduce duplication and to point you, the developer, towards +the authoritative documentation for those pieces. + +.. toctree:: + :maxdepth: 2 + + plugin_parameters + registering_plugins + formatters diff --git a/docs/source/dev/plugin_parameters.rst b/docs/source/dev/plugin_parameters.rst index 39db95e..9b074ec 100644 --- a/docs/source/dev/plugin_parameters.rst +++ b/docs/source/dev/plugin_parameters.rst @@ -1,6 +1,8 @@ -==================================== - Receiving Information For A Plugin -==================================== +.. _plugin-parameters: + +========================================== + Receiving Information For A Check Plugin +========================================== Plugins to Flake8 have a great deal of information that they can request from a :class:`~flake8.processor.FileProcessor` instance. Historically, Flake8 has diff --git a/docs/source/index.rst b/docs/source/index.rst index e1b4165..563e33d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -18,9 +18,7 @@ Plugin Developer Guide .. toctree:: :maxdepth: 2 - dev/formatters - dev/plugin_parameters - dev/registering_plugins + dev/index Developer Guide --------------- From e1fe99a90b5979cae7419b28c3ec4d7c6067e955 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 18 May 2016 22:16:53 -0500 Subject: [PATCH 255/364] Nest internal docs --- docs/source/index.rst | 7 +------ docs/source/internal/index.rst | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+), 6 deletions(-) create mode 100644 docs/source/internal/index.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 563e33d..f940e3d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -26,12 +26,7 @@ Developer Guide .. toctree:: :maxdepth: 2 - internal/checker - internal/cli - internal/formatters - internal/option_handling - internal/plugin_handling - internal/utils + internal/index Indices and tables ================== diff --git a/docs/source/internal/index.rst b/docs/source/internal/index.rst new file mode 100644 index 0000000..4220fe9 --- /dev/null +++ b/docs/source/internal/index.rst @@ -0,0 +1,23 @@ +============================== + Exploring Flake8's Internals +============================== + +While writing Flake8 3.0, the developers attempted to capture some reasoning +and decision information in internal documentation meant for future developers +and maintaners. Most of this information is unnecessary for users and plugin +developers. Some of it, however, is linked to from the plugin development +documentation. + +Keep in mind that not everything will be here and you may need to help pull +information out of the developers' heads and into these documents. Please +pull gently. + +.. toctree:: + :maxdepth: 2 + + checker + cli + formatters + option_handling + plugin_handling + utils From 25df2df55c42edd79db78e078c0bcd11ec8b4d3d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 18 May 2016 22:28:32 -0500 Subject: [PATCH 256/364] Add installation and quickstart to docs start page --- docs/source/index.rst | 57 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/docs/source/index.rst b/docs/source/index.rst index f940e3d..b5a4ad5 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -6,6 +6,63 @@ Flake8: Your Tool For Style Guide Enforcement ============================================= +Installation +------------ + +To install Flake8, open an interactive shell and run: + +.. code:: + + python -m pip install flake8 + +If you want Flake8 to be installed for your default Python installation, you +can instead use: + +.. code:: + + python -m pip install flake8 + +.. note:: + + It is **very** important to install Flake8 on the *correct* version of + Python for your needs. If you want Flake8 to properly parse new language + features in Python 3.5 (for example), you need it to be installed on 3.5 + for those features to be understandable. In many ways, Flake8 is tied to + the version of Python on which it runs. + +Quickstart +---------- + +To start using Flake8, open an interactive shell and run: + +.. code:: + + flake8 path/to/code/to/check.py + # or + flake8 path/to/code/ + +.. note:: + + If you have installed Flake8 on a particular version of Python (or on + several versions), it may be best to instead run ``python -m + flake8``. + +If you only want to see the instances of a specific warning or error, you can +*select* that error like so: + +.. code:: + + flake8 --select path/to/code/ + +Alternatively, if you want to *ignore* only one specific warning or error: + +.. code:: + + flake8 --ignore path/to/code/ + +Please read our user guide for more information about how to use and configure +Flake8. + User Guide ---------- From 28a97a84999fd4a7bcced54193ab25edd1d1d889 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 18 May 2016 22:28:44 -0500 Subject: [PATCH 257/364] Add user guide index page --- docs/source/index.rst | 2 ++ docs/source/user/index.rst | 4 ++++ 2 files changed, 6 insertions(+) create mode 100644 docs/source/user/index.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index b5a4ad5..d7ed3a2 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -69,6 +69,8 @@ User Guide .. toctree:: :maxdepth: 2 + user/index + Plugin Developer Guide ---------------------- diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst new file mode 100644 index 0000000..81732a4 --- /dev/null +++ b/docs/source/user/index.rst @@ -0,0 +1,4 @@ +============== + Using Flake8 +============== + From f353a5db907a97b6513c00500f9c593bc35cc76e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 18 May 2016 23:05:09 -0500 Subject: [PATCH 258/364] Add a tiny bit to Usage Guide index --- docs/source/user/index.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 81732a4..8368509 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -2,3 +2,17 @@ Using Flake8 ============== +Flake8 can be used in many ways: + +- invoking it on the command-line + +- invoking it via Python + +- having it called by Git or Mercurial on or around committing + +This guide will cover all of these and the nuances for using Flake8. + +.. config files +.. command-line tutorial +.. VCS usage +.. installing and using plugins From a7898e038932e771d0499f6dc30b8151d347c472 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 26 May 2016 13:21:25 -0500 Subject: [PATCH 259/364] Begin adding user-facing documentation Especially around invocation and configuration --- docs/source/conf.py | 1 + docs/source/index.rst | 17 ++-- docs/source/user/configuration.rst | 29 ++++++ docs/source/user/index.rst | 11 +++ docs/source/user/invocation.rst | 144 +++++++++++++++++++++++++++++ tox.ini | 1 + 6 files changed, 196 insertions(+), 7 deletions(-) create mode 100644 docs/source/user/configuration.rst create mode 100644 docs/source/user/invocation.rst diff --git a/docs/source/conf.py b/docs/source/conf.py index f30410b..a44bd97 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -35,6 +35,7 @@ extensions = [ 'sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.viewcode', + 'sphinx-prompt', ] # Add any paths that contain templates here, relative to this directory. diff --git a/docs/source/index.rst b/docs/source/index.rst index d7ed3a2..71b3c94 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -3,11 +3,14 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. -Flake8: Your Tool For Style Guide Enforcement -============================================= +=============================================== + Flake8: Your Tool For Style Guide Enforcement +=============================================== + +.. _installation-guide: Installation ------------- +============ To install Flake8, open an interactive shell and run: @@ -31,7 +34,7 @@ can instead use: the version of Python on which it runs. Quickstart ----------- +========== To start using Flake8, open an interactive shell and run: @@ -64,7 +67,7 @@ Please read our user guide for more information about how to use and configure Flake8. User Guide ----------- +========== .. toctree:: :maxdepth: 2 @@ -72,7 +75,7 @@ User Guide user/index Plugin Developer Guide ----------------------- +====================== .. toctree:: :maxdepth: 2 @@ -80,7 +83,7 @@ Plugin Developer Guide dev/index Developer Guide ---------------- +=============== .. toctree:: :maxdepth: 2 diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst new file mode 100644 index 0000000..c3f3e2c --- /dev/null +++ b/docs/source/user/configuration.rst @@ -0,0 +1,29 @@ +==================== + Configuring Flake8 +==================== + +Once you have learned how to :ref:`invoke ` Flake8, you will soon +want to learn how to configure it so you do not have to specify the same +options every time you use it. + +This section will show you how to make + +.. prompt:: bash + + flake8 + +Remember that you want to specify certain options without writing + +.. prompt:: bash + + flake8 --select E123,W456 --enable-extensions H111 + + +Configuration Locations +======================= + +Presently, Flake8 supports storing its configuration in the following places: + +- Your top-level user directory + +- In your project in one of ``setup.cfg``, ``tox.ini``, or ``.flake8``. diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 8368509..2085178 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -12,6 +12,17 @@ Flake8 can be used in many ways: This guide will cover all of these and the nuances for using Flake8. +.. note:: + + This portion of Flake8's documentation does not cover installation. See + the :ref:`installation-guide` section for how to install Flake8. + +.. toctree:: + :maxdepth: 2 + + invocation + configuration + .. config files .. command-line tutorial .. VCS usage diff --git a/docs/source/user/invocation.rst b/docs/source/user/invocation.rst new file mode 100644 index 0000000..ef763ae --- /dev/null +++ b/docs/source/user/invocation.rst @@ -0,0 +1,144 @@ +.. _invocation: + +================= + Invoking Flake8 +================= + +Once you have :ref:`installed ` Flake8, you can begin +using it. Most of the time, you will be able to generically invoke Flake8 +like so: + +.. prompt:: bash + + flake8 ... + +Where you simply allow the shell running in your terminal to locate Flake8. +In some cases, though, you may have installed Flake8 for multiple versions +of Python (e.g., Python 2.7 and Python 3.5) and you need to call a specific +version. In that case, you will have much better results using: + +.. prompt:: bash + + python2.7 -m flake8 + +Or + +.. prompt:: bash + + python3.5 -m flake8 + +Since that will tell the correct version of Python to run Flake8. + +.. note:: + + Installing Flake8 once will not install it on both Python 2.7 and + Python 3.5. It will only install it for the version of Python that + is running pip. + +It is also possible to specify command-line options directly to Flake8: + +.. prompt:: bash + + flake8 --select E123 + +Or + +.. prompt:: bash + + python -m flake8 --select E123 + +.. note:: + + This is the last time we will show both versions of an invocation. + From now on, we'll simply use ``flake8`` and assume that the user + knows they can instead use ``python -m flake8`` instead. + +It's also possible to narrow what Flake8 will try to check by specifying +exactly the paths and directories you want it to check. Let's assume that +we have a directory with python files and subdirectories which have python +files (and may have more sub-directories) called ``my_project``. Then if +we only want errors from files found inside ``my_project`` we can do: + +.. prompt:: bash + + flake8 my_project + +And if we only want certain errors (e.g., ``E123``) from files in that +directory we can also do: + +.. prompt:: bash + + flake8 --select E123 my_project + +If you want to explore more options that can be passed on the command-line, +you can use the ``--help`` option: + +.. prompt:: bash + + flake8 --help + +And you should see something like: + +.. code:: + + Usage: flake8 [options] file file ... + + Options: + --version show program's version number and exit + -h, --help show this help message and exit + -v, --verbose Print more information about what is happening in + flake8. This option is repeatable and will increase + verbosity each time it is repeated. + -q, --quiet Report only file names, or nothing. This option is + repeatable. + --count Print total number of errors and warnings to standard + error and set the exit code to 1 if total is not + empty. + --diff Report changes only within line number ranges in the + unified diff provided on standard in by the user. + --exclude=patterns Comma-separated list of files or directories to + exclude.(Default: + .svn,CVS,.bzr,.hg,.git,__pycache__,.tox) + --filename=patterns Only check for filenames matching the patterns in this + comma-separated list. (Default: *.py) + --format=format Format errors according to the chosen formatter. + --hang-closing Hang closing bracket instead of matching indentation + of opening bracket's line. + --ignore=errors Comma-separated list of errors and warnings to ignore + (or skip). For example, ``--ignore=E4,E51,W234``. + (Default: E121,E123,E126,E226,E24,E704) + --max-line-length=n Maximum allowed line length for the entirety of this + run. (Default: 79) + --select=errors Comma-separated list of errors and warnings to enable. + For example, ``--select=E4,E51,W234``. (Default: ) + --disable-noqa Disable the effect of "# noqa". This will report + errors on lines with "# noqa" at the end. + --show-source Show the source generate each error or warning. + --statistics Count errors and warnings. + --enabled-extensions=ENABLED_EXTENSIONS + Enable plugins and extensions that are otherwise + disabled by default + --exit-zero Exit with status code "0" even if there are errors. + -j JOBS, --jobs=JOBS Number of subprocesses to use to run checks in + parallel. This is ignored on Windows. The default, + "auto", will auto-detect the number of processors + available to use. (Default: auto) + --output-file=OUTPUT_FILE + Redirect report to a file. + --append-config=APPEND_CONFIG + Provide extra config files to parse in addition to the + files found by Flake8 by default. These files are the + last ones read and so they take the highest precedence + when multiple files provide the same option. + --config=CONFIG Path to the config file that will be the authoritative + config source. This will cause Flake8 to ignore all + other configuration files. + --isolated Ignore all found configuration files. + --builtins=BUILTINS define more built-ins, comma separated + --doctests check syntax of the doctests + --include-in-doctest=INCLUDE_IN_DOCTEST + Run doctests only on these files + --exclude-from-doctest=EXCLUDE_FROM_DOCTEST + Skip these files when running doctests + + Installed plugins: pyflakes: 1.0.0, pep8: 1.7.0 diff --git a/tox.ini b/tox.ini index cfbc877..24ef6e8 100644 --- a/tox.ini +++ b/tox.ini @@ -89,6 +89,7 @@ commands = deps = sphinx>=1.3.0 sphinx_rtd_theme + sphinx-prompt commands = sphinx-build -E -W -c docs/source/ -b html docs/source/ docs/build/html From 50d74e3cce34e3047bcb24a2cf7cd85e5a7c1163 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 27 May 2016 11:03:36 -0500 Subject: [PATCH 260/364] Add more to configuration documentation --- docs/source/user/configuration.rst | 49 ++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst index c3f3e2c..1e51e0a 100644 --- a/docs/source/user/configuration.rst +++ b/docs/source/user/configuration.rst @@ -27,3 +27,52 @@ Presently, Flake8 supports storing its configuration in the following places: - Your top-level user directory - In your project in one of ``setup.cfg``, ``tox.ini``, or ``.flake8``. + + +"User" Configuration +-------------------- + +Flake8 allows a user to use "global" configuration file to store preferences. +The user configuration file is expected to be stored somewhere in the user's +"home" directory. + +- On Windows the "home" directory will be something like + ``C:\\Users\sigmavirus24``, a.k.a, ``~\``. + +- On Linux and other Unix like systems (including OS X) we will look in + ``~/``. + +Note that Flake8 looks for ``~\.flake8`` on Windows and ``~/.config/flake8`` +on Linux and other Unix systems. + + +Project Configuration +--------------------- + +Flake8 is written with the understanding that people organize projects into +sub-directories. Let's take for example Flake8's own project structure + +.. code:: + + flake8 + ├── docs + │   ├── build + │   └── source + │   ├── _static + │   ├── _templates + │   ├── dev + │   ├── internal + │   └── user + ├── flake8 + │   ├── formatting + │   ├── main + │   ├── options + │   └── plugins + └── tests + ├── fixtures + │   └── config_files + ├── integration + └── unit + +In the top-level ``flake8`` directory (which contains ``docs``, ``flake8``, +and ``tests``) there's also ``tox.ini`` and ``setup.cfg`` files. From 91e07ebcffc0d4ecc472bb0031113f6030f43a20 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 28 May 2016 07:54:07 -0500 Subject: [PATCH 261/364] Refactor off-by-default plugins and enabling them We move the logic to add or remove a plugin from the default ignore list to individual methods on the Plugin class (Plugin#enable, Plugin#disable) and use that when registering and parsing options. If the plugin is off-by-default, Plugin#register_options will use Plugin#disable. When parsing options via Plugin#provide_options, if the plugin has been specified in --enable-extensions then it will be re-enabled via Plugin#enable. --- flake8/main/cli.py | 4 ++-- flake8/options/manager.py | 15 +++++++++++++++ flake8/plugins/manager.py | 13 ++++++++++++- tests/unit/test_plugin.py | 7 +++++-- 4 files changed, 34 insertions(+), 5 deletions(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index ef9b328..e9eedb7 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -34,7 +34,7 @@ def register_default_options(option_manager): - ``--disable-noqa`` - ``--show-source`` - ``--statistics`` - - ``--enabled-extensions`` + - ``--enable-extensions`` - ``--exit-zero`` - ``-j``/``--jobs`` - ``--output-file`` @@ -141,7 +141,7 @@ def register_default_options(option_manager): # Flake8 options add_option( - '--enabled-extensions', default='', parse_from_config=True, + '--enable-extensions', default='', parse_from_config=True, comma_separated_list=True, type='string', help='Enable plugins and extensions that are otherwise disabled ' 'by default', diff --git a/flake8/options/manager.py b/flake8/options/manager.py index bf144c9..439cba2 100644 --- a/flake8/options/manager.py +++ b/flake8/options/manager.py @@ -189,6 +189,21 @@ class OptionManager(object): self.config_options_dict[option.config_name] = option LOG.debug('Registered option "%s".', option) + def remove_from_default_ignore(self, error_codes): + """Remove specified error codes from the default ignore list. + + :param list error_codes: + List of strings that are the error/warning codes to attempt to + remove from the extended default ignore list. + """ + LOG.debug('Removing %r from the default ignore list', error_codes) + for error_code in error_codes: + try: + self.extend_default_ignore.remove(error_code) + except ValueError: + LOG.debug('Attempted to remove %s from default ignore' + ' but it was not a member of the list.', error_code) + def extend_default_ignore(self, error_codes): """Extend the default ignore list with the error codes provided. diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index fe51391..dffc2f6 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -158,6 +158,14 @@ class Plugin(object): LOG.critical(str(failed_to_load)) raise failed_to_load + def enable(self, optmanager): + """Remove plugin name from the default ignore list.""" + optmanager.remove_from_default_ignore([self.name]) + + def disable(self, optmanager): + """Add the plugin name to the default ignore list.""" + optmanager.extend_default_ignore([self.name]) + def provide_options(self, optmanager, options, extra_args): """Pass the parsed options and extra arguments to the plugin.""" parse_options = getattr(self.plugin, 'parse_options', None) @@ -168,6 +176,9 @@ class Plugin(object): except TypeError: parse_options(options) + if self.name in options.enable_extensions: + self.enable(optmanager) + def register_options(self, optmanager): """Register the plugin's command-line options on the OptionManager. @@ -187,7 +198,7 @@ class Plugin(object): add_options(optmanager) if self.off_by_default: - optmanager.extend_default_ignore([self.name]) + self.disable(optmanager) class PluginManager(object): # pylint: disable=too-few-public-methods diff --git a/tests/unit/test_plugin.py b/tests/unit/test_plugin.py index e995c7b..d3d1185 100644 --- a/tests/unit/test_plugin.py +++ b/tests/unit/test_plugin.py @@ -1,4 +1,6 @@ """Tests for flake8.plugins.manager.Plugin.""" +import optparse + from flake8 import exceptions from flake8.plugins import manager @@ -138,14 +140,15 @@ def test_provide_options(): entry_point = mock.Mock(spec=['require', 'resolve', 'load']) plugin_obj = mock.Mock(spec_set=['name', 'version', 'add_options', 'parse_options']) + option_values = optparse.Values({'enable_extensions': []}) option_manager = mock.Mock() plugin = manager.Plugin('T000', entry_point) plugin._plugin = plugin_obj # Call the method we're testing. - plugin.provide_options(option_manager, 'options', None) + plugin.provide_options(option_manager, option_values, None) # Assert that we call add_options plugin_obj.parse_options.assert_called_once_with( - option_manager, 'options', None + option_manager, option_values, None ) From 466ef2e5966d2b36a11dbf205a7bf3835f427d70 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 28 May 2016 12:00:47 -0500 Subject: [PATCH 262/364] Add example configuration sections to the docs Add tests to verify our examples do not regress --- docs/source/user/configuration.rst | 125 +++++++++++++++++- .../cli-specified-with-inline-comments.ini | 16 +++ .../cli-specified-without-inline-comments.ini | 16 +++ tests/unit/test_merged_config_parser.py | 33 +++++ 4 files changed, 189 insertions(+), 1 deletion(-) create mode 100644 tests/fixtures/config_files/cli-specified-with-inline-comments.ini create mode 100644 tests/fixtures/config_files/cli-specified-without-inline-comments.ini diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst index 1e51e0a..ff671a6 100644 --- a/docs/source/user/configuration.rst +++ b/docs/source/user/configuration.rst @@ -75,4 +75,127 @@ sub-directories. Let's take for example Flake8's own project structure └── unit In the top-level ``flake8`` directory (which contains ``docs``, ``flake8``, -and ``tests``) there's also ``tox.ini`` and ``setup.cfg`` files. +and ``tests``) there's also ``tox.ini`` and ``setup.cfg`` files. In our case, +we keep our Flake8 configuration in ``tox.ini``. Regardless of whether you +keep your config in ``.flake8``, ``setup.cfg``, or ``tox.ini`` we expect you +to use INI to configure Flake8 (since each of these files already uses INI +as a format). This means that any Flake8 configuration you wish to set needs +to be in the ``flake8`` section, which means it needs to start like so: + +.. code-block:: ini + + [flake8] + +Each command-line option that you want to specify in your config file can +be named in either of two ways: + +#. Using underscores (``_``) instead of hyphens (``-``) + +#. Simply using hyphens (without the leading hyphens) + +So let's actually look at Flake8's own configuration section: + +.. code-block:: ini + + [flake8] + ignore = D203 + exclude = .git,__pycache__,docs/source/conf.py,old,build,dist + max-complexity = 10 + +This is equivalent to: + +.. prompt:: bash + + flake8 --ignore D203 \ + --exclude .git,__pycache__,docs/source/conf.py,old,build,dist \ + --max-complexity 10 + +In our case, if we wanted to, we could also do + +.. code-block:: ini + + [flake8] + ignore = D203 + exclude = + .git, + __pycache__, + docs/source/conf.py, + old, + build, + dist + max-complexity = 10 + +This would allow us to add comments for why we're excluding items, e.g., + +.. code-block:: ini + + [flake8] + ignore = D203 + exclude = + # No need to traverse our git directory + .git, + # There's no value in checking cache directories + __pycache__, + # The conf file is mostly autogenerated, ignore it + docs/source/conf.py, + # The old directory contains Flake8 2.0 + old, + # This contains our built documentation + build, + # This contains builds of flake8 that we don't want to check + dist + max-complexity = 10 + +This is also useful if you have a long list of error codes to ignore. Let's +look at a portion of OpenStack's Swift project configuration: + +.. code-block:: ini + + [flake8] + # it's not a bug that we aren't using all of hacking, ignore: + # F812: list comprehension redefines ... + # H101: Use TODO(NAME) + # H202: assertRaises Exception too broad + # H233: Python 3.x incompatible use of print operator + # H301: one import per line + # H306: imports not in alphabetical order (time, os) + # H401: docstring should not start with a space + # H403: multi line docstrings should end on a new line + # H404: multi line docstring should start without a leading new line + # H405: multi line docstring summary not separated with an empty line + # H501: Do not use self.__dict__ for string formatting + ignore = F812,H101,H202,H233,H301,H306,H401,H403,H404,H405,H501 + +They use the comments to describe the check but they could also write this as: + +.. code-block:: ini + + [flake8] + # it's not a bug that we aren't using all of hacking + ignore = + # F812: list comprehension redefines ... + F812, + # H101: Use TODO(NAME) + H101, + # H202: assertRaises Exception too broad + H202, + # H233: Python 3.x incompatible use of print operator + H233, + # H301: one import per line + H301, + # H306: imports not in alphabetical order (time, os) + H306, + # H401: docstring should not start with a space + H401, + # H403: multi line docstrings should end on a new line + H403, + # H404: multi line docstring should start without a leading new line + H404, + # H405: multi line docstring summary not separated with an empty line + H405, + # H501: Do not use self.__dict__ for string formatting + H501 + +Or they could use each comment to describe **why** they've ignored the check. +:program:`Flake8` knows how to parse these lists and will appropriatey handle +these situations. diff --git a/tests/fixtures/config_files/cli-specified-with-inline-comments.ini b/tests/fixtures/config_files/cli-specified-with-inline-comments.ini new file mode 100644 index 0000000..4d57e85 --- /dev/null +++ b/tests/fixtures/config_files/cli-specified-with-inline-comments.ini @@ -0,0 +1,16 @@ +[flake8] +# This is a flake8 config, there are many like it, but this is mine +ignore = + # Disable E123 + E123, + # Disable W234 + W234, + # Also disable E111 + E111 +exclude = + # Exclude foo/ + foo/, + # Exclude bar/ while we're at it + bar/, + # Exclude bogus/ + bogus/ diff --git a/tests/fixtures/config_files/cli-specified-without-inline-comments.ini b/tests/fixtures/config_files/cli-specified-without-inline-comments.ini new file mode 100644 index 0000000..f50ba75 --- /dev/null +++ b/tests/fixtures/config_files/cli-specified-without-inline-comments.ini @@ -0,0 +1,16 @@ +[flake8] +# This is a flake8 config, there are many like it, but this is mine +# Disable E123 +# Disable W234 +# Also disable E111 +ignore = + E123, + W234, + E111 +# Exclude foo/ +# Exclude bar/ while we're at it +# Exclude bogus/ +exclude = + foo/, + bar/, + bogus/ diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index c64cae6..baaa57a 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -172,3 +172,36 @@ def test_parse_uses_cli_config(ConfigFileManager, optmanager): parser.parse(cli_config='foo.ini') parser.config_finder.cli_config.assert_called_once_with('foo.ini') + + +@pytest.mark.parametrize('config_fixture_path', [ + 'tests/fixtures/config_files/cli-specified.ini', + 'tests/fixtures/config_files/cli-specified-with-inline-comments.ini', + 'tests/fixtures/config_files/cli-specified-without-inline-comments.ini', +]) +def test_parsed_configs_are_equivalent(optmanager, config_fixture_path): + """Verify the each file matches the expected parsed output. + + This is used to ensure our documented behaviour does not regress. + """ + optmanager.add_option('--exclude', parse_from_config=True, + comma_separated_list=True, + normalize_paths=True) + optmanager.add_option('--ignore', parse_from_config=True, + comma_separated_list=True) + parser = config.MergedConfigParser(optmanager) + config_finder = parser.config_finder + + with mock.patch.object(config_finder, 'local_config_files') as localcfs: + localcfs.return_value = [config_fixture_path] + with mock.patch.object(config_finder, + 'user_config_file') as usercf: + usercf.return_value = [] + parsed_config = parser.merge_user_and_local_config() + + assert parsed_config['ignore'] == ['E123', 'W234', 'E111'] + assert parsed_config['exclude'] == [ + os.path.abspath('foo/'), + os.path.abspath('bar/'), + os.path.abspath('bogus/'), + ] From 89c9ec795ee2905638ab23947ae4360adce0d710 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 28 May 2016 12:20:46 -0500 Subject: [PATCH 263/364] Start adding descriptions of options --- docs/source/user/configuration.rst | 15 ++++- docs/source/user/index.rst | 1 + docs/source/user/options.rst | 90 ++++++++++++++++++++++++++++++ 3 files changed, 105 insertions(+), 1 deletion(-) create mode 100644 docs/source/user/options.rst diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst index ff671a6..eb9054d 100644 --- a/docs/source/user/configuration.rst +++ b/docs/source/user/configuration.rst @@ -45,6 +45,9 @@ The user configuration file is expected to be stored somewhere in the user's Note that Flake8 looks for ``~\.flake8`` on Windows and ``~/.config/flake8`` on Linux and other Unix systems. +User configuration files use the same syntax as Project Configuration files. +Keep reading to see that syntax. + Project Configuration --------------------- @@ -93,7 +96,13 @@ be named in either of two ways: #. Simply using hyphens (without the leading hyphens) -So let's actually look at Flake8's own configuration section: +.. note:: + + Not every Flake8 command-line option can be specified in the configuration + file. See :ref:`our list of options ` to determine which + options will be parsed from the configuration files. + +Let's actually look at Flake8's own configuration section: .. code-block:: ini @@ -146,6 +155,10 @@ This would allow us to add comments for why we're excluding items, e.g., dist max-complexity = 10 +.. note:: + + You can also specify ``--max-complexity`` as ``max_complexity = 10``. + This is also useful if you have a long list of error codes to ignore. Let's look at a portion of OpenStack's Swift project configuration: diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 2085178..8d80402 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -22,6 +22,7 @@ This guide will cover all of these and the nuances for using Flake8. invocation configuration + options .. config files .. command-line tutorial diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst new file mode 100644 index 0000000..5a0d15f --- /dev/null +++ b/docs/source/user/options.rst @@ -0,0 +1,90 @@ +.. _options-list: + +================================================ + Full Listing of Options and Their Descriptions +================================================ + +.. + -v, --verbose Print more information about what is happening in + flake8. This option is repeatable and will increase + verbosity each time it is repeated. + -q, --quiet Report only file names, or nothing. This option is + repeatable. + --count Print total number of errors and warnings to standard + error and set the exit code to 1 if total is not + empty. + --diff Report changes only within line number ranges in the + unified diff provided on standard in by the user. + --exclude=patterns Comma-separated list of files or directories to + exclude.(Default: + .svn,CVS,.bzr,.hg,.git,__pycache__,.tox) + --filename=patterns Only check for filenames matching the patterns in this + comma-separated list. (Default: *.py) + --format=format Format errors according to the chosen formatter. + --hang-closing Hang closing bracket instead of matching indentation + of opening bracket's line. + --ignore=errors Comma-separated list of errors and warnings to ignore + (or skip). For example, ``--ignore=E4,E51,W234``. + (Default: E121,E123,E126,E226,E24,E704) + --max-line-length=n Maximum allowed line length for the entirety of this + run. (Default: 79) + --select=errors Comma-separated list of errors and warnings to enable. + For example, ``--select=E4,E51,W234``. (Default: ) + --disable-noqa Disable the effect of "# noqa". This will report + errors on lines with "# noqa" at the end. + --show-source Show the source generate each error or warning. + --statistics Count errors and warnings. + --enable-extensions=ENABLE_EXTENSIONS + Enable plugins and extensions that are otherwise + disabled by default + --exit-zero Exit with status code "0" even if there are errors. + -j JOBS, --jobs=JOBS Number of subprocesses to use to run checks in + parallel. This is ignored on Windows. The default, + "auto", will auto-detect the number of processors + available to use. (Default: auto) + --output-file=OUTPUT_FILE + Redirect report to a file. + --append-config=APPEND_CONFIG + Provide extra config files to parse in addition to the + files found by Flake8 by default. These files are the + last ones read and so they take the highest precedence + when multiple files provide the same option. + --config=CONFIG Path to the config file that will be the authoritative + config source. This will cause Flake8 to ignore all + other configuration files. + --isolated Ignore all found configuration files. + --builtins=BUILTINS define more built-ins, comma separated + --doctests check syntax of the doctests + --include-in-doctest=INCLUDE_IN_DOCTEST + Run doctests only on these files + --exclude-from-doctest=EXCLUDE_FROM_DOCTEST + Skip these files when running doctests + +.. option:: --version + + When specified on the command-line, this will show :program:`Flake8`\ 's + version as well as the versions of all plugins installed. + + **This cannot be specified in the config files.** + + +.. option:: -h, --help + + When specified on the command-line, this will show a description of how + to use :program:`Flake8` and it soptions. + + **This cannot be specified in the config files.** + +.. option:: -v, --verbose + + When specified on the command-line or in configuration, this will + increase the verbosity of Flake8's output. Each time you specify + it, it will print more and more information. + + **This can be specified in the config file.** + + Example config file specification: + + .. code-block:: ini + + verbose = 2 From 2fcb222ff9895413610a5e6e087c582cf410abb9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 28 May 2016 21:09:05 -0500 Subject: [PATCH 264/364] Fix default printing for --exclude help text --- flake8/main/cli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index e9eedb7..455629f 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -75,7 +75,7 @@ def register_default_options(option_manager): comma_separated_list=True, parse_from_config=True, normalize_paths=True, help='Comma-separated list of files or directories to exclude.' - '(Default: %default)', + ' (Default: %default)', ) add_option( From 4c3d176f9ac952d17ebc0391624ced4df02998dc Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 29 May 2016 20:56:30 -0500 Subject: [PATCH 265/364] Add a first pass at documenting cli options --- docs/source/user/options.rst | 669 +++++++++++++++++++++++++++++++---- 1 file changed, 604 insertions(+), 65 deletions(-) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index 5a0d15f..ba897b7 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -4,87 +4,626 @@ Full Listing of Options and Their Descriptions ================================================ -.. - -v, --verbose Print more information about what is happening in - flake8. This option is repeatable and will increase - verbosity each time it is repeated. - -q, --quiet Report only file names, or nothing. This option is - repeatable. - --count Print total number of errors and warnings to standard - error and set the exit code to 1 if total is not - empty. - --diff Report changes only within line number ranges in the - unified diff provided on standard in by the user. - --exclude=patterns Comma-separated list of files or directories to - exclude.(Default: - .svn,CVS,.bzr,.hg,.git,__pycache__,.tox) - --filename=patterns Only check for filenames matching the patterns in this - comma-separated list. (Default: *.py) - --format=format Format errors according to the chosen formatter. - --hang-closing Hang closing bracket instead of matching indentation - of opening bracket's line. - --ignore=errors Comma-separated list of errors and warnings to ignore - (or skip). For example, ``--ignore=E4,E51,W234``. - (Default: E121,E123,E126,E226,E24,E704) - --max-line-length=n Maximum allowed line length for the entirety of this - run. (Default: 79) - --select=errors Comma-separated list of errors and warnings to enable. - For example, ``--select=E4,E51,W234``. (Default: ) - --disable-noqa Disable the effect of "# noqa". This will report - errors on lines with "# noqa" at the end. - --show-source Show the source generate each error or warning. - --statistics Count errors and warnings. - --enable-extensions=ENABLE_EXTENSIONS - Enable plugins and extensions that are otherwise - disabled by default - --exit-zero Exit with status code "0" even if there are errors. - -j JOBS, --jobs=JOBS Number of subprocesses to use to run checks in - parallel. This is ignored on Windows. The default, - "auto", will auto-detect the number of processors - available to use. (Default: auto) - --output-file=OUTPUT_FILE - Redirect report to a file. - --append-config=APPEND_CONFIG - Provide extra config files to parse in addition to the - files found by Flake8 by default. These files are the - last ones read and so they take the highest precedence - when multiple files provide the same option. - --config=CONFIG Path to the config file that will be the authoritative - config source. This will cause Flake8 to ignore all - other configuration files. - --isolated Ignore all found configuration files. - --builtins=BUILTINS define more built-ins, comma separated - --doctests check syntax of the doctests - --include-in-doctest=INCLUDE_IN_DOCTEST - Run doctests only on these files - --exclude-from-doctest=EXCLUDE_FROM_DOCTEST - Skip these files when running doctests +.. program:: flake8 .. option:: --version - When specified on the command-line, this will show :program:`Flake8`\ 's - version as well as the versions of all plugins installed. + Show :program:`Flake8`\ 's version as well as the versions of all plugins + installed. - **This cannot be specified in the config files.** + Command-line usage: + + .. prompt:: bash + + flake8 --version + + This **can not** be specified in the config files. .. option:: -h, --help - When specified on the command-line, this will show a description of how - to use :program:`Flake8` and it soptions. + Show a description of how to use :program:`Flake8` and its options. + + Command-line usage: + + .. prompt:: bash + + flake8 --help + flake8 -h + + This **can not** be specified in the config files. - **This cannot be specified in the config files.** .. option:: -v, --verbose - When specified on the command-line or in configuration, this will - increase the verbosity of Flake8's output. Each time you specify + Increase the verbosity of Flake8's output. Each time you specify it, it will print more and more information. - **This can be specified in the config file.** + Command-line example: - Example config file specification: + .. prompt:: bash + + flake8 -vv + + This **can** be specified in the config file. + + Example config file usage: .. code-block:: ini verbose = 2 + + +.. option:: -q, --quiet + + Decrease the verbosity of Flake8's output. Each time you specify it, + it will print less and less information. + + Command-line example: + + .. prompt:: bash + + flake8 -q + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + quiet = 1 + + +.. option:: --count + + Print the total number of errors. + + Command-line example: + + .. prompt:: bash + + flake8 --count dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + count = True + + +.. option:: --diff + + Use the unified diff provided on standard in to only check the modified + files and report errors included in the diff. + + Command-line example: + + .. prompt:: bash + + git diff -u | flake8 --diff + + This **can not** be specified in the config file. + + +.. option:: --exclude= + + Provide a comma-separated list of glob patterns to exclude from checks. + + This defaults to: ``.svn,CVS,.bzr,.hg,.git,__pycache__,.tox`` + + Example patterns: + + - ``*.pyc`` will match any file that ends with ``.pyc`` + + - ``__pycache__`` will match any path that has ``__pycache__`` in it + + - ``lib/python`` will look expand that using :func:`os.path.abspath` and + look for matching paths + + Command-line example: + + .. prompt:: bash + + flake8 --exclude=*.pyc dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + exclude = + .tox, + __pycache__ + + +.. option:: --filename= + + Provide a comma-separate list of glob patterns to include for checks. + + This defaults to: ``*.py`` + + Example patterns: + + - ``*.py`` will match any file that ends with ``.py`` + + - ``__pycache__`` will match any path that has ``__pycache__`` in it + + - ``lib/python`` will look expand that using :func:`os.path.abspath` and + look for matching paths + + Command-line example: + + .. prompt:: bash + + flake8 --filename=*.py dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + filename = + example.py, + another-example*.py + + +.. option:: --format= + + Select the formatter used to display errors to the user. + + This defaults to: ``default`` + + By default, there are two formatters available: + + - default + + - pylint + + Other formatters can be installed. Refer to their documentation for the + name to use to select them. + + Command-line example: + + .. prompt:: bash + + flake8 --format=pylint dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + format=pylint + + +.. option:: --hang-closing + + Toggle whether pycodestyle should enforce matching the indentation of the + opening bracket's line. When you specify this, it will prefer that you + hang the closing bracket rather than match the indentation. + + Command-line example: + + .. prompt:: bash + + flake8 --hang-closing dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + hang_closing = True + hang-closing = True + + +.. option:: --ignore= + + Specify a list of codes to ignore. The list is expected to be + comma-separated, and does not need to specify an error code exactly. + Since Flake8 3.0, this **can** be combined with :option:`--select`. See + :option:`--select` for more information. + + For example, if you wish to only ignore ``W234``, then you can specify + that. But if you want to ignore all codes that start with ``W23`` you + need only specify ``W23`` to ignore them. This also works for ``W2`` and + ``W`` (for example). + + This defaults to: ``E121,E123,E126,E226,E24,E704`` + + Command-line example: + + .. prompt:: bash + + flake8 --ignore=E121,E123 dir/ + flake8 --ignore=E24,E704 dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + ignore = + E121, + E123 + ignore = E121,E123 + + +.. option:: --max-line-length= + + Set the maximum length that any line (with some exceptions) may be. + + Exceptions include lines that are either strings or comments which are + entirely URLs. For example: + + .. code-block:: python + + # https://some-super-long-domain-name.com/with/some/very/long/path + + url = ( + 'http://...' + ) + + This defaults to: 79 + + Command-line example: + + .. prompt:: bash + + flake8 --max-line-length 99 dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + max-line-length = 79 + + +.. option:: --select= + + Specify the list of error codes you wish Flake8 to report. Similarly to + :option:`--ignore`. You can specify a portion of an error code to get all + that start with that string. For example, you can use ``E``, ``E4``, + ``E43``, and ``E431``. + + This has no default value. + + Command-line example: + + .. prompt:: bash + + flake8 --select=E431,E5,W,F dir/ + flake8 --select=E,W dir/ + + This can also be combined with :option:`--ignore`: + + .. prompt:: bash + + flake8 --select=E --ignore=E432 dir/ + + This will report all codes that start with ``E``, but ignore ``E432`` + specifically. This is more flexibly than the Flake8 2.x and 1.x used + to be. + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + select = + E431, + W, + F + + +.. option:: --disable-noqa + + Report all errors, even if it is on the same line as a ``# NOQA`` comment. + ``# NOQA`` can be used to silence messages on specific lines. Sometimes, + users will want to see what errors are being silenced without editing the + file. This option allows you to see all the warnings, errors, etc. + reported. + + Command-line example: + + .. prompt:: bash + + flake8 --disable-noqa dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + disable_noqa = True + disable-noqa = True + + +.. option:: --show-source + + Print the source code generating the error/warning in question. + + Command-line example: + + .. prompt:: bash + + flake8 --show-source dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + show_source = True + show-source = True + + +.. option:: --statistics + + Count the number of occurrences of each error/warning code and + print a report. + + Command-line example: + + .. prompt:: bash + + flake8 --statistics + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + statistics = True + + +.. option:: --enable-extensions= + + Enable off-by-default extensions. + + Plugins to Flake8 have the option of registering themselves as + off-by-default. These plugins effectively add themselves to the + default ignore list. + + Command-line example: + + .. prompt:: bash + + flake8 --enable-extensions=H111 dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + enable-extensions = + H111, + G123 + enable_extensions = + H111, + G123 + + +.. option:: --exit-zero + + Force Flake8 to use the exit status code 0 even if there are errors. + + By default Flake8 will exit with a non-zero integer if there are errors. + + Command-line example: + + .. prompt:: bash + + flake8 --exit-zero dir/ + + This **can not** be specified in the config file. + + +.. option:: --jobs= + + Specify the number of subprocesses that Flake8 will use to run checks in + parallel. + + .. note:: + + This option is ignored on Windows because :mod:`multiprocessing` does + not support Windows across all supported versions of Python. + + This defaults to: ``auto`` + + The default behaviour will use the number of CPUs on your machine as + reported by :func:`multiprocessing.cpu_count`. + + Command-line example: + + .. prompt:: bash + + flake8 --jobs=8 dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + jobs = 8 + + +.. option:: --output-file= + + Redirect all output to the specified file. + + Command-line example: + + .. prompt:: bash + + flake8 --output-file=output.txt dir/ + flake8 -vv --output-file=output.txt dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + output-file = output.txt + output_file = output.txt + + +.. option:: --append-config= + + Provide extra config files to parse in after and in addition to the files + that Flake8 found on its own. Since these files are the last ones read + into the Configuration Parser, so it has the highest precedence if it + provides an option specified in another config file. + + Command-line example: + + .. prompt:: bash + + flake8 --append-config=my-extra-config.ini dir/ + + This **can not** be specified in the config file. + + +.. option:: --config= + + Provide a path to a config file that will be the only config file read and + used. This will cause Flake8 to ignore all other config files that exist. + + Command-line example: + + .. prompt:: bash + + flake8 --config=my-only-config.ini dir/ + + This **can not** be specified in the config file. + + +.. option:: --isolated + + Ignore any config files and use Flake8 as if there were no config files + found. + + Command-line example: + + .. prompt:: bash + + flake8 --isolated dir/ + + This **can not** be specified in the config file. + + +.. option:: --builtins= + + Provide a custom list of builtin functions, objects, names, etc. + + This allows you to let pyflakes know about builtins that it may + not immediately recognize so it does not report warnings for using + an undefined name. + + This is registered by the default PyFlakes plugin. + + Command-line example: + + .. prompt:: bash + + flake8 --builtins=_,_LE,_LW dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + builtins = + _, + _LE, + _LW + + +.. option:: --doctests + + Enable PyFlakes syntax checking of doctests in docstrings. + + This is registered by the default PyFlakes plugin. + + Command-line example: + + .. prompt:: bash + + flake8 --doctests dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + doctests = True + + +.. option:: --include-in-doctest= + + Specify which files are checked by PyFlakes for doctest syntax. + + This is registered by the default PyFlakes plugin. + + Command-line example: + + .. prompt:: bash + + flake8 --include-in-doctest=dir/subdir/file.py,dir/other/file.py dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + include-in-doctest = + dir/subdir/file.py, + dir/other/file.py + include_in_doctest = + dir/subdir/file.py, + dir/other/file.py + + +.. option:: --exclude-from-doctest= + + Specify which files are not to be checked by PyFlakes for doctest syntax. + + This is registered by the default PyFlakes plugin. + + Command-line example: + + .. prompt:: bash + + flake8 --exclude-in-doctest=dir/subdir/file.py,dir/other/file.py dir/ + + This **can** be specified in the config file. + + Example config file usage: + + .. code-block:: ini + + exclude-in-doctest = + dir/subdir/file.py, + dir/other/file.py + exclude_in_doctest = + dir/subdir/file.py, + dir/other/file.py From e9ddf7533b7bbe142a5450c41f667fb1115b9262 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 29 May 2016 21:04:03 -0500 Subject: [PATCH 266/364] Add template in comments for new options Also ensure that existing documentation is consistent with the template --- docs/source/user/options.rst | 75 +++++++++++++++++++++++------------- 1 file changed, 48 insertions(+), 27 deletions(-) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index ba897b7..ce30bdf 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -4,6 +4,27 @@ Full Listing of Options and Their Descriptions ================================================ +.. + NOTE(sigmavirus24): When adding new options here, please follow the + following _rough_ template: + + .. option:: --[=] + + Active description of option's purpose (note that each description + starts with an active verb) + + Command-line usage: + + .. prompt:: bash + + flake8 --[=] [positional params] + + This **can[ not]** be specified in config files. + + (If it can be, an example using .. code-block:: ini) + + Thank you for your contribution to Flake8's documentation. + .. program:: flake8 .. option:: --version @@ -17,7 +38,7 @@ flake8 --version - This **can not** be specified in the config files. + This **can not** be specified in config files. .. option:: -h, --help @@ -31,7 +52,7 @@ flake8 --help flake8 -h - This **can not** be specified in the config files. + This **can not** be specified in config files. .. option:: -v, --verbose @@ -45,7 +66,7 @@ flake8 -vv - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -65,7 +86,7 @@ flake8 -q - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -84,7 +105,7 @@ flake8 --count dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -104,7 +125,7 @@ git diff -u | flake8 --diff - This **can not** be specified in the config file. + This **can not** be specified in config files. .. option:: --exclude= @@ -128,7 +149,7 @@ flake8 --exclude=*.pyc dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -160,7 +181,7 @@ flake8 --filename=*.py dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -192,7 +213,7 @@ flake8 --format=pylint dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -213,7 +234,7 @@ flake8 --hang-closing dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -244,7 +265,7 @@ flake8 --ignore=E121,E123 dir/ flake8 --ignore=E24,E704 dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -279,7 +300,7 @@ flake8 --max-line-length 99 dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -314,7 +335,7 @@ specifically. This is more flexibly than the Flake8 2.x and 1.x used to be. - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -340,7 +361,7 @@ flake8 --disable-noqa dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -360,7 +381,7 @@ flake8 --show-source dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -381,7 +402,7 @@ flake8 --statistics - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -404,7 +425,7 @@ flake8 --enable-extensions=H111 dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -430,7 +451,7 @@ flake8 --exit-zero dir/ - This **can not** be specified in the config file. + This **can not** be specified in config files. .. option:: --jobs= @@ -454,7 +475,7 @@ flake8 --jobs=8 dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -474,7 +495,7 @@ flake8 --output-file=output.txt dir/ flake8 -vv --output-file=output.txt dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -497,7 +518,7 @@ flake8 --append-config=my-extra-config.ini dir/ - This **can not** be specified in the config file. + This **can not** be specified in config files. .. option:: --config= @@ -511,7 +532,7 @@ flake8 --config=my-only-config.ini dir/ - This **can not** be specified in the config file. + This **can not** be specified in config files. .. option:: --isolated @@ -525,7 +546,7 @@ flake8 --isolated dir/ - This **can not** be specified in the config file. + This **can not** be specified in config files. .. option:: --builtins= @@ -544,7 +565,7 @@ flake8 --builtins=_,_LE,_LW dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -568,7 +589,7 @@ flake8 --doctests dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -589,7 +610,7 @@ flake8 --include-in-doctest=dir/subdir/file.py,dir/other/file.py dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: @@ -615,7 +636,7 @@ flake8 --exclude-in-doctest=dir/subdir/file.py,dir/other/file.py dir/ - This **can** be specified in the config file. + This **can** be specified in config files. Example config file usage: From 33f982b446aa1698dcc53d3cd6277f387c79b7a3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 30 May 2016 12:44:58 -0500 Subject: [PATCH 267/364] Rely on Python 3.4 backport of configparser Python 2.7's ConfigParser module does not allow for the behaviour we have documented for config files in Flake8 3.0. To compensate for that, we add a dependency on the configparser backport on PyPI for Python 2.7 --- docs/source/user/configuration.rst | 8 ++++++++ flake8/options/config.py | 6 +----- setup.cfg | 1 + setup.py | 2 +- tests/unit/test_config_file_finder.py | 6 +----- tox.ini | 1 + 6 files changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst index eb9054d..6a176ec 100644 --- a/docs/source/user/configuration.rst +++ b/docs/source/user/configuration.rst @@ -155,6 +155,14 @@ This would allow us to add comments for why we're excluding items, e.g., dist max-complexity = 10 +.. note:: + + If you're using Python 2, you will notice that we download the + :mod:`configparser` backport from PyPI. That backport enables us to + support this behaviour on all supported versions of Python. + + Please do **not** open issues about this dependency. + .. note:: You can also specify ``--max-complexity`` as ``max_complexity = 10``. diff --git a/flake8/options/config.py b/flake8/options/config.py index ecc40f7..9bd7e19 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -1,13 +1,9 @@ """Config handling logic for Flake8.""" +import configparser import logging import os.path import sys -if sys.version_info < (3, 0): - import ConfigParser as configparser -else: - import configparser - LOG = logging.getLogger(__name__) __all__ = ('ConfigFileFinder', 'MergedConfigParser') diff --git a/setup.cfg b/setup.cfg index 8af25a3..831bd62 100644 --- a/setup.cfg +++ b/setup.cfg @@ -7,6 +7,7 @@ universal=1 [metadata] requires-dist = enum34; python_version<"3.4" + configparser; python_version<"3.2" [pytest] norecursedirs = .git .* *.egg* old docs dist build diff --git a/setup.py b/setup.py index c090263..8e789d0 100644 --- a/setup.py +++ b/setup.py @@ -31,7 +31,7 @@ if mock is None: requires = [ "pyflakes >= 0.8.1, < 1.1", "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", - # "mccabe >= 0.2.1, < 0.4", + "mccabe >= 0.2.1, < 0.4", ] if sys.version_info < (3, 4): diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 58009f7..8d7e920 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -1,9 +1,5 @@ """Tests for the ConfigFileFinder.""" -try: - import ConfigParser as configparser -except ImportError: - import configparser - +import configparser import os import sys diff --git a/tox.ini b/tox.ini index 24ef6e8..45514ea 100644 --- a/tox.ini +++ b/tox.ini @@ -6,6 +6,7 @@ envlist = py27,py33,py34,py35,flake8 deps = mock pytest + py27: configparser commands = py.test {posargs} From df0a799c943fc53588b145730c7c43ea99cfb540 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 30 May 2016 12:58:15 -0500 Subject: [PATCH 268/364] Add configparser to pypy testenv --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 45514ea..01d6637 100644 --- a/tox.ini +++ b/tox.ini @@ -6,7 +6,7 @@ envlist = py27,py33,py34,py35,flake8 deps = mock pytest - py27: configparser + pypy,py27: configparser commands = py.test {posargs} From f54c0874f8ae6783f54b71a4969f21ce2bcbfdcd Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 30 May 2016 16:00:45 -0500 Subject: [PATCH 269/364] Handle sdist installation without tox --- setup.py | 3 +++ tox.ini | 1 - 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 8e789d0..beadf73 100644 --- a/setup.py +++ b/setup.py @@ -37,6 +37,9 @@ requires = [ if sys.version_info < (3, 4): requires.append("enum34") +if sys.version_info < (3, 2): + requires.append("configparser") + def get_long_description(): """Generate a long description from the README and CHANGES files.""" diff --git a/tox.ini b/tox.ini index 01d6637..24ef6e8 100644 --- a/tox.ini +++ b/tox.ini @@ -6,7 +6,6 @@ envlist = py27,py33,py34,py35,flake8 deps = mock pytest - pypy,py27: configparser commands = py.test {posargs} From 8300e0f97ce6e2dba00d17e82bd15cc23f4cae06 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 30 May 2016 16:00:59 -0500 Subject: [PATCH 270/364] Update dependency version constraints --- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.py b/setup.py index beadf73..9fa91fd 100644 --- a/setup.py +++ b/setup.py @@ -29,9 +29,9 @@ if mock is None: requires = [ - "pyflakes >= 0.8.1, < 1.1", + "pyflakes >= 0.8.1, != 1.2.0, != 1.2.1, != 1.2.2", "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", - "mccabe >= 0.2.1, < 0.4", + "mccabe >= 0.5.0", ] if sys.version_info < (3, 4): From 467672fc5c5d9133155629cf1fe7694cc9427b0b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 30 May 2016 16:43:11 -0500 Subject: [PATCH 271/364] Refactor Error formatting and handling This allows us to handle --show-source in our formatters by default. This also adds the physical line information to the Error class instead of passing it to is_inline_ignored. This allows us to avoid using linecache in our formatters. --- flake8/formatting/base.py | 38 ++++++++++++++++++++++++++++------ flake8/style_guide.py | 24 ++++++++++++++------- tests/unit/test_style_guide.py | 9 +++++--- 3 files changed, 54 insertions(+), 17 deletions(-) diff --git a/flake8/formatting/base.py b/flake8/formatting/base.py index e419470..ecc5973 100644 --- a/flake8/formatting/base.py +++ b/flake8/formatting/base.py @@ -55,8 +55,9 @@ class BaseFormatter(object): def handle(self, error): """Handle an error reported by Flake8. - This defaults to calling :meth:`format` and then :meth:`write`. To - extend how errors are handled, override this method. + This defaults to calling :meth:`format`, :meth:`format_source`, and + then :meth:`write`. To extend how errors are handled, override this + method. :param error: This will be an instance of :class:`~flake8.style_guide.Error`. @@ -64,7 +65,8 @@ class BaseFormatter(object): flake8.style_guide.Error """ line = self.format(error) - self.write(line) + source = self.format_source(error) + self.write(line, source) def format(self, error): """Format an error reported by Flake8. @@ -83,7 +85,26 @@ class BaseFormatter(object): raise NotImplementedError('Subclass of BaseFormatter did not implement' ' format.') - def write(self, line): + def format_source(self, error): + """Format the physical line generating the error. + + :param error: + This will be an instance of :class:`~flake8.style_guide.Error`. + :returns: + The formatted error string if the user wants to show the source. + If the user does not want to show the source, this will return + ``None``. + :rtype: + str + """ + if not self.options.show_source: + return None + pointer = (' ' * error.column_number) + '^' + # Physical lines have a newline at the end, no need to add an extra + # one + return error.physical_line + pointer + + def write(self, line, source): """Write the line either to the output file or stdout. This handles deciding whether to write to a file or print to standard @@ -94,9 +115,14 @@ class BaseFormatter(object): The formatted string to print or write. """ if self.output_fd is not None: - self.output_fd.write(line + self.newline) + write = self.output_fd.write + output_func = lambda line: write(line + self.newline) else: - print(line) + output_func = print + + output_func(line) + if source: + output_func(source) def stop(self): """Clean up after reporting is finished.""" diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 2c18c9d..fd4692c 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -36,11 +36,17 @@ class Decision(enum.Enum): Selected = 'selected error' -Error = collections.namedtuple('Error', ['code', - 'filename', - 'line_number', - 'column_number', - 'text']) +Error = collections.namedtuple( + 'Error', + [ + 'code', + 'filename', + 'line_number', + 'column_number', + 'text', + 'physical_line', + ], +) class StyleGuide(object): @@ -157,9 +163,10 @@ class StyleGuide(object): LOG.debug('"%s" will be "%s"', code, decision) return decision - def is_inline_ignored(self, error, physical_line=None): + def is_inline_ignored(self, error): # type: (Error) -> bool """Determine if an comment has been added to ignore this line.""" + physical_line = error.physical_line # TODO(sigmavirus24): Determine how to handle stdin with linecache if self.options.disable_noqa: return False @@ -191,9 +198,10 @@ class StyleGuide(object): physical_line=None): # type: (str, str, int, int, str) -> NoneType """Handle an error reported by a check.""" - error = Error(code, filename, line_number, column_number, text) + error = Error(code, filename, line_number, column_number, text, + physical_line) if (self.should_report_error(error.code) is Decision.Selected and - self.is_inline_ignored(error, physical_line) is False): + self.is_inline_ignored(error) is False): self.formatter.handle(error) self.listener.notify(error.code, error) diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index 973281b..2b14721 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -134,7 +134,8 @@ def test_is_inline_ignored(error_code, physical_line, expected_result): guide = style_guide.StyleGuide(create_options(select=['E', 'W', 'F']), listener_trie=None, formatter=None) - error = style_guide.Error(error_code, 'filename.py', 1, 1, 'error text') + error = style_guide.Error(error_code, 'filename.py', 1, 1, 'error text', + physical_line) with mock.patch('linecache.getline', return_value=physical_line): assert guide.is_inline_ignored(error) is expected_result @@ -145,7 +146,8 @@ def test_disable_is_inline_ignored(): guide = style_guide.StyleGuide(create_options(disable_noqa=True), listener_trie=None, formatter=None) - error = style_guide.Error('E121', 'filename.py', 1, 1, 'error text') + error = style_guide.Error('E121', 'filename.py', 1, 1, 'error text', + 'line') with mock.patch('linecache.getline') as getline: assert guide.is_inline_ignored(error) is False @@ -171,7 +173,8 @@ def test_handle_error_notifies_listeners(select_list, ignore_list, error_code): with mock.patch('linecache.getline', return_value=''): guide.handle_error(error_code, 'stdin', 1, 1, 'error found') - error = style_guide.Error(error_code, 'stdin', 1, 1, 'error found') + error = style_guide.Error(error_code, 'stdin', 1, 1, 'error found', + None) listener_trie.notify.assert_called_once_with(error_code, error) formatter.handle.assert_called_once_with(error) From 15043a4ab7a7d3eab7f3906e40f744c40ba83d17 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 30 May 2016 16:48:30 -0500 Subject: [PATCH 272/364] Update our linecache test to correct it --- tests/unit/test_style_guide.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index 2b14721..4aa3c54 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -135,7 +135,9 @@ def test_is_inline_ignored(error_code, physical_line, expected_result): listener_trie=None, formatter=None) error = style_guide.Error(error_code, 'filename.py', 1, 1, 'error text', - physical_line) + None) + # We want `None` to be passed as the physical line so we actually use our + # monkey-patched linecache.getline value. with mock.patch('linecache.getline', return_value=physical_line): assert guide.is_inline_ignored(error) is expected_result From ba2d94888c2989da70fe025aad50eaee1206dc6f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 1 Jun 2016 16:56:17 -0500 Subject: [PATCH 273/364] Add tests for BaseFormatter --- flake8/formatting/base.py | 36 +++++++---- tests/unit/test_base_formatter.py | 103 ++++++++++++++++++++++++++++++ 2 files changed, 127 insertions(+), 12 deletions(-) create mode 100644 tests/unit/test_base_formatter.py diff --git a/flake8/formatting/base.py b/flake8/formatting/base.py index ecc5973..f66fb1f 100644 --- a/flake8/formatting/base.py +++ b/flake8/formatting/base.py @@ -55,7 +55,7 @@ class BaseFormatter(object): def handle(self, error): """Handle an error reported by Flake8. - This defaults to calling :meth:`format`, :meth:`format_source`, and + This defaults to calling :meth:`format`, :meth:`show_source`, and then :meth:`write`. To extend how errors are handled, override this method. @@ -65,7 +65,7 @@ class BaseFormatter(object): flake8.style_guide.Error """ line = self.format(error) - source = self.format_source(error) + source = self.show_source(error) self.write(line, source) def format(self, error): @@ -85,11 +85,19 @@ class BaseFormatter(object): raise NotImplementedError('Subclass of BaseFormatter did not implement' ' format.') - def format_source(self, error): - """Format the physical line generating the error. + def show_benchmarks(self, benchmarks): + pass + + def show_source(self, error): + """Show the physical line generating the error. + + This also adds an indicator for the particular part of the line that + is reported as generating the problem. :param error: This will be an instance of :class:`~flake8.style_guide.Error`. + :type error: + flake8.style_guide.Error :returns: The formatted error string if the user wants to show the source. If the user does not want to show the source, this will return @@ -104,6 +112,13 @@ class BaseFormatter(object): # one return error.physical_line + pointer + def _write(self, output): + """Handle logic of whether to use an output file or print().""" + if self.output_fd is not None: + self.output_fd.write(output + self.newline) + else: + print(output) + def write(self, line, source): """Write the line either to the output file or stdout. @@ -113,16 +128,13 @@ class BaseFormatter(object): :param str line: The formatted string to print or write. + :param str source: + The source code that has been formatted and associated with the + line of output. """ - if self.output_fd is not None: - write = self.output_fd.write - output_func = lambda line: write(line + self.newline) - else: - output_func = print - - output_func(line) + self._write(line) if source: - output_func(source) + self._write(source) def stop(self): """Clean up after reporting is finished.""" diff --git a/tests/unit/test_base_formatter.py b/tests/unit/test_base_formatter.py new file mode 100644 index 0000000..625e4c1 --- /dev/null +++ b/tests/unit/test_base_formatter.py @@ -0,0 +1,103 @@ +"""Tests for the BaseFormatter object.""" +import optparse + +import mock +import pytest + +from flake8.formatting import base +from flake8 import style_guide + + +def options(**kwargs): + """Create an optparse.Values instance.""" + kwargs.setdefault('output_file', None) + return optparse.Values(kwargs) + + +@pytest.mark.parametrize('filename', [None, 'out.txt']) +def test_start(filename): + """Verify we open a new file in the start method.""" + mock_open = mock.mock_open() + formatter = base.BaseFormatter(options(output_file=filename)) + with mock.patch('flake8.formatting.base.open', mock_open): + formatter.start() + + if filename is None: + assert mock_open.called is False + else: + mock_open.assert_called_once_with(filename, 'w') + + +def test_stop(): + """Verify we close open file objects.""" + filemock = mock.Mock() + formatter = base.BaseFormatter(options()) + formatter.output_fd = filemock + formatter.stop() + + filemock.close.assert_called_once_with() + assert formatter.output_fd is None + + +def test_format_needs_to_be_implemented(): + """Ensure BaseFormatter#format raises a NotImplementedError.""" + formatter = base.BaseFormatter(options()) + with pytest.raises(NotImplementedError): + formatter.format('foo') + + +def test_show_source_returns_nothing_when_not_showing_source(): + """Ensure we return nothing when users want nothing.""" + formatter = base.BaseFormatter(options(show_source=False)) + assert formatter.show_source( + style_guide.Error('A000', 'file.py', 1, 1, 'error text', 'line') + ) is None + + +@pytest.mark.parametrize('line, column', [ + ('x=1\n', 2), + (' x=(1\n +2)\n', 5), + # TODO(sigmavirus24): Add more examples +]) +def test_show_source_updates_physical_line_appropriately(line, column): + """Ensure the error column is appropriately indicated.""" + formatter = base.BaseFormatter(options(show_source=True)) + error = style_guide.Error('A000', 'file.py', 1, column, 'error', line) + output = formatter.show_source(error) + _, pointer = output.rsplit('\n', 1) + assert pointer.count(' ') == column + + +def test_write_uses_an_output_file(): + """Verify that we use the output file when it's present.""" + line = 'Something to write' + source = 'source' + filemock = mock.Mock() + + formatter = base.BaseFormatter(options()) + formatter.output_fd = filemock + formatter.write(line, source) + + assert filemock.write.called is True + assert filemock.write.call_count == 2 + assert filemock.write.mock_calls == [ + mock.call(line + formatter.newline), + mock.call(source + formatter.newline), + ] + + +@mock.patch('flake8.formatting.base.print') +def test_write_uses_print(print_function): + """Verify that we use the print function without an output file.""" + line = 'Something to write' + source = 'source' + + formatter = base.BaseFormatter(options()) + formatter.write(line, source) + + assert print_function.called is True + assert print_function.call_count == 2 + assert print_function.mock_calls == [ + mock.call(line), + mock.call(source), + ] From 250005b100a4d932643e169027fc6cfe78dff520 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 3 Jun 2016 07:44:41 -0500 Subject: [PATCH 274/364] Use platform independent path separator When normalizing paths in flake8.utils, we use the os.path.sep constant to determine if the item is intended to be a path. If Windows users then have a path like foo\bar Specified, they will get the same behaviour as a *nix user with foo/bar --- flake8/utils.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/flake8/utils.py b/flake8/utils.py index 1ceb76f..8341a06 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -48,9 +48,12 @@ def normalize_path(path, parent=os.curdir): :rtype: str """ - if '/' in path: + # NOTE(sigmavirus24): Using os.path.sep allows for Windows paths to + # be specified and work appropriately. + separator = os.path.sep + if separator in path: path = os.path.abspath(os.path.join(parent, path)) - return path.rstrip('/') + return path.rstrip(separator) def stdin_get_value(): From 9c2e747528766efdabcd48a7651d8e5206217d11 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 12:37:56 -0500 Subject: [PATCH 275/364] Add tests around BaseFormatter#after_init --- tests/unit/test_base_formatter.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tests/unit/test_base_formatter.py b/tests/unit/test_base_formatter.py index 625e4c1..879ca35 100644 --- a/tests/unit/test_base_formatter.py +++ b/tests/unit/test_base_formatter.py @@ -101,3 +101,17 @@ def test_write_uses_print(print_function): mock.call(line), mock.call(source), ] + + +class AfterInitFormatter(base.BaseFormatter): + """Subclass for testing after_init.""" + + def after_init(self): + """Define method to verify operation.""" + self.post_initialized = True + + +def test_after_init_is_always_called(): + """Verify after_init is called.""" + formatter = AfterInitFormatter() + assert getattr(formatter, 'post_initialized') is True From d4777150640e21ac222c559a660e6a3c8c27a031 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 13:02:09 -0500 Subject: [PATCH 276/364] Fix linting issues --- flake8/formatting/base.py | 1 + tests/unit/test_base_formatter.py | 9 +++++---- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/flake8/formatting/base.py b/flake8/formatting/base.py index f66fb1f..19e21e4 100644 --- a/flake8/formatting/base.py +++ b/flake8/formatting/base.py @@ -86,6 +86,7 @@ class BaseFormatter(object): ' format.') def show_benchmarks(self, benchmarks): + """Format and print the benchmarks.""" pass def show_source(self, error): diff --git a/tests/unit/test_base_formatter.py b/tests/unit/test_base_formatter.py index 879ca35..5422aec 100644 --- a/tests/unit/test_base_formatter.py +++ b/tests/unit/test_base_formatter.py @@ -1,11 +1,12 @@ """Tests for the BaseFormatter object.""" import optparse -import mock -import pytest - -from flake8.formatting import base from flake8 import style_guide +from flake8.formatting import base + +import mock + +import pytest def options(**kwargs): From 9ecca93a9b400c20af613cbb71d3d531ce8c7d5a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 13:02:29 -0500 Subject: [PATCH 277/364] Add more BaseFormatter subclass tests --- tests/unit/test_base_formatter.py | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/tests/unit/test_base_formatter.py b/tests/unit/test_base_formatter.py index 5422aec..a23a9cb 100644 --- a/tests/unit/test_base_formatter.py +++ b/tests/unit/test_base_formatter.py @@ -114,5 +114,31 @@ class AfterInitFormatter(base.BaseFormatter): def test_after_init_is_always_called(): """Verify after_init is called.""" - formatter = AfterInitFormatter() + formatter = AfterInitFormatter(options()) assert getattr(formatter, 'post_initialized') is True + + +class FormatFormatter(base.BaseFormatter): + """Subclass for testing format.""" + + def format(self, error): + """Define method to verify operation.""" + return repr(error) + + +def test_handle_formats_the_error(): + """Verify that a formatter will call format from handle.""" + formatter = FormatFormatter(options(show_source=False)) + filemock = formatter.output_fd = mock.Mock() + error = style_guide.Error( + code='A001', + filename='example.py', + line_number=1, + column_number=1, + text='Fake error', + physical_line='a = 1', + ) + + formatter.handle(error) + + filemock.write.assert_called_once_with(repr(error) + '\n') From 699d116fb07b3746e449fcd3ea6e79171034392b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 13:53:02 -0500 Subject: [PATCH 278/364] Use correct basepython for docs testenvs --- tox.ini | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 24ef6e8..1be408f 100644 --- a/tox.ini +++ b/tox.ini @@ -86,6 +86,7 @@ commands = # Documentation [testenv:docs] +basepython = python3 deps = sphinx>=1.3.0 sphinx_rtd_theme @@ -94,7 +95,7 @@ commands = sphinx-build -E -W -c docs/source/ -b html docs/source/ docs/build/html [testenv:serve-docs] -basepython = python3.4 +basepython = python3 skipsdist = true skip_install = true use_develop = false From eb9fcac064fec006cb776d17bbcec1bc2b642579 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 14:18:44 -0500 Subject: [PATCH 279/364] Add contributing documentation --- docs/source/internal/contributing.rst | 200 ++++++++++++++++++++++++++ docs/source/internal/index.rst | 1 + 2 files changed, 201 insertions(+) create mode 100644 docs/source/internal/contributing.rst diff --git a/docs/source/internal/contributing.rst b/docs/source/internal/contributing.rst new file mode 100644 index 0000000..f9feb2e --- /dev/null +++ b/docs/source/internal/contributing.rst @@ -0,0 +1,200 @@ +======================== + Contributing to Flake8 +======================== + +We encourage multiple methods of participation in Flake8: + +- contributing bug reports and feature requests + +- contributing documenation (and yes that includes this document) + +- reviewing and triaging bugs and merge requests + +Before you go any further, please allow me to reassure you that I do want +*your* contribution. If you think your contribution might not be valuable, I +reassure you that any help you can provide *is* valuable. + + +Code of Conduct +=============== + +Flake8 adheres to the `Python Code Quality Authority's Code of Conduct`_. +Any violations of the Code of Conduct should be reported to Ian Cordasco +(graffatcolmingov [at] gmail [dot] com). + + +Setting Up A Development Environment +==================================== + +To contribute to Flake8's development, you simply need: + +- Python (one of the versions we support) + +- `tox`_ + + We suggest installing this like: + + .. prompt:: bash + + pip install --user tox + + Or + + .. prompt:: bash + + python -m pip install --user tox + +- and your favorite editor + + +Filing a Bug +============ + +When filing a bug against Flake8, please fill out the issue template as it is +provided to you by `GitLab`_. If your bug is in reference to one of the +checks that Flake8 reports by default, please do not report them to Flake8 +unless Flake8 is doing something to prevent the check from running or you +have some reason to believe Flake8 is inhibiting the effectiveness of the +check. + +**Please search for closed and open bug reports before opening new ones.** + +All bug reports about checks should go to their respective projects: + +- Check codes starting with ``E`` and ``W`` should be reported to + `pycodestyle`_. + +- Check codes starting with ``F`` should be reported to `pyflakes`_ + +- Check codes starting with ``C`` should be reported to `mccabe`_ + + +Requesting a New Feature +======================== + +When requesting a new feature in Flake8, please fill out the issue template. +Please also note if there are any existing alternatives to your new feature +either via plugins, or combining command-line options. Please provide example +use cases. For example, do not ask for a feature like this: + + I need feature frobulate for my job. + +Instead ask: + + I need Flake8 to frobulate these files because my team expects them to + frobulated but Flake8 currently does not frobulate them. We tried using + ``--filename`` but we could not create a pattern that worked. + +The more you explain about *why* you need a feature, the more likely we are to +understand your needs and help you to the best of our ability. + + +Contributing Documentation +========================== + +To contribute to Flake8's documentation, you should first familiarize yourself +with reStructuredText and Sphinx. For the most part, you should be fine +following the structure and style of the rest of Flake8's documentation. + +All of Flake8's documentation is written in reStructuredText and rendered by +Sphinx. The source (reStructuredText) lives in ``docs/source/``. To build +the documentation the way our Continuous Integration does, run: + +.. prompt:: bash + + tox -e docs + +To view the documentation locally, you can also run: + +.. prompt:: bash + + tox -e serve-docs + +You can run the latter in a separate terminal and continuously re-run the +documentation generation and refresh the documentation you're working on. + +.. note:: + + We lint our documentation just like we lint our code. + You should also run: + + .. prompt:: bash + + tox -e linters + + After making changes and before pushing them to ensure that they will + pass our CI tests. + + +Contributing Code +================= + +Flake8 development happens on `GitLab`_. Code contributions should be +submitted there. + +Merge requests should: + +- Fix one issue and fix it well + + Fix the issue, but do not include extraneous refactoring or code + reformatting. In other words, keep the diff short, but only as short + as is necessary to fix the bug appropriately and add sufficient testing + around it. Long diffs are fine, so long as everything that it includes + is necessary to the purpose of the merge request. + +- Have descriptive titles and descriptions + + Searching old merge requests is made easier when a merge request is well + described. + +- Have commits that follow this style: + + .. code:: + + Create a short title that is 50 characters long + + Ensure the title and commit message use the imperative voice. The + commit and you are doing something. Also, please ensure that the + body of the commit message does not exceed 72 characters. + + The body may have multiple paragraphs as necessary. + + The final line of the body references the issue appropriately. + + +Reviewing and Triaging Issues and Merge Requests +================================================ + +When reviewing other people's merge requests and issues, please be +**especially** mindful of how the words you choose can be ready by someone +else. We strive for professional code reviews that do not insult the +contributor's intelligence or impugn their character. The code review +should be focused on the code, it's effectiveness, and whether it is +appropriate for Flake8. + +If you have the ability to edit an issue or merge request's labels, please do +so to make search and prioritization easier. + +Flake8 uses milestones with both issues and merge requests to provide +other contributors direction about when an issue or merge request will be +delivered. + + +.. links +.. _Python Code Quality Authority's Code of Conduct: + http://meta.pycqa.org/en/latest/code-of-conduct.html + +.. _tox: + https://tox.readthedocs.io/ + +.. _GitLab: + https://gitlab.com/pycqa/flake8 + +.. _pycodestyle: + https://github.com/pycqa/pycodestyle + +.. _pyflakes: + https://github.com/pyflakes/pyflakes + +.. _mccabe: + https://github.com/pycqa/mccabe diff --git a/docs/source/internal/index.rst b/docs/source/internal/index.rst index 4220fe9..81efa2c 100644 --- a/docs/source/internal/index.rst +++ b/docs/source/internal/index.rst @@ -15,6 +15,7 @@ pull gently. .. toctree:: :maxdepth: 2 + contributing checker cli formatters From 0a418f8d1effb0d64a89f9ad693ca0d68d788ae7 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 14:26:37 -0500 Subject: [PATCH 280/364] Refactor docs requirements into file for RTD --- docs/source/requirements.txt | 4 ++++ tox.ini | 4 +--- 2 files changed, 5 insertions(+), 3 deletions(-) create mode 100644 docs/source/requirements.txt diff --git a/docs/source/requirements.txt b/docs/source/requirements.txt new file mode 100644 index 0000000..77bd874 --- /dev/null +++ b/docs/source/requirements.txt @@ -0,0 +1,4 @@ +sphinx>=1.3.0 +sphinx_rtd_theme +sphinx-prompt +configparser diff --git a/tox.ini b/tox.ini index 1be408f..03d30c8 100644 --- a/tox.ini +++ b/tox.ini @@ -88,9 +88,7 @@ commands = [testenv:docs] basepython = python3 deps = - sphinx>=1.3.0 - sphinx_rtd_theme - sphinx-prompt + -rdocs/source/requirements.txt commands = sphinx-build -E -W -c docs/source/ -b html docs/source/ docs/build/html From 0fbfeb919a070d79e5020210e50d5b0d66b1add6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 15:19:11 -0500 Subject: [PATCH 281/364] Trim trailing whitespace in contributing docs --- docs/source/internal/contributing.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/internal/contributing.rst b/docs/source/internal/contributing.rst index f9feb2e..88f8c16 100644 --- a/docs/source/internal/contributing.rst +++ b/docs/source/internal/contributing.rst @@ -11,7 +11,7 @@ We encourage multiple methods of participation in Flake8: - reviewing and triaging bugs and merge requests Before you go any further, please allow me to reassure you that I do want -*your* contribution. If you think your contribution might not be valuable, I +*your* contribution. If you think your contribution might not be valuable, I reassure you that any help you can provide *is* valuable. @@ -31,7 +31,7 @@ To contribute to Flake8's development, you simply need: - Python (one of the versions we support) - `tox`_ - + We suggest installing this like: .. prompt:: bash From e93091575cf0e9989e2658fef55567495066cb36 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 15:23:31 -0500 Subject: [PATCH 282/364] Remove last trailing whitespace --- docs/source/internal/contributing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/internal/contributing.rst b/docs/source/internal/contributing.rst index 88f8c16..255929b 100644 --- a/docs/source/internal/contributing.rst +++ b/docs/source/internal/contributing.rst @@ -129,7 +129,7 @@ documentation generation and refresh the documentation you're working on. Contributing Code ================= -Flake8 development happens on `GitLab`_. Code contributions should be +Flake8 development happens on `GitLab`_. Code contributions should be submitted there. Merge requests should: From c4f463e4fbab8739bd622e67a305991185b3e16f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 4 Jun 2016 15:46:51 -0500 Subject: [PATCH 283/364] Symlink the contributing docs to CONTRIBUTING --- CONTRIBUTING.rst | 1 + 1 file changed, 1 insertion(+) create mode 120000 CONTRIBUTING.rst diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst new file mode 120000 index 0000000..8deb188 --- /dev/null +++ b/CONTRIBUTING.rst @@ -0,0 +1 @@ +docs/source/internal/contributing.rst \ No newline at end of file From 0cdde196e8eb7a94e3781fa0c14ede70eabedbb3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 6 Jun 2016 08:41:22 -0500 Subject: [PATCH 284/364] Add utils.parse_unified_diff function This will read stdin using utils.stdin_get_value and parse it into a dictionary similar to pep8/pycodestyle's parse_udiff function. This differs from that function in the order in which it parses the diff and also in that it does not try to filter the paths in the diff. It lets a different level take care of that for it. This will likely be used to send paths to the FileCheckerManager and also send the ranges to the StyleGuide so it can filter errors not in those ranges. --- flake8/utils.py | 71 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/flake8/utils.py b/flake8/utils.py index 8341a06..a0db636 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -1,10 +1,14 @@ """Utility methods for flake8.""" +import collections import fnmatch as _fnmatch import inspect import io import os +import re import sys +DIFF_HUNK_REGEXP = re.compile(r'^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@.*$') + def parse_comma_separated_list(value): # type: (Union[Sequence[str], str]) -> List[str] @@ -70,6 +74,73 @@ def stdin_get_value(): return cached_value.getvalue() +def parse_unified_diff(): + # type: () -> List[str] + """Parse the unified diff passed on stdin. + + :returns: dictionary mapping file names to sets of ranges + :rtype: dict + """ + diff = stdin_get_value() + number_of_rows = None + current_path = None + parsed_paths = collections.defaultdict(set) + for line in diff.splitlines(): + if number_of_rows: + # NOTE(sigmavirus24): Below we use a slice because stdin may be + # bytes instead of text on Python 3. + if line[:1] != '-': + number_of_rows -= 1 + # We're in the part of the diff that has lines starting with +, -, + # and ' ' to show context and the changes made. We skip these + # because the information we care about is the filename and the + # range within it. + # When number_of_rows reaches 0, we will once again start + # searching for filenames and ranges. + continue + + # NOTE(sigmavirus24): Diffs that we support look roughly like: + # diff a/file.py b/file.py + # ... + # --- a/file.py + # +++ b/file.py + # Below we're looking for that last line. Every diff tool that + # gives us this output may have additional information after + # ``b/file.py`` which it will separate with a \t, e.g., + # +++ b/file.py\t100644 + # Which is an example that has the new file permissions/mode. + # In this case we only care about the file name. + if line[:3] == '+++': + current_path = line[4:].split('\t', 1)[0] + # NOTE(sigmavirus24): This check is for diff output from git. + if current_path[:2] == 'b/': + current_path = current_path[2:] + # We don't need to do anything else. We have set up our local + # ``current_path`` variable. We can skip the rest of this loop. + # The next line we will see will give us the hung information + # which is in the next section of logic. + continue + + hunk_match = DIFF_HUNK_REGEXP.match(line) + # NOTE(sigmavirus24): pep8/pycodestyle check for: + # line[:3] == '@@ ' + # But the DIFF_HUNK_REGEXP enforces that the line start with that + # So we can more simply check for a match instead of slicing and + # comparing. + if hunk_match: + (row, number_of_rows) = [ + 1 if not group else int(group) + for group in hunk_match.groups() + ] + parsed_paths[current_path].update( + range(row, row + number_of_rows) + ) + + # We have now parsed our diff into a dictionary that looks like: + # {'file.py': set(range(10, 16), range(18, 20)), ...} + return parsed_paths + + def is_windows(): # type: () -> bool """Determine if we're running on Windows. From 59c3ba75e6ee53c07ac07adc46b715701deb3fec Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 6 Jun 2016 13:03:55 -0500 Subject: [PATCH 285/364] Wire up last pieces to make --diff work To make sure we reduce as much duplication as possible, we parse the diff output in our main command-line Application. That then takes responsibility for telling the StyleGuide about the line numbers that are in the diff as well as telling the file checker manager which files from the diff should be checked. --- flake8/checker.py | 11 ++++++--- flake8/main/cli.py | 18 +++++++++++++- flake8/style_guide.py | 56 +++++++++++++++++++++++++++++++++++++++++-- flake8/utils.py | 7 ++++-- 4 files changed, 84 insertions(+), 8 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index c96d6f6..75c98c1 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -280,10 +280,15 @@ class Manager(object): LOG.warning('Running in serial after OS exception, %r', oserr) self.run_serial() - def start(self): - """Start checking files.""" + def start(self, paths=None): + """Start checking files. + + :param list paths: + Path names to check. This is passed directly to + :meth:`~Manager.make_checkers`. + """ LOG.info('Making checkers') - self.make_checkers() + self.make_checkers(paths) if not self.using_multiprocessing: return diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 455629f..b426e1e 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -8,6 +8,7 @@ import flake8 from flake8 import checker from flake8 import defaults from flake8 import style_guide +from flake8 import utils from flake8.options import aggregator from flake8.options import manager from flake8.plugins import manager as plugin_manager @@ -266,6 +267,11 @@ class Application(object): #: flake8 self.result_count = 0 + #: Whether the program is processing a diff or not + self.running_against_diff = False + #: The parsed diff information + self.parsed_diff = {} + def exit(self): # type: () -> NoneType """Handle finalization and exiting the program. @@ -318,6 +324,10 @@ class Application(object): self.option_manager, argv ) + self.running_against_diff = self.options.diff + if self.running_against_diff: + self.parsed_diff = utils.parse_unified_diff() + self.check_plugins.provide_options(self.option_manager, self.options, self.args) self.listening_plugins.provide_options(self.option_manager, @@ -349,6 +359,9 @@ class Application(object): self.options, self.listener_trie, self.formatter ) + if self.running_against_diff: + self.guide.add_diff_ranges(self.parsed_diff) + def make_file_checker_manager(self): # type: () -> NoneType """Initialize our FileChecker Manager.""" @@ -367,7 +380,10 @@ class Application(object): :class:`~flake8.checker.Manger` instance run the checks it is managing. """ - self.file_checker_manager.start() + files = None + if self.running_against_diff: + files = list(sorted(self.parsed_diff.keys())) + self.file_checker_manager.start(files) self.file_checker_manager.run() LOG.info('Finished running') self.file_checker_manager.stop() diff --git a/flake8/style_guide.py b/flake8/style_guide.py index fd4692c..386edb1 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -77,6 +77,7 @@ class StyleGuide(object): self._selected = tuple(options.select) self._ignored = tuple(options.ignore) self._decision_cache = {} + self._parsed_diff = {} def is_user_selected(self, code): # type: (str) -> Union[Selected, Ignored] @@ -194,17 +195,68 @@ class StyleGuide(object): error, codes_str) return False + def is_in_diff(self, error): + # type: (Error) -> bool + """Determine if an error is included in a diff's line ranges. + + This function relies on the parsed data added via + :meth:`~StyleGuide.add_diff_ranges`. If that has not been called and + we are not evaluating files in a diff, then this will always return + True. If there are diff ranges, then this will return True if the + line number in the error falls inside one of the ranges for the file + (and assuming the file is part of the diff data). If there are diff + ranges, this will return False if the file is not part of the diff + data or the line number of the error is not in any of the ranges of + the diff. + + :returns: + True if there is no diff or if the error is in the diff's line + number ranges. False if the error's line number falls outside + the diff's line number ranges. + :rtype: + bool + """ + if not self._parsed_diff: + return True + + # NOTE(sigmavirus24): The parsed diff will be a defaultdict with + # a set as the default value (if we have received it from + # flake8.utils.parse_unified_diff). In that case ranges below + # could be an empty set (which is False-y) or if someone else + # is using this API, it could be None. If we could guarantee one + # or the other, we would check for it more explicitly. + line_numbers = self._parsed_diff.get(error.filename) + if not line_numbers: + return False + + return error.line_number in line_numbers + def handle_error(self, code, filename, line_number, column_number, text, physical_line=None): # type: (str, str, int, int, str) -> NoneType """Handle an error reported by a check.""" error = Error(code, filename, line_number, column_number, text, physical_line) - if (self.should_report_error(error.code) is Decision.Selected and - self.is_inline_ignored(error) is False): + error_is_selected = (self.should_report_error(error.code) is + Decision.Selected) + is_not_inline_ignored = self.is_inline_ignored(error) is False + is_included_in_diff = self.is_in_diff(error) + if (error_is_selected and is_not_inline_ignored and + is_included_in_diff): self.formatter.handle(error) self.listener.notify(error.code, error) + def add_diff_ranges(self, diffinfo): + """Update the StyleGuide to filter out information not in the diff. + + This provides information to the StyleGuide so that only the errors + in the line number ranges are reported. + + :param dict diffinfo: + Dictionary mapping filenames to sets of line number ranges. + """ + self._parsed_diff = diffinfo + # Should separate style guide logic from code that runs checks # StyleGuide should manage select/ignore logic as well as include/exclude # logic. See also https://github.com/PyCQA/pep8/pull/433 diff --git a/flake8/utils.py b/flake8/utils.py index a0db636..f6ce384 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -71,6 +71,7 @@ def stdin_get_value(): else: cached_type = io.StringIO stdin_get_value.cached_stdin = cached_type(stdin_value) + cached_value = stdin_get_value.cached_stdin return cached_value.getvalue() @@ -78,8 +79,10 @@ def parse_unified_diff(): # type: () -> List[str] """Parse the unified diff passed on stdin. - :returns: dictionary mapping file names to sets of ranges - :rtype: dict + :returns: + dictionary mapping file names to sets of line numbers + :rtype: + dict """ diff = stdin_get_value() number_of_rows = None From 689562f1e8a03b0930b4beb1d8d5c35b75d08f65 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 6 Jun 2016 19:42:12 -0500 Subject: [PATCH 286/364] Allow reporting on files passed via stdin --- flake8/checker.py | 2 +- flake8/style_guide.py | 11 ----------- 2 files changed, 1 insertion(+), 12 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 75c98c1..8b1a8ec 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -217,7 +217,7 @@ class Manager(object): for argument in paths for filename in utils.filenames_from(argument, self.is_path_excluded) - if utils.fnmatch(filename, filename_patterns) + if utils.fnmatch(filename, filename_patterns) or filename == '-' ] def report(self): diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 386edb1..57d86b2 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -256,14 +256,3 @@ class StyleGuide(object): Dictionary mapping filenames to sets of line number ranges. """ self._parsed_diff = diffinfo - -# Should separate style guide logic from code that runs checks -# StyleGuide should manage select/ignore logic as well as include/exclude -# logic. See also https://github.com/PyCQA/pep8/pull/433 - -# StyleGuide shoud dispatch check execution in a way that can use -# multiprocessing but also retry in serial. See also: -# https://gitlab.com/pycqa/flake8/issues/74 - -# StyleGuide should interface with Reporter and aggregate errors/notify -# listeners From 02bcbee245b9b5a0c75345ada0f1d7e70d19b3c0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 6 Jun 2016 19:46:29 -0500 Subject: [PATCH 287/364] Add --stdin-display-name to modify report output --- docs/source/user/options.rst | 24 ++++++++++++++++++++++++ flake8/main/cli.py | 7 +++++++ flake8/style_guide.py | 2 ++ 3 files changed, 33 insertions(+) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index ce30bdf..e336daa 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -192,6 +192,30 @@ another-example*.py +.. option:: --stdin-display-name= + + Provide the name to use to report warnings and errors from code on stdin. + + Instead of reporting an error as something like: + + .. code:: + + stdin:82:73 E501 line too long + + You can specify this option to have it report whatever value you want + instead of stdin. + + This defaults to: ``stdin`` + + Command-line example: + + .. prompt:: bash + + cat file.py | flake8 --stdin-display-name=file.py - + + This **can not** be specified in config files. + + .. option:: --format= Select the formatter used to display errors to the user. diff --git a/flake8/main/cli.py b/flake8/main/cli.py index b426e1e..0b50e27 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -86,6 +86,13 @@ def register_default_options(option_manager): 'separated list. (Default: %default)', ) + add_option( + '--stdin-display-name', default='stdin', + help='The name used when reporting errors from code passed via stdin.' + ' This is useful for editors piping the file contents to flake8.' + ' (Default: %default)', + ) + # TODO(sigmavirus24): Figure out --first/--repeat add_option( diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 57d86b2..6be9ff6 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -237,6 +237,8 @@ class StyleGuide(object): """Handle an error reported by a check.""" error = Error(code, filename, line_number, column_number, text, physical_line) + if error.filename is None or error.filename == '-': + error = error._replace(filename=self.options.stdin_display_name) error_is_selected = (self.should_report_error(error.code) is Decision.Selected) is_not_inline_ignored = self.is_inline_ignored(error) is False From 9ebaa5c69c7dc7077d1f59141a863d22e23284c0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 10:14:45 -0500 Subject: [PATCH 288/364] Add tests for parse_unified_diff We could probably use non-git diff fixtures, but those are what we have for now. --- flake8/utils.py | 9 +- tests/fixtures/diffs/multi_file_diff | 130 ++++++++++++++++++++++++++ tests/fixtures/diffs/single_file_diff | 27 ++++++ tests/fixtures/diffs/two_file_diff | 45 +++++++++ tests/unit/test_utils.py | 35 +++++++ 5 files changed, 243 insertions(+), 3 deletions(-) create mode 100644 tests/fixtures/diffs/multi_file_diff create mode 100644 tests/fixtures/diffs/single_file_diff create mode 100644 tests/fixtures/diffs/two_file_diff diff --git a/flake8/utils.py b/flake8/utils.py index f6ce384..7cd12b0 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -75,8 +75,8 @@ def stdin_get_value(): return cached_value.getvalue() -def parse_unified_diff(): - # type: () -> List[str] +def parse_unified_diff(diff=None): + # type: (str) -> List[str] """Parse the unified diff passed on stdin. :returns: @@ -84,7 +84,10 @@ def parse_unified_diff(): :rtype: dict """ - diff = stdin_get_value() + # Allow us to not have to patch out stdin_get_value + if diff is None: + diff = stdin_get_value() + number_of_rows = None current_path = None parsed_paths = collections.defaultdict(set) diff --git a/tests/fixtures/diffs/multi_file_diff b/tests/fixtures/diffs/multi_file_diff new file mode 100644 index 0000000..de86209 --- /dev/null +++ b/tests/fixtures/diffs/multi_file_diff @@ -0,0 +1,130 @@ +diff --git a/flake8/utils.py b/flake8/utils.py +index f6ce384..7cd12b0 100644 +--- a/flake8/utils.py ++++ b/flake8/utils.py +@@ -75,8 +75,8 @@ def stdin_get_value(): + return cached_value.getvalue() + + +-def parse_unified_diff(): +- # type: () -> List[str] ++def parse_unified_diff(diff=None): ++ # type: (str) -> List[str] + """Parse the unified diff passed on stdin. + + :returns: +@@ -84,7 +84,10 @@ def parse_unified_diff(): + :rtype: + dict + """ +- diff = stdin_get_value() ++ # Allow us to not have to patch out stdin_get_value ++ if diff is None: ++ diff = stdin_get_value() ++ + number_of_rows = None + current_path = None + parsed_paths = collections.defaultdict(set) +diff --git a/tests/fixtures/diffs/single_file_diff b/tests/fixtures/diffs/single_file_diff +new file mode 100644 +index 0000000..77ca534 +--- /dev/null ++++ b/tests/fixtures/diffs/single_file_diff +@@ -0,0 +1,27 @@ ++diff --git a/flake8/utils.py b/flake8/utils.py ++index f6ce384..7cd12b0 100644 ++--- a/flake8/utils.py +++++ b/flake8/utils.py ++@@ -75,8 +75,8 @@ def stdin_get_value(): ++ return cached_value.getvalue() ++ ++ ++-def parse_unified_diff(): ++- # type: () -> List[str] +++def parse_unified_diff(diff=None): +++ # type: (str) -> List[str] ++ """Parse the unified diff passed on stdin. ++ ++ :returns: ++@@ -84,7 +84,10 @@ def parse_unified_diff(): ++ :rtype: ++ dict ++ """ ++- diff = stdin_get_value() +++ # Allow us to not have to patch out stdin_get_value +++ if diff is None: +++ diff = stdin_get_value() +++ ++ number_of_rows = None ++ current_path = None ++ parsed_paths = collections.defaultdict(set) +diff --git a/tests/fixtures/diffs/two_file_diff b/tests/fixtures/diffs/two_file_diff +new file mode 100644 +index 0000000..5bd35cd +--- /dev/null ++++ b/tests/fixtures/diffs/two_file_diff +@@ -0,0 +1,45 @@ ++diff --git a/flake8/utils.py b/flake8/utils.py ++index f6ce384..7cd12b0 100644 ++--- a/flake8/utils.py +++++ b/flake8/utils.py ++@@ -75,8 +75,8 @@ def stdin_get_value(): ++ return cached_value.getvalue() ++ ++ ++-def parse_unified_diff(): ++- # type: () -> List[str] +++def parse_unified_diff(diff=None): +++ # type: (str) -> List[str] ++ """Parse the unified diff passed on stdin. ++ ++ :returns: ++@@ -84,7 +84,10 @@ def parse_unified_diff(): ++ :rtype: ++ dict ++ """ ++- diff = stdin_get_value() +++ # Allow us to not have to patch out stdin_get_value +++ if diff is None: +++ diff = stdin_get_value() +++ ++ number_of_rows = None ++ current_path = None ++ parsed_paths = collections.defaultdict(set) ++diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py ++index d69d939..21482ce 100644 ++--- a/tests/unit/test_utils.py +++++ b/tests/unit/test_utils.py ++@@ -115,3 +115,13 @@ def test_parameters_for_function_plugin(): ++ plugin = plugin_manager.Plugin('plugin-name', object()) ++ plugin._plugin = fake_plugin ++ assert utils.parameters_for(plugin) == ['physical_line', 'self', 'tree'] +++ +++ +++def read_diff_file(filename): +++ """Read the diff file in its entirety.""" +++ with open(filename, 'r') as fd: +++ content = fd.read() +++ return content +++ +++ +++SINGLE_FILE_DIFF = read_diff_file('tests/fixtures/diffs/single_file_diff') +diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py +index d69d939..1461369 100644 +--- a/tests/unit/test_utils.py ++++ b/tests/unit/test_utils.py +@@ -115,3 +115,14 @@ def test_parameters_for_function_plugin(): + plugin = plugin_manager.Plugin('plugin-name', object()) + plugin._plugin = fake_plugin + assert utils.parameters_for(plugin) == ['physical_line', 'self', 'tree'] ++ ++ ++def read_diff_file(filename): ++ """Read the diff file in its entirety.""" ++ with open(filename, 'r') as fd: ++ content = fd.read() ++ return content ++ ++ ++SINGLE_FILE_DIFF = read_diff_file('tests/fixtures/diffs/single_file_diff') ++TWO_FILE_DIFF = read_diff_file('tests/fixtures/diffs/two_file_diff') diff --git a/tests/fixtures/diffs/single_file_diff b/tests/fixtures/diffs/single_file_diff new file mode 100644 index 0000000..77ca534 --- /dev/null +++ b/tests/fixtures/diffs/single_file_diff @@ -0,0 +1,27 @@ +diff --git a/flake8/utils.py b/flake8/utils.py +index f6ce384..7cd12b0 100644 +--- a/flake8/utils.py ++++ b/flake8/utils.py +@@ -75,8 +75,8 @@ def stdin_get_value(): + return cached_value.getvalue() + + +-def parse_unified_diff(): +- # type: () -> List[str] ++def parse_unified_diff(diff=None): ++ # type: (str) -> List[str] + """Parse the unified diff passed on stdin. + + :returns: +@@ -84,7 +84,10 @@ def parse_unified_diff(): + :rtype: + dict + """ +- diff = stdin_get_value() ++ # Allow us to not have to patch out stdin_get_value ++ if diff is None: ++ diff = stdin_get_value() ++ + number_of_rows = None + current_path = None + parsed_paths = collections.defaultdict(set) diff --git a/tests/fixtures/diffs/two_file_diff b/tests/fixtures/diffs/two_file_diff new file mode 100644 index 0000000..5bd35cd --- /dev/null +++ b/tests/fixtures/diffs/two_file_diff @@ -0,0 +1,45 @@ +diff --git a/flake8/utils.py b/flake8/utils.py +index f6ce384..7cd12b0 100644 +--- a/flake8/utils.py ++++ b/flake8/utils.py +@@ -75,8 +75,8 @@ def stdin_get_value(): + return cached_value.getvalue() + + +-def parse_unified_diff(): +- # type: () -> List[str] ++def parse_unified_diff(diff=None): ++ # type: (str) -> List[str] + """Parse the unified diff passed on stdin. + + :returns: +@@ -84,7 +84,10 @@ def parse_unified_diff(): + :rtype: + dict + """ +- diff = stdin_get_value() ++ # Allow us to not have to patch out stdin_get_value ++ if diff is None: ++ diff = stdin_get_value() ++ + number_of_rows = None + current_path = None + parsed_paths = collections.defaultdict(set) +diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py +index d69d939..21482ce 100644 +--- a/tests/unit/test_utils.py ++++ b/tests/unit/test_utils.py +@@ -115,3 +115,13 @@ def test_parameters_for_function_plugin(): + plugin = plugin_manager.Plugin('plugin-name', object()) + plugin._plugin = fake_plugin + assert utils.parameters_for(plugin) == ['physical_line', 'self', 'tree'] ++ ++ ++def read_diff_file(filename): ++ """Read the diff file in its entirety.""" ++ with open(filename, 'r') as fd: ++ content = fd.read() ++ return content ++ ++ ++SINGLE_FILE_DIFF = read_diff_file('tests/fixtures/diffs/single_file_diff') diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index d69d939..54aa3a7 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -115,3 +115,38 @@ def test_parameters_for_function_plugin(): plugin = plugin_manager.Plugin('plugin-name', object()) plugin._plugin = fake_plugin assert utils.parameters_for(plugin) == ['physical_line', 'self', 'tree'] + + +def read_diff_file(filename): + """Read the diff file in its entirety.""" + with open(filename, 'r') as fd: + content = fd.read() + return content + + +SINGLE_FILE_DIFF = read_diff_file('tests/fixtures/diffs/single_file_diff') +SINGLE_FILE_INFO = { + 'flake8/utils.py': set(range(75, 83)).union(set(range(84, 94))), +} +TWO_FILE_DIFF = read_diff_file('tests/fixtures/diffs/two_file_diff') +TWO_FILE_INFO = { + 'flake8/utils.py': set(range(75, 83)).union(set(range(84, 94))), + 'tests/unit/test_utils.py': set(range(115, 128)), +} +MULTI_FILE_DIFF = read_diff_file('tests/fixtures/diffs/multi_file_diff') +MULTI_FILE_INFO = { + 'flake8/utils.py': set(range(75, 83)).union(set(range(84, 94))), + 'tests/unit/test_utils.py': set(range(115, 129)), + 'tests/fixtures/diffs/single_file_diff': set(range(1, 28)), + 'tests/fixtures/diffs/two_file_diff': set(range(1, 46)), +} + + +@pytest.mark.parametrize("diff, parsed_diff", [ + (SINGLE_FILE_DIFF, SINGLE_FILE_INFO), + (TWO_FILE_DIFF, TWO_FILE_INFO), + (MULTI_FILE_DIFF, MULTI_FILE_INFO), +]) +def test_parse_unified_diff(diff, parsed_diff): + """Verify that what we parse from a diff matches expectations.""" + assert utils.parse_unified_diff(diff) == parsed_diff From 8362fa7acdf90693fa745dc3ada33406170f43bd Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 18:02:53 -0500 Subject: [PATCH 289/364] Add distinction between reported and ignored errors This allows us to properly exit if no errors were reported (due to, for example, # noqa). --- flake8/checker.py | 17 +++++++++++++---- flake8/main/cli.py | 6 +++++- flake8/style_guide.py | 27 +++++++++++++++++++++++++-- 3 files changed, 43 insertions(+), 7 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 8b1a8ec..eefc5fd 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -163,8 +163,9 @@ class Manager(object): def _handle_results(self, filename, results): style_guide = self.style_guide + reported_results_count = 0 for (error_code, line_number, column, text, physical_line) in results: - style_guide.handle_error( + reported_results_count += style_guide.handle_error( code=error_code, filename=filename, line_number=line_number, @@ -172,6 +173,7 @@ class Manager(object): text=text, physical_line=physical_line, ) + return reported_results_count def _run_checks_from_queue(self): LOG.info('Running checks in parallel') @@ -221,17 +223,24 @@ class Manager(object): ] def report(self): + # type: () -> (int, int) """Report all of the errors found in the managed file checkers. This iterates over each of the checkers and reports the errors sorted by line number. + + :returns: + A tuple of the total results found and the results reported. + :rtype: + tuple(int, int) """ - results_found = 0 + results_reported = results_found = 0 for checker in self.checkers: results = sorted(checker.results, key=lambda tup: (tup[2], tup[3])) - self._handle_results(checker.filename, results) + results_reported += self._handle_results(checker.filename, + results) results_found += len(results) - return results_found + return (results_found, results_reported) def run_parallel(self): """Run the checkers in parallel.""" diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 0b50e27..93c4a14 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -403,7 +403,11 @@ class Application(object): number of errors, warnings, and other messages found. """ LOG.info('Reporting errors') - self.result_count = self.file_checker_manager.report() + results = self.file_checker_manager.report() + self.total_result_count, self.result_count = results + LOG.info('Found a total of %d results and reported %d', + self.total_result_count, + self.result_count) def _run(self, argv): # type: (Union[NoneType, List[str]]) -> NoneType diff --git a/flake8/style_guide.py b/flake8/style_guide.py index 6be9ff6..89890ba 100644 --- a/flake8/style_guide.py +++ b/flake8/style_guide.py @@ -233,8 +233,29 @@ class StyleGuide(object): def handle_error(self, code, filename, line_number, column_number, text, physical_line=None): - # type: (str, str, int, int, str) -> NoneType - """Handle an error reported by a check.""" + # type: (str, str, int, int, str) -> int + """Handle an error reported by a check. + + :param str code: + The error code found, e.g., E123. + :param str filename: + The file in which the error was found. + :param int line_number: + The line number (where counting starts at 1) at which the error + occurs. + :param int column_number: + The column number (where counting starts at 1) at which the error + occurs. + :param str text: + The text of the error message. + :param str physical_line: + The actual physical line causing the error. + :returns: + 1 if the error was reported. 0 if it was ignored. This is to allow + for counting of the number of errors found that were not ignored. + :rtype: + int + """ error = Error(code, filename, line_number, column_number, text, physical_line) if error.filename is None or error.filename == '-': @@ -247,6 +268,8 @@ class StyleGuide(object): is_included_in_diff): self.formatter.handle(error) self.listener.notify(error.code, error) + return 1 + return 0 def add_diff_ranges(self, diffinfo): """Update the StyleGuide to filter out information not in the diff. From 3d512ae2f585b8c97f88299b0b4e44df89514751 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 18:03:37 -0500 Subject: [PATCH 290/364] Update default ignore for when we upgrade pep8 --- flake8/defaults.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/defaults.py b/flake8/defaults.py index 62939b5..d484f4c 100644 --- a/flake8/defaults.py +++ b/flake8/defaults.py @@ -1,7 +1,7 @@ """Constants that define defaults.""" EXCLUDE = '.svn,CVS,.bzr,.hg,.git,__pycache__,.tox' -IGNORE = 'E121,E123,E126,E226,E24,E704' +IGNORE = 'E121,E123,E126,E226,E24,E704,W503,W504' MAX_LINE_LENGTH = 79 # Other consants From 2cd3ac91d1f3006a0062351869935c1a1b72d14f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 18:03:58 -0500 Subject: [PATCH 291/364] Start adding test fixture files This adds E501 which is actually a poor test. Currently, pep8.maximum_line_length does its own check for noqa instead of relying on the Checker. See also: https://github.com/PyCQA/pycodestyle/pull/539 --- tests/fixtures/example-code/inline-ignores/E501.py | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 tests/fixtures/example-code/inline-ignores/E501.py diff --git a/tests/fixtures/example-code/inline-ignores/E501.py b/tests/fixtures/example-code/inline-ignores/E501.py new file mode 100644 index 0000000..62e5c0c --- /dev/null +++ b/tests/fixtures/example-code/inline-ignores/E501.py @@ -0,0 +1,3 @@ +from some.module.that.has.nested.sub.modules import ClassWithVeryVeryVeryVeryLongName # noqa: E501,F401 + +# ClassWithVeryVeryVeryVeryLongName() From 27dbd61f125a048612890d1d0c2ef0dd3f52df7d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 18:46:24 -0500 Subject: [PATCH 292/364] Add documentation around ignoring errors --- docs/source/user/configuration.rst | 2 + docs/source/user/ignoring-errors.rst | 94 ++++++++++++++++++++++++++++ docs/source/user/index.rst | 1 + 3 files changed, 97 insertions(+) create mode 100644 docs/source/user/ignoring-errors.rst diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst index 6a176ec..ddf85f7 100644 --- a/docs/source/user/configuration.rst +++ b/docs/source/user/configuration.rst @@ -1,3 +1,5 @@ +.. _configuration: + ==================== Configuring Flake8 ==================== diff --git a/docs/source/user/ignoring-errors.rst b/docs/source/user/ignoring-errors.rst new file mode 100644 index 0000000..1630c10 --- /dev/null +++ b/docs/source/user/ignoring-errors.rst @@ -0,0 +1,94 @@ +============================= + Ignoring Errors with Flake8 +============================= + +By default, |Flake8| has a list of error codes that it ignores. The list used +by a version of |Flake8| may be different than the list used by a different +version. To determine the default list, using :option:`flake8 --help` will +show the output with the current default list. + + +Changing the Ignore List +======================== + +If we want to change the list of ignored codes for a single run, we can use +:option:`flake8 --ignore` to specify a comma-separated list of codes for a +specific run on the command-line, e.g., + +.. prompt:: bash + + flake8 --ignore=E1,E23,W503 path/to/files/ path/to/more/files/ + +This tells |Flake8| to ignore any error codes starting with ``E1``, ``E23``, +or ``W503`` while it is running. + +.. note:: + + The documentation for :option:`flake8 --ignore` shows examples for how + to change the ignore list in the configuration file. See also + :ref:`configuration` as well for details about how to use configuration + files. + + +In-line Ignoring Errors +======================= + +In some cases, we might not want to ignore an error code (or class of error +codes) for the entirety of our project. Instead, we might want to ignore the +specific error code on a specific line. Let's take for example a line like + +.. code-block:: python + + example = lambda: 'example' + +Sometimes we genuinely need something this simple. We could instead define +a function like we normally would but in some contexts that distracts from +what is actually happening. In those cases, we can also do: + +.. code-block:: python + + example = lambda: 'example' # noqa: E731 + +This will only ignore the error from pycodestyle that checks for lambda +assignments and generates an ``E731``. If there are other errors on the line +then those will be reported. + +.. note:: + + If we ever want to disable Flake8 respecting ``# noqa`` comments, we can + can refer to :option:`flake8 --disable-noqa`. + +If we instead had more than one error that we wished to ignore, we could +list all of the errors with commas separating them: + +.. code-block:: python + + # noqa: E731,E123 + +Finally, if we have a particularly bad line of code, we can ignore every error +using simply ``# noqa`` with nothing after it. + + +Ignoring Entire Files +===================== + +Imagine a situation where we are adding |Flake8| to a codebase. Let's further +imagine that with the exception of a few particularly bad files, we can add +|Flake8| easily and move on with our lives. There are two ways to ignore the +file: + +#. By explicitly adding it to our list of excluded paths (see: :option:`flake8 + --exclude`) + +#. By adding ``# flake8: noqa`` to the file + +The former is the **recommended** way of ignoring entire files. By using our +exclude list, we can include it in our configuration file and have one central +place to find what files aren't included in Flake8 checks. The latter has the +benefit that when we run |Flake8| with :option:`flake8 --disable-noqa` all of +the errors in that file will show up without having to modify our +configuration. Both exist so we can choose which is better for us. + + +.. replacements +.. |Flake8| replace:: :program:`Flake8` diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 8d80402..5e7503f 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -23,6 +23,7 @@ This guide will cover all of these and the nuances for using Flake8. invocation configuration options + ignoring-errors .. config files .. command-line tutorial From 4c0b1cd5e180b2c5a7ea31eeff083d64d9fdf6b8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 18:46:43 -0500 Subject: [PATCH 293/364] Fix tests broken by updating the default ignore list --- tests/integration/test_aggregator.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/integration/test_aggregator.py b/tests/integration/test_aggregator.py index 2186c35..bb87be0 100644 --- a/tests/integration/test_aggregator.py +++ b/tests/integration/test_aggregator.py @@ -43,6 +43,6 @@ def test_aggregate_options_when_isolated(optmanager): assert options.isolated is True assert options.select == ['E11', 'E34', 'E402', 'W', 'F'] assert sorted(options.ignore) == [ - 'E121', 'E123', 'E126', 'E226', 'E24', 'E704', 'E8', + 'E121', 'E123', 'E126', 'E226', 'E24', 'E704', 'E8', 'W503', 'W504', ] assert options.exclude == [os.path.abspath('tests/*')] From ba7f4db7e9cec133f3baaa34da9cdadcef2ee68d Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 18:46:58 -0500 Subject: [PATCH 294/364] Add test fixture that mirrors documented usage --- tests/fixtures/example-code/inline-ignores/E731.py | 1 + 1 file changed, 1 insertion(+) create mode 100644 tests/fixtures/example-code/inline-ignores/E731.py diff --git a/tests/fixtures/example-code/inline-ignores/E731.py b/tests/fixtures/example-code/inline-ignores/E731.py new file mode 100644 index 0000000..866c79e --- /dev/null +++ b/tests/fixtures/example-code/inline-ignores/E731.py @@ -0,0 +1 @@ +example = lambda: 'example' # noqa: E731 From 2e5c646d7480c49637d106acfec439ef26426158 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 19:08:54 -0500 Subject: [PATCH 295/364] Exclude our fixtures from Flake8 --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 03d30c8..045abb6 100644 --- a/tox.ini +++ b/tox.ini @@ -119,5 +119,5 @@ ignore = D203 # NOTE(sigmavirus24): Once we release 3.0.0 this exclude option can be specified # across multiple lines. Presently it cannot be specified across multiple lines. # :-( -exclude = .git,__pycache__,docs/source/conf.py,old,build,dist +exclude = .git,__pycache__,docs/source/conf.py,old,build,dist,tests/fixtures/ max-complexity = 10 From 49d1cf953c4cd2b6e5d839691f25ab0d2640bbbb Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 19:37:06 -0500 Subject: [PATCH 296/364] Ensure that a file exists before processing it If we don't check for a file's existence, then we'll find ourselves with a hung subprocess due to an exception that happens with in it. --- flake8/checker.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/flake8/checker.py b/flake8/checker.py index eefc5fd..43482dd 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -214,12 +214,24 @@ class Manager(object): if paths is None: paths = self.arguments filename_patterns = self.options.filename + + # NOTE(sigmavirus24): Yes this is a little unsightly, but it's our + # best solution right now. + def should_create_file_checker(filename): + """Determine if we should create a file checker.""" + matches_filename_patterns = utils.fnmatch( + filename, filename_patterns + ) + is_stdin = filename == '-' + file_exists = os.path.exists(filename) + return (file_exists and matches_filename_patterns) or is_stdin + self.checkers = [ FileChecker(filename, self.checks, self.style_guide) for argument in paths for filename in utils.filenames_from(argument, self.is_path_excluded) - if utils.fnmatch(filename, filename_patterns) or filename == '-' + if should_create_file_checker(filename) ] def report(self): From 55f10211a37133b77f52d526de80ad808620bc6e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 19:56:41 -0500 Subject: [PATCH 297/364] Check for version of Python on Windows Certain versions of Python on Windows are capable of using multiprocessing safely and correctly. Instead of completely disabling multiprocessing on Windows, this allows us to check for safe versions. This also updates our internal utility documentation to add missing functions and add a strongly worded warning about the API of those functions. --- docs/source/internal/utils.rst | 29 ++++++++++++++++++++++++++++- flake8/checker.py | 3 ++- flake8/utils.py | 14 ++++++++++++++ 3 files changed, 44 insertions(+), 2 deletions(-) diff --git a/docs/source/internal/utils.rst b/docs/source/internal/utils.rst index d8adeac..98b2f27 100644 --- a/docs/source/internal/utils.rst +++ b/docs/source/internal/utils.rst @@ -2,7 +2,20 @@ Utility Functions =================== -Flake8 has a few utility functions that it uses and provides to plugins. +Flake8 has a few utility functions that it uses internally. + +.. warning:: + + As should be implied by where these are documented, these are all + **internal** utility functions. Their signatures and return types + may change between releases without notice. + + Bugs reported about these **internal** functions will be closed + immediately. + + If functions are needed by plugin developers, they may be requested + in the bug tracker and after careful consideration they *may* be added + to the *documented* stable API. .. autofunction:: flake8.utils.parse_comma_separated_list @@ -51,6 +64,12 @@ allows plugins to use this to retrieve ``stdin`` if necessary. This provides a convenient and explicitly named function that checks if we are currently running on a Windows (or ``nt``) operating system. +.. autofunction:: flake8.utils.can_run_multiprocessing_on_windows + +This provides a separate and distinct check from +:func:`~flake8.utils.is_windows` that allows us to check if the version of +Python we're using can actually use multiprocessing on Windows. + .. autofunction:: flake8.utils.is_using_stdin Another helpful function that is named only to be explicit given it is a very @@ -98,3 +117,11 @@ plugin. This function will return the parameters in a consistent way across versions of Python and will handle both classes and functions that are used as plugins. Further, if the plugin is a class, it will strip the ``self`` argument so we can check the parameters of the plugin consistently. + +.. autofunction:: flake8.utils.parse_unified_diff + +In order to handle usage of :option:`flake8 --diff`, Flake8 needs to be able +to parse the name of the files in the diff as well as the ranges indicated the +sections that have been changed. This function either accepts the diff as an +argument or reads the diff from standard-in. It then returns a dictionary with +filenames as the keys and sets of line numbers as the value. diff --git a/flake8/checker.py b/flake8/checker.py index 43482dd..03709dd 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -113,7 +113,8 @@ class Manager(object): 'Ignoring --jobs arguments.') return 0 - if utils.is_windows(): + if (utils.is_windows() and + not utils.can_run_multiprocessing_on_windows()): LOG.warning('The --jobs option is not available on Windows. ' 'Ignoring --jobs arguments.') return 0 diff --git a/flake8/utils.py b/flake8/utils.py index 7cd12b0..597dea6 100644 --- a/flake8/utils.py +++ b/flake8/utils.py @@ -159,6 +159,20 @@ def is_windows(): return os.name == 'nt' +def can_run_multiprocessing_on_windows(): + # type: () -> bool + """Determine if we can use multiprocessing on Windows. + + :returns: + True if the version of Python is modern enough, otherwise False + :rtype: + bool + """ + is_new_enough_python27 = sys.version_info >= (2, 7, 11) + is_new_enough_python3 = sys.version_info > (3, 2) + return is_new_enough_python27 or is_new_enough_python3 + + def is_using_stdin(paths): # type: (List[str]) -> bool """Determine if we're going to read from stdin. From e21628ad827c5f687ab0c3b5fc367351dfdfd743 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 7 Jun 2016 20:00:16 -0500 Subject: [PATCH 298/364] Update our warning message about Windows --- flake8/checker.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 03709dd..655387e 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -115,8 +115,10 @@ class Manager(object): if (utils.is_windows() and not utils.can_run_multiprocessing_on_windows()): - LOG.warning('The --jobs option is not available on Windows. ' - 'Ignoring --jobs arguments.') + LOG.warning('The --jobs option is only available on Windows on ' + 'Python 2.7.11+ and 3.3+. We have detected that you ' + 'are running an unsupported version of Python on ' + 'Windows. Ignoring --jobs arguments.') return 0 if utils.is_using_stdin(self.arguments): From 107f595d13fda6c76ad2ea218359d11209b5d4a4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 8 Jun 2016 11:48:36 -0500 Subject: [PATCH 299/364] Add flake8.api and docs for it Stub out the primary method people seem to use. --- docs/source/user/index.rst | 1 + docs/source/user/python-api.rst | 11 +++++++++++ flake8/api.py | 10 ++++++++++ 3 files changed, 22 insertions(+) create mode 100644 docs/source/user/python-api.rst create mode 100644 flake8/api.py diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 5e7503f..2ed779e 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -24,6 +24,7 @@ This guide will cover all of these and the nuances for using Flake8. configuration options ignoring-errors + python-api .. config files .. command-line tutorial diff --git a/docs/source/user/python-api.rst b/docs/source/user/python-api.rst new file mode 100644 index 0000000..2e2a16e --- /dev/null +++ b/docs/source/user/python-api.rst @@ -0,0 +1,11 @@ +=================== + Public Python API +=================== + +Flake8 3.0.0 presently does not have a public, stable Python API. + +When it does it will be located in :mod:`flake8.api` and that will +be documented here. + +.. automodule:: flake8.api + :members: diff --git a/flake8/api.py b/flake8/api.py new file mode 100644 index 0000000..9f95557 --- /dev/null +++ b/flake8/api.py @@ -0,0 +1,10 @@ +"""Module containing all public entry-points for Flake8. + +This is the only submodule in Flake8 with a guaranteed stable API. All other +submodules are considered internal only and are subject to change. +""" + + +def get_style_guide(**kwargs): + """Stub out the only function I'm aware of people using.""" + pass From b10aa91b82a35e5eac6ddb79365a5c97706b0147 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 9 Jun 2016 19:57:49 -0500 Subject: [PATCH 300/364] Use function for pep8 plugins This will make the transition to pycodestyle much easier --- setup.py | 70 +++++++++++++++++++++++++++----------------------------- 1 file changed, 34 insertions(+), 36 deletions(-) diff --git a/setup.py b/setup.py index 9fa91fd..be77c07 100644 --- a/setup.py +++ b/setup.py @@ -2,6 +2,7 @@ # -*- coding: utf-8 -*- from __future__ import with_statement +import functools import sys import setuptools @@ -49,6 +50,10 @@ def get_long_description(): descr.append(f.read()) return '\n\n'.join(descr) +PEP8 = 'pep8' +_FORMAT = '{0}.{1} = {0}:{1}' +PEP8_PLUGIN = functools.partial(_FORMAT.format, PEP8) + setuptools.setup( name="flake8", @@ -75,42 +80,35 @@ setuptools.setup( 'flake8.extension': [ 'F = flake8.plugins.pyflakes:FlakesChecker', # PEP-0008 checks provied by PyCQA/pycodestyle - 'pep8.tabs_or_spaces = pep8:tabs_or_spaces', - 'pep8.tabs_obsolete = pep8:tabs_obsolete', - 'pep8.trailing_whitespace = pep8:trailing_whitespace', - 'pep8.trailing_blank_lines = pep8:trailing_blank_lines', - 'pep8.maximum_line_length = pep8:maximum_line_length', - 'pep8.blank_lines = pep8:blank_lines', - 'pep8.extraneous_whitespace = pep8:extraneous_whitespace', - ('pep8.whitespace_around_keywords = ' - 'pep8:whitespace_around_keywords'), - 'pep8.missing_whitespace = pep8:missing_whitespace', - 'pep8.indentation = pep8:indentation', - 'pep8.continued_indentation = pep8:continued_indentation', - ('pep8.whitespace_before_parameters = ' - 'pep8:whitespace_before_parameters'), - ('pep8.whitespace_around_operator = ' - 'pep8:whitespace_around_operator'), - ('pep8.missing_whitespace_around_operator = ' - 'pep8:missing_whitespace_around_operator'), - 'pep8.whitespace_around_comma = pep8:whitespace_around_comma', - ('pep8.whitespace_around_named_parameter_equals = ' - 'pep8:whitespace_around_named_parameter_equals'), - 'pep8.whitespace_before_comment = pep8:whitespace_before_comment', - 'pep8.imports_on_separate_lines = pep8:imports_on_separate_lines', - ('pep8.module_imports_on_top_of_file = ' - 'pep8:module_imports_on_top_of_file'), - 'pep8.compound_statements = pep8:compound_statements', - 'pep8.explicit_line_join = pep8:explicit_line_join', - ('pep8.break_around_binary_operator = ' - 'pep8:break_around_binary_operator'), - 'pep8.comparison_to_singleton = pep8:comparison_to_singleton', - 'pep8.comparison_negative = pep8:comparison_negative', - 'pep8.comparison_type = pep8:comparison_type', - 'pep8.python_3000_has_key = pep8:python_3000_has_key', - 'pep8.python_3000_raise_comma = pep8:python_3000_raise_comma', - 'pep8.python_3000_not_equal = pep8:python_3000_not_equal', - 'pep8.python_3000_backticks = pep8:python_3000_backticks', + PEP8_PLUGIN('tabs_or_spaces'), + PEP8_PLUGIN('tabs_obsolete'), + PEP8_PLUGIN('trailing_whitespace'), + PEP8_PLUGIN('trailing_blank_lines'), + PEP8_PLUGIN('maximum_line_length'), + PEP8_PLUGIN('blank_lines'), + PEP8_PLUGIN('extraneous_whitespace'), + PEP8_PLUGIN('whitespace_around_keywords'), + PEP8_PLUGIN('missing_whitespace'), + PEP8_PLUGIN('indentation'), + PEP8_PLUGIN('continued_indentation'), + PEP8_PLUGIN('whitespace_before_parameters'), + PEP8_PLUGIN('whitespace_around_operator'), + PEP8_PLUGIN('missing_whitespace_around_operator'), + PEP8_PLUGIN('whitespace_around_comma'), + PEP8_PLUGIN('whitespace_around_named_parameter_equals'), + PEP8_PLUGIN('whitespace_before_comment'), + PEP8_PLUGIN('imports_on_separate_lines'), + PEP8_PLUGIN('module_imports_on_top_of_file'), + PEP8_PLUGIN('compound_statements'), + PEP8_PLUGIN('explicit_line_join'), + PEP8_PLUGIN('break_around_binary_operator'), + PEP8_PLUGIN('comparison_to_singleton'), + PEP8_PLUGIN('comparison_negative'), + PEP8_PLUGIN('comparison_type'), + PEP8_PLUGIN('python_3000_has_key'), + PEP8_PLUGIN('python_3000_raise_comma'), + PEP8_PLUGIN('python_3000_not_equal'), + PEP8_PLUGIN('python_3000_backticks'), ], 'flake8.report': [ 'default = flake8.formatting.default:Default', From 9c47fe3c088c9deff081862ed6514cc8d175b53e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 11 Jun 2016 16:43:41 -0500 Subject: [PATCH 301/364] Convert flake8.api to a submodule --- flake8/{api.py => api/__init__.py} | 0 setup.py | 1 + 2 files changed, 1 insertion(+) rename flake8/{api.py => api/__init__.py} (100%) diff --git a/flake8/api.py b/flake8/api/__init__.py similarity index 100% rename from flake8/api.py rename to flake8/api/__init__.py diff --git a/setup.py b/setup.py index be77c07..6c3ccdc 100644 --- a/setup.py +++ b/setup.py @@ -68,6 +68,7 @@ setuptools.setup( url="https://gitlab.com/pycqa/flake8", packages=[ "flake8", + "flake8.api", "flake8.formatting", "flake8.main", "flake8.options", From 8b9b9bbe89af7064096312c7fcd9296e41409b6a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 13 Jun 2016 07:15:14 -0500 Subject: [PATCH 302/364] Make a first pass at a git pre-commit hook --- flake8/api/git.py | 192 +++++++++++++++++++++++++++++++++++++++++++ flake8/defaults.py | 2 + flake8/exceptions.py | 9 ++ flake8/main/cli.py | 21 +++-- 4 files changed, 219 insertions(+), 5 deletions(-) create mode 100644 flake8/api/git.py diff --git a/flake8/api/git.py b/flake8/api/git.py new file mode 100644 index 0000000..087e31b --- /dev/null +++ b/flake8/api/git.py @@ -0,0 +1,192 @@ +"""Module containing the main git hook interface and helpers. + +.. autofunction:: hook + +""" +import contextlib +import os +import shutil +import stat +import subprocess +import tempdir + +from flake8 import defaults +from flake8 import exceptions +from flake8.main import cli + +__all__ = ('hook', 'install') + + +def hook(lazy=False, strict=False): + """Execute Flake8 on the files in git's index. + + Determine which files are about to be committed and run Flake8 over them + to check for violations. + + :param bool lazy: + Find files not added to the index prior to committing. This is useful + if you frequently use ``git commit -a`` for example. This defaults to + False since it will otherwise include files not in the index. + :param bool strict: + If True, return the total number of errors/violations found by Flake8. + This will cause the hook to fail. + :returns: + Total number of errors found during the run. + :rtype: + int + """ + app = cli.Application() + with make_temporary_directory() as tempdir: + filepaths = list(copy_indexed_files_to(tempdir, lazy)) + app.initialize(filepaths) + app.run_checks() + + app.report_errors() + if strict: + return app.result_count + return 0 + + +def install(): + """Install the git hook script. + + This searches for the ``.git`` directory and will install an executable + pre-commit python script in the hooks sub-directory if one does not + already exist. + + :returns: + True if successful, False if the git directory doesn't exist. + :rtype: + bool + :raises: + flake8.exceptions.GitHookAlreadyExists + """ + git_directory = find_git_directory() + if git_directory is None or not os.path.exists(git_directory): + return False + + hooks_directory = os.path.join(git_directory, 'hooks') + if not os.path.exists(hooks_directory): + os.mkdir(hooks_directory) + + pre_commit_file = os.path.join(hooks_directory, 'hooks', 'pre-commit') + if os.path.exists(pre_commit_file): + raise exceptions.GitHookAlreadyExists( + 'File already exists', + path=pre_commit_file, + ) + + with open(pre_commit_file, 'w') as fd: + fd.write(_HOOK_TEMPLATE) + + # NOTE(sigmavirus24): The following sets: + # - read, write, and execute permissions for the owner + # - read permissions for people in the group + # - read permissions for other people + # The owner needs the file to be readable, writable, and executable + # so that git can actually execute it as a hook. + pre_commit_permissions = stat.S_IRWXU | stat.S_IRGRP | stat.IROTH + os.chmod(pre_commit_file, pre_commit_permissions) + return True + + +def find_git_directory(): + rev_parse = piped_process(['git', 'rev-parse', '--git-dir']) + + (stdout, _) = rev_parse.communicate() + stdout = to_text(stdout) + + if rev_parse.returncode == 0: + return stdout.strip() + return None + + +def copy_indexed_files_to(temporary_directory, lazy): + modified_files = find_modified_files(lazy) + for filename in modified_files: + contents = get_staged_contents_from(filename) + yield copy_file_to(temporary_directory, filename, contents) + + +def copy_file_to(destination_directory, filepath, contents): + directory, filename = os.path.split(os.path.abspath(filepath)) + temporary_directory = make_temporary_directory_from(destination_directory, + directory) + temporary_filepath = os.path.join(temporary_directory, filename) + with open(temporary_filepath, 'wb') as fd: + fd.write(contents) + return temporary_filepath + + +def make_temporary_directory_from(destination, directory): + prefix = os.path.commonprefix([directory, destination]) + common_directory_path = os.path.relpath(directory, start=prefix) + return os.path.join(destination, common_directory_path) + + +def find_modified_files(lazy): + diff_index = piped_process( + ['git', 'diff-index', '--cached', '--name-only', + '--diff-filter=ACMRTUXB', 'HEAD'], + ) + + (stdout, _) = diff_index.communicate() + stdout = to_text(stdout) + return stdout.splitlines() + + +def get_staged_contents_from(filename): + git_show = piped_process(['git', 'show', ':{0}'.format(filename)]) + (stdout, _) = git_show.communicate() + return stdout + + +@contextlib.contextmanager +def make_temporary_directory(): + temporary_directory = tempdir.mkdtemp() + yield temporary_directory + shutil.rmtree(temporary_directory, ignore_errors=True) + + +def to_text(string): + """Ensure that the string is text.""" + if callable(getattr(string, 'decode', None)): + return string.decode('utf-8') + return string + + +def piped_process(command): + return subprocess.Popen( + command, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + +def git_config_for(parameter): + config = piped_process(['git', 'config', '--get', '--bool', parameter]) + (stdout, ) = config.communicate() + return to_text(stdout) + + +def config_for(parameter): + environment_variable = 'flake8_{0}'.format(parameter).upper() + git_variable = 'flake8.{0}'.format(parameter) + value = os.environ.get(environment_variable, git_config_for(git_variable)) + return value.lower() in defaults.TRUTHY_VALUES + + +_HOOK_TEMPLATE = """#!/usr/bin/env python +import os +import sys + +from flake8.api import git + +if __name__ == '__main__': + sys.exit( + git.hook( + strict=git.config_for('strict'), + lazy=git.config_for('lazy'), + ) + ) +""" diff --git a/flake8/defaults.py b/flake8/defaults.py index d484f4c..2007eed 100644 --- a/flake8/defaults.py +++ b/flake8/defaults.py @@ -4,5 +4,7 @@ EXCLUDE = '.svn,CVS,.bzr,.hg,.git,__pycache__,.tox' IGNORE = 'E121,E123,E126,E226,E24,E704,W503,W504' MAX_LINE_LENGTH = 79 +TRUTHY_VALUES = set(['true', '1', 't']) + # Other consants WHITESPACE = frozenset(' \t') diff --git a/flake8/exceptions.py b/flake8/exceptions.py index 18ee90c..c5f3c5a 100644 --- a/flake8/exceptions.py +++ b/flake8/exceptions.py @@ -41,3 +41,12 @@ class InvalidSyntax(Flake8Exception): # strerror attribute instead of a message attribute self.error_message = self.original_exception.strerror super(InvalidSyntax, self).__init__(*args, **kwargs) + + +class GitHookAlreadyExists(Flake8Exception): + """Exception raised when the git pre-commit hook file already exists.""" + + def __init__(self, *args, **kwargs): + """Initialize the path attribute.""" + self.path = kwargs.pop('path') + super(GitHookAlreadyExists, self).__init__(*args, **kwargs) diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 93c4a14..23135c4 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -271,8 +271,11 @@ class Application(object): #: :attr:`option_manager` self.args = None #: The number of errors, warnings, and other messages after running - #: flake8 + #: flake8 and taking into account ignored errors and lines. self.result_count = 0 + #: The total number of errors before accounting for ignored errors and + #: lines. + self.total_result_count = 0 #: Whether the program is processing a diff or not self.running_against_diff = False @@ -406,11 +409,15 @@ class Application(object): results = self.file_checker_manager.report() self.total_result_count, self.result_count = results LOG.info('Found a total of %d results and reported %d', - self.total_result_count, - self.result_count) + self.total_result_count, self.result_count) - def _run(self, argv): - # type: (Union[NoneType, List[str]]) -> NoneType + def initialize(self, argv): + # type: () -> NoneType + """Initialize the application to be run. + + This finds the plugins, registers their options, and parses the + command-line arguments. + """ self.find_plugins() self.register_plugin_options() self.parse_configuration_and_cli(argv) @@ -418,6 +425,10 @@ class Application(object): self.make_notifier() self.make_guide() self.make_file_checker_manager() + + def _run(self, argv): + # type: (Union[NoneType, List[str]]) -> NoneType + self.initialize(argv) self.run_checks() self.report_errors() From 6f08d9d4a44dbab667f28c940698176950b0e59c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 13 Jun 2016 10:33:15 -0500 Subject: [PATCH 303/364] Move git integration into flake8.main Also add the --install-hook option and plumb it's installation through flake8.main.vcs's function that understands how to install the desired VCS integration bits. Finally, we also mock out the mercurial integration. --- flake8/main/cli.py | 24 +++++++++++++++++++----- flake8/{api => main}/git.py | 3 ++- flake8/main/mercurial.py | 16 ++++++++++++++++ flake8/main/vcs.py | 27 +++++++++++++++++++++++++++ 4 files changed, 64 insertions(+), 6 deletions(-) rename flake8/{api => main}/git.py (99%) create mode 100644 flake8/main/mercurial.py create mode 100644 flake8/main/vcs.py diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 23135c4..531527d 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -9,6 +9,7 @@ from flake8 import checker from flake8 import defaults from flake8 import style_guide from flake8 import utils +from flake8.main import vcs from flake8.options import aggregator from flake8.options import manager from flake8.plugins import manager as plugin_manager @@ -16,7 +17,7 @@ from flake8.plugins import manager as plugin_manager LOG = logging.getLogger(__name__) -def register_default_options(option_manager): +def register_default_options(option_manager, formatters=None): """Register the default options on our OptionManager. The default options include: @@ -97,7 +98,7 @@ def register_default_options(option_manager): add_option( '--format', metavar='format', default='default', - parse_from_config=True, + parse_from_config=True, choices=(formatters or []), help='Format errors according to the chosen formatter.', ) @@ -160,6 +161,13 @@ def register_default_options(option_manager): help='Exit with status code "0" even if there are errors.', ) + add_option( + '--install-hook', action='callback', type='string', + choices=vcs.choices(), callback=vcs.install, + help='Install a hook that is run prior to a commit for the supported ' + 'version control systema.' + ) + add_option( '-j', '--jobs', type='string', default='auto', parse_from_config=True, help='Number of subprocesses to use to run checks in parallel. ' @@ -215,10 +223,11 @@ class Application(object): self.version = version #: The instance of :class:`flake8.options.manager.OptionManager` used #: to parse and handle the options and arguments passed by the user - self.option_manager = manager.OptionManager( + self.option_manager = None + temp_option_manager = manager.OptionManager( prog='flake8', version=flake8.__version__ ) - register_default_options(self.option_manager) + register_default_options(temp_option_manager) # We haven't found or registered our plugins yet, so let's defer # printing the version until we aggregate options from config files @@ -242,7 +251,7 @@ class Application(object): except ValueError: pass - preliminary_opts, _ = self.option_manager.parse_args(args) + preliminary_opts, _ = temp_option_manager.parse_args(args) # Set the verbosity of the program flake8.configure_logging(preliminary_opts.verbose, preliminary_opts.output_file) @@ -418,7 +427,12 @@ class Application(object): This finds the plugins, registers their options, and parses the command-line arguments. """ + self.option_manager = manager.OptionManager( + prog='flake8', version=flake8.__version__ + ) self.find_plugins() + register_default_options(self.option_manager, + self.formatting_plugins.names) self.register_plugin_options() self.parse_configuration_and_cli(argv) self.make_formatter() diff --git a/flake8/api/git.py b/flake8/main/git.py similarity index 99% rename from flake8/api/git.py rename to flake8/main/git.py index 087e31b..61312d4 100644 --- a/flake8/api/git.py +++ b/flake8/main/git.py @@ -1,6 +1,7 @@ """Module containing the main git hook interface and helpers. .. autofunction:: hook +.. autofunction:: install """ import contextlib @@ -180,7 +181,7 @@ _HOOK_TEMPLATE = """#!/usr/bin/env python import os import sys -from flake8.api import git +from flake8.main import git if __name__ == '__main__': sys.exit( diff --git a/flake8/main/mercurial.py b/flake8/main/mercurial.py new file mode 100644 index 0000000..9767f34 --- /dev/null +++ b/flake8/main/mercurial.py @@ -0,0 +1,16 @@ +"""Module containing the main mecurial hook interface and helpers. + +.. autofunction:: hook +.. autofunction:: install + +""" + +__all__ = ('hook', 'install') + + +def hook(lazy=False, strict=False): + pass + + +def install(): + pass diff --git a/flake8/main/vcs.py b/flake8/main/vcs.py new file mode 100644 index 0000000..95cbfa1 --- /dev/null +++ b/flake8/main/vcs.py @@ -0,0 +1,27 @@ +"""Module containing some of the logic for our VCS installation logic.""" +from flake8.main import git +from flake8.main import mercurial + + +# NOTE(sigmavirus24): In the future, we may allow for VCS hooks to be defined +# as plugins, e.g., adding a flake8.vcs entry-point. In that case, this +# dictionary should disappear, and this module might contain more code for +# managing those bits (in conjuntion with flake8.plugins.manager). +_INSTALLERS = { + 'git': git.install, + 'mercurial': mercurial.install, +} + + +def install(option, option_string, value, parser): + """Determine which version control hook to install. + + For more information about the callback signature, see: + https://docs.python.org/2/library/optparse.html#optparse-option-callbacks + """ + installer = _INSTALLERS.get(value) + installer() + + +def choices(): + return _INSTALLERS.keys() From de587a9ccbaf140c556aac457e0179f37391b74e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 13 Jun 2016 10:43:46 -0500 Subject: [PATCH 304/364] Correct usage of tempfile The stdlib module is tempfile not tempdir ;-) --- flake8/main/git.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/flake8/main/git.py b/flake8/main/git.py index 61312d4..c24f294 100644 --- a/flake8/main/git.py +++ b/flake8/main/git.py @@ -9,7 +9,7 @@ import os import shutil import stat import subprocess -import tempdir +import tempfile from flake8 import defaults from flake8 import exceptions @@ -144,7 +144,7 @@ def get_staged_contents_from(filename): @contextlib.contextmanager def make_temporary_directory(): - temporary_directory = tempdir.mkdtemp() + temporary_directory = tempfile.mkdtemp() yield temporary_directory shutil.rmtree(temporary_directory, ignore_errors=True) From 53438fca1db94a57ba43c5217f4fe937208d33a9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 13 Jun 2016 20:36:22 -0500 Subject: [PATCH 305/364] Refactor parts of flake8.main.cli into separate modules --- flake8/main/application.py | 277 ++++++++++++++++++++++ flake8/main/cli.py | 464 +------------------------------------ flake8/main/git.py | 5 +- flake8/main/options.py | 190 +++++++++++++++ 4 files changed, 472 insertions(+), 464 deletions(-) create mode 100644 flake8/main/application.py create mode 100644 flake8/main/options.py diff --git a/flake8/main/application.py b/flake8/main/application.py new file mode 100644 index 0000000..d776bbf --- /dev/null +++ b/flake8/main/application.py @@ -0,0 +1,277 @@ +"""Module containing the application logic for Flake8.""" +from __future__ import print_function + +import logging +import sys + +import flake8 +from flake8 import checker +from flake8 import style_guide +from flake8 import utils +from flake8.main import options +from flake8.options import aggregator +from flake8.options import manager +from flake8.plugins import manager as plugin_manager + +LOG = logging.getLogger(__name__) + + +class Application(object): + """Abstract our application into a class.""" + + def __init__(self, program='flake8', version=flake8.__version__): + # type: (str, str) -> NoneType + """Initialize our application. + + :param str program: + The name of the program/application that we're executing. + :param str version: + The version of the program/application we're executing. + """ + #: The name of the program being run + self.program = program + #: The version of the program being run + self.version = version + #: The instance of :class:`flake8.options.manager.OptionManager` used + #: to parse and handle the options and arguments passed by the user + self.option_manager = None + temp_option_manager = manager.OptionManager( + prog='flake8', version=flake8.__version__ + ) + options.register_default_options(temp_option_manager) + + # We haven't found or registered our plugins yet, so let's defer + # printing the version until we aggregate options from config files + # and the command-line. First, let's clone our arguments on the CLI, + # then we'll attempt to remove ``--version`` so that we can avoid + # triggering the "version" action in optparse. If it's not there, we + # do not need to worry and we can continue. If it is, we successfully + # defer printing the version until just a little bit later. + # Similarly we have to defer printing the help text until later. + args = sys.argv[:] + try: + args.remove('--version') + except ValueError: + pass + try: + args.remove('--help') + except ValueError: + pass + try: + args.remove('-h') + except ValueError: + pass + + preliminary_opts, _ = temp_option_manager.parse_args(args) + # Set the verbosity of the program + flake8.configure_logging(preliminary_opts.verbose, + preliminary_opts.output_file) + + #: The instance of :class:`flake8.plugins.manager.Checkers` + self.check_plugins = None + #: The instance of :class:`flake8.plugins.manager.Listeners` + self.listening_plugins = None + #: The instance of :class:`flake8.plugins.manager.ReportFormatters` + self.formatting_plugins = None + #: The user-selected formatter from :attr:`formatting_plugins` + self.formatter = None + #: The :class:`flake8.plugins.notifier.Notifier` for listening plugins + self.listener_trie = None + #: The :class:`flake8.style_guide.StyleGuide` built from the user's + #: options + self.guide = None + #: The :class:`flake8.checker.Manager` that will handle running all of + #: the checks selected by the user. + self.file_checker_manager = None + + #: The user-supplied options parsed into an instance of + #: :class:`optparse.Values` + self.options = None + #: The left over arguments that were not parsed by + #: :attr:`option_manager` + self.args = None + #: The number of errors, warnings, and other messages after running + #: flake8 and taking into account ignored errors and lines. + self.result_count = 0 + #: The total number of errors before accounting for ignored errors and + #: lines. + self.total_result_count = 0 + + #: Whether the program is processing a diff or not + self.running_against_diff = False + #: The parsed diff information + self.parsed_diff = {} + + def exit(self): + # type: () -> NoneType + """Handle finalization and exiting the program. + + This should be the last thing called on the application instance. It + will check certain options and exit appropriately. + """ + if self.options.count: + print(self.result_count) + + if not self.options.exit_zero: + raise SystemExit(self.result_count > 0) + + def find_plugins(self): + # type: () -> NoneType + """Find and load the plugins for this application. + + If :attr:`check_plugins`, :attr:`listening_plugins`, or + :attr:`formatting_plugins` are ``None`` then this method will update + them with the appropriate plugin manager instance. Given the expense + of finding plugins (via :mod:`pkg_resources`) we want this to be + idempotent and so only update those attributes if they are ``None``. + """ + if self.check_plugins is None: + self.check_plugins = plugin_manager.Checkers() + + if self.listening_plugins is None: + self.listening_plugins = plugin_manager.Listeners() + + if self.formatting_plugins is None: + self.formatting_plugins = plugin_manager.ReportFormatters() + + self.check_plugins.load_plugins() + self.listening_plugins.load_plugins() + self.formatting_plugins.load_plugins() + + def register_plugin_options(self): + # type: () -> NoneType + """Register options provided by plugins to our option manager.""" + self.check_plugins.register_options(self.option_manager) + self.check_plugins.register_plugin_versions(self.option_manager) + self.listening_plugins.register_options(self.option_manager) + self.formatting_plugins.register_options(self.option_manager) + + def parse_configuration_and_cli(self, argv=None): + # type: (Union[NoneType, List[str]]) -> NoneType + """Parse configuration files and the CLI options. + + :param list argv: + Command-line arguments passed in directly. + """ + if self.options is None and self.args is None: + self.options, self.args = aggregator.aggregate_options( + self.option_manager, argv + ) + + self.running_against_diff = self.options.diff + if self.running_against_diff: + self.parsed_diff = utils.parse_unified_diff() + + self.check_plugins.provide_options(self.option_manager, self.options, + self.args) + self.listening_plugins.provide_options(self.option_manager, + self.options, + self.args) + self.formatting_plugins.provide_options(self.option_manager, + self.options, + self.args) + + def make_formatter(self): + # type: () -> NoneType + """Initialize a formatter based on the parsed options.""" + if self.formatter is None: + self.formatter = self.formatting_plugins.get( + self.options.format, self.formatting_plugins['default'] + ).execute(self.options) + + def make_notifier(self): + # type: () -> NoneType + """Initialize our listener Notifier.""" + if self.listener_trie is None: + self.listener_trie = self.listening_plugins.build_notifier() + + def make_guide(self): + # type: () -> NoneType + """Initialize our StyleGuide.""" + if self.guide is None: + self.guide = style_guide.StyleGuide( + self.options, self.listener_trie, self.formatter + ) + + if self.running_against_diff: + self.guide.add_diff_ranges(self.parsed_diff) + + def make_file_checker_manager(self): + # type: () -> NoneType + """Initialize our FileChecker Manager.""" + if self.file_checker_manager is None: + self.file_checker_manager = checker.Manager( + style_guide=self.guide, + arguments=self.args, + checker_plugins=self.check_plugins, + ) + + def run_checks(self): + # type: () -> NoneType + """Run the actual checks with the FileChecker Manager. + + This method encapsulates the logic to make a + :class:`~flake8.checker.Manger` instance run the checks it is + managing. + """ + files = None + if self.running_against_diff: + files = list(sorted(self.parsed_diff.keys())) + self.file_checker_manager.start(files) + self.file_checker_manager.run() + LOG.info('Finished running') + self.file_checker_manager.stop() + + def report_errors(self): + # type: () -> NoneType + """Report all the errors found by flake8 3.0. + + This also updates the :attr:`result_count` attribute with the total + number of errors, warnings, and other messages found. + """ + LOG.info('Reporting errors') + results = self.file_checker_manager.report() + self.total_result_count, self.result_count = results + LOG.info('Found a total of %d results and reported %d', + self.total_result_count, self.result_count) + + def initialize(self, argv): + # type: () -> NoneType + """Initialize the application to be run. + + This finds the plugins, registers their options, and parses the + command-line arguments. + """ + self.option_manager = manager.OptionManager( + prog='flake8', version=flake8.__version__ + ) + self.find_plugins() + options.register_default_options(self.option_manager, + self.formatting_plugins.names) + self.register_plugin_options() + self.parse_configuration_and_cli(argv) + self.make_formatter() + self.make_notifier() + self.make_guide() + self.make_file_checker_manager() + + def _run(self, argv): + # type: (Union[NoneType, List[str]]) -> NoneType + self.initialize(argv) + self.run_checks() + self.report_errors() + + def run(self, argv=None): + # type: (Union[NoneType, List[str]]) -> NoneType + """Run our application. + + This method will also handle KeyboardInterrupt exceptions for the + entirety of the flake8 application. If it sees a KeyboardInterrupt it + will forcibly clean up the :class:`~flake8.checker.Manager`. + """ + try: + self._run(argv) + except KeyboardInterrupt as exc: + LOG.critical('Caught keyboard interrupt from user') + LOG.exception(exc) + self.file_checker_manager._force_cleanup() diff --git a/flake8/main/cli.py b/flake8/main/cli.py index 531527d..29bd159 100644 --- a/flake8/main/cli.py +++ b/flake8/main/cli.py @@ -1,465 +1,5 @@ """Command-line implementation of flake8.""" -from __future__ import print_function - -import logging -import sys - -import flake8 -from flake8 import checker -from flake8 import defaults -from flake8 import style_guide -from flake8 import utils -from flake8.main import vcs -from flake8.options import aggregator -from flake8.options import manager -from flake8.plugins import manager as plugin_manager - -LOG = logging.getLogger(__name__) - - -def register_default_options(option_manager, formatters=None): - """Register the default options on our OptionManager. - - The default options include: - - - ``-v``/``--verbose`` - - ``-q``/``--quiet`` - - ``--count`` - - ``--diff`` - - ``--exclude`` - - ``--filename`` - - ``--format`` - - ``--hang-closing`` - - ``--ignore`` - - ``--max-line-length`` - - ``--select`` - - ``--disable-noqa`` - - ``--show-source`` - - ``--statistics`` - - ``--enable-extensions`` - - ``--exit-zero`` - - ``-j``/``--jobs`` - - ``--output-file`` - - ``--append-config`` - - ``--config`` - - ``--isolated`` - """ - add_option = option_manager.add_option - - # pep8 options - add_option( - '-v', '--verbose', default=0, action='count', - parse_from_config=True, - help='Print more information about what is happening in flake8.' - ' This option is repeatable and will increase verbosity each ' - 'time it is repeated.', - ) - add_option( - '-q', '--quiet', default=0, action='count', - parse_from_config=True, - help='Report only file names, or nothing. This option is repeatable.', - ) - - add_option( - '--count', action='store_true', parse_from_config=True, - help='Print total number of errors and warnings to standard error and' - ' set the exit code to 1 if total is not empty.', - ) - - add_option( - '--diff', action='store_true', - help='Report changes only within line number ranges in the unified ' - 'diff provided on standard in by the user.', - ) - - add_option( - '--exclude', metavar='patterns', default=defaults.EXCLUDE, - comma_separated_list=True, parse_from_config=True, - normalize_paths=True, - help='Comma-separated list of files or directories to exclude.' - ' (Default: %default)', - ) - - add_option( - '--filename', metavar='patterns', default='*.py', - parse_from_config=True, comma_separated_list=True, - help='Only check for filenames matching the patterns in this comma-' - 'separated list. (Default: %default)', - ) - - add_option( - '--stdin-display-name', default='stdin', - help='The name used when reporting errors from code passed via stdin.' - ' This is useful for editors piping the file contents to flake8.' - ' (Default: %default)', - ) - - # TODO(sigmavirus24): Figure out --first/--repeat - - add_option( - '--format', metavar='format', default='default', - parse_from_config=True, choices=(formatters or []), - help='Format errors according to the chosen formatter.', - ) - - add_option( - '--hang-closing', action='store_true', parse_from_config=True, - help='Hang closing bracket instead of matching indentation of opening' - " bracket's line.", - ) - - add_option( - '--ignore', metavar='errors', default=defaults.IGNORE, - parse_from_config=True, comma_separated_list=True, - help='Comma-separated list of errors and warnings to ignore (or skip).' - ' For example, ``--ignore=E4,E51,W234``. (Default: %default)', - ) - - add_option( - '--max-line-length', type='int', metavar='n', - default=defaults.MAX_LINE_LENGTH, parse_from_config=True, - help='Maximum allowed line length for the entirety of this run. ' - '(Default: %default)', - ) - - add_option( - '--select', metavar='errors', default='', - parse_from_config=True, comma_separated_list=True, - help='Comma-separated list of errors and warnings to enable.' - ' For example, ``--select=E4,E51,W234``. (Default: %default)', - ) - - add_option( - '--disable-noqa', default=False, parse_from_config=True, - action='store_true', - help='Disable the effect of "# noqa". This will report errors on ' - 'lines with "# noqa" at the end.' - ) - - # TODO(sigmavirus24): Decide what to do about --show-pep8 - - add_option( - '--show-source', action='store_true', parse_from_config=True, - help='Show the source generate each error or warning.', - ) - - add_option( - '--statistics', action='store_true', parse_from_config=True, - help='Count errors and warnings.', - ) - - # Flake8 options - add_option( - '--enable-extensions', default='', parse_from_config=True, - comma_separated_list=True, type='string', - help='Enable plugins and extensions that are otherwise disabled ' - 'by default', - ) - - add_option( - '--exit-zero', action='store_true', - help='Exit with status code "0" even if there are errors.', - ) - - add_option( - '--install-hook', action='callback', type='string', - choices=vcs.choices(), callback=vcs.install, - help='Install a hook that is run prior to a commit for the supported ' - 'version control systema.' - ) - - add_option( - '-j', '--jobs', type='string', default='auto', parse_from_config=True, - help='Number of subprocesses to use to run checks in parallel. ' - 'This is ignored on Windows. The default, "auto", will ' - 'auto-detect the number of processors available to use.' - ' (Default: %default)', - ) - - add_option( - '--output-file', default=None, type='string', parse_from_config=True, - # callback=callbacks.redirect_stdout, - help='Redirect report to a file.', - ) - - # Config file options - - add_option( - '--append-config', action='append', - help='Provide extra config files to parse in addition to the files ' - 'found by Flake8 by default. These files are the last ones read ' - 'and so they take the highest precedence when multiple files ' - 'provide the same option.', - ) - - add_option( - '--config', default=None, - help='Path to the config file that will be the authoritative config ' - 'source. This will cause Flake8 to ignore all other ' - 'configuration files.' - ) - - add_option( - '--isolated', default=False, action='store_true', - help='Ignore all found configuration files.', - ) - - -class Application(object): - """Abstract our application into a class.""" - - def __init__(self, program='flake8', version=flake8.__version__): - # type: (str, str) -> NoneType - """Initialize our application. - - :param str program: - The name of the program/application that we're executing. - :param str version: - The version of the program/application we're executing. - """ - #: The name of the program being run - self.program = program - #: The version of the program being run - self.version = version - #: The instance of :class:`flake8.options.manager.OptionManager` used - #: to parse and handle the options and arguments passed by the user - self.option_manager = None - temp_option_manager = manager.OptionManager( - prog='flake8', version=flake8.__version__ - ) - register_default_options(temp_option_manager) - - # We haven't found or registered our plugins yet, so let's defer - # printing the version until we aggregate options from config files - # and the command-line. First, let's clone our arguments on the CLI, - # then we'll attempt to remove ``--version`` so that we can avoid - # triggering the "version" action in optparse. If it's not there, we - # do not need to worry and we can continue. If it is, we successfully - # defer printing the version until just a little bit later. - # Similarly we have to defer printing the help text until later. - args = sys.argv[:] - try: - args.remove('--version') - except ValueError: - pass - try: - args.remove('--help') - except ValueError: - pass - try: - args.remove('-h') - except ValueError: - pass - - preliminary_opts, _ = temp_option_manager.parse_args(args) - # Set the verbosity of the program - flake8.configure_logging(preliminary_opts.verbose, - preliminary_opts.output_file) - - #: The instance of :class:`flake8.plugins.manager.Checkers` - self.check_plugins = None - #: The instance of :class:`flake8.plugins.manager.Listeners` - self.listening_plugins = None - #: The instance of :class:`flake8.plugins.manager.ReportFormatters` - self.formatting_plugins = None - #: The user-selected formatter from :attr:`formatting_plugins` - self.formatter = None - #: The :class:`flake8.plugins.notifier.Notifier` for listening plugins - self.listener_trie = None - #: The :class:`flake8.style_guide.StyleGuide` built from the user's - #: options - self.guide = None - #: The :class:`flake8.checker.Manager` that will handle running all of - #: the checks selected by the user. - self.file_checker_manager = None - - #: The user-supplied options parsed into an instance of - #: :class:`optparse.Values` - self.options = None - #: The left over arguments that were not parsed by - #: :attr:`option_manager` - self.args = None - #: The number of errors, warnings, and other messages after running - #: flake8 and taking into account ignored errors and lines. - self.result_count = 0 - #: The total number of errors before accounting for ignored errors and - #: lines. - self.total_result_count = 0 - - #: Whether the program is processing a diff or not - self.running_against_diff = False - #: The parsed diff information - self.parsed_diff = {} - - def exit(self): - # type: () -> NoneType - """Handle finalization and exiting the program. - - This should be the last thing called on the application instance. It - will check certain options and exit appropriately. - """ - if self.options.count: - print(self.result_count) - - if not self.options.exit_zero: - raise SystemExit(self.result_count > 0) - - def find_plugins(self): - # type: () -> NoneType - """Find and load the plugins for this application. - - If :attr:`check_plugins`, :attr:`listening_plugins`, or - :attr:`formatting_plugins` are ``None`` then this method will update - them with the appropriate plugin manager instance. Given the expense - of finding plugins (via :mod:`pkg_resources`) we want this to be - idempotent and so only update those attributes if they are ``None``. - """ - if self.check_plugins is None: - self.check_plugins = plugin_manager.Checkers() - - if self.listening_plugins is None: - self.listening_plugins = plugin_manager.Listeners() - - if self.formatting_plugins is None: - self.formatting_plugins = plugin_manager.ReportFormatters() - - def register_plugin_options(self): - # type: () -> NoneType - """Register options provided by plugins to our option manager.""" - self.check_plugins.register_options(self.option_manager) - self.check_plugins.register_plugin_versions(self.option_manager) - self.listening_plugins.register_options(self.option_manager) - self.formatting_plugins.register_options(self.option_manager) - - def parse_configuration_and_cli(self, argv=None): - # type: (Union[NoneType, List[str]]) -> NoneType - """Parse configuration files and the CLI options. - - :param list argv: - Command-line arguments passed in directly. - """ - if self.options is None and self.args is None: - self.options, self.args = aggregator.aggregate_options( - self.option_manager, argv - ) - - self.running_against_diff = self.options.diff - if self.running_against_diff: - self.parsed_diff = utils.parse_unified_diff() - - self.check_plugins.provide_options(self.option_manager, self.options, - self.args) - self.listening_plugins.provide_options(self.option_manager, - self.options, - self.args) - self.formatting_plugins.provide_options(self.option_manager, - self.options, - self.args) - - def make_formatter(self): - # type: () -> NoneType - """Initialize a formatter based on the parsed options.""" - if self.formatter is None: - self.formatter = self.formatting_plugins.get( - self.options.format, self.formatting_plugins['default'] - ).execute(self.options) - - def make_notifier(self): - # type: () -> NoneType - """Initialize our listener Notifier.""" - if self.listener_trie is None: - self.listener_trie = self.listening_plugins.build_notifier() - - def make_guide(self): - # type: () -> NoneType - """Initialize our StyleGuide.""" - if self.guide is None: - self.guide = style_guide.StyleGuide( - self.options, self.listener_trie, self.formatter - ) - - if self.running_against_diff: - self.guide.add_diff_ranges(self.parsed_diff) - - def make_file_checker_manager(self): - # type: () -> NoneType - """Initialize our FileChecker Manager.""" - if self.file_checker_manager is None: - self.file_checker_manager = checker.Manager( - style_guide=self.guide, - arguments=self.args, - checker_plugins=self.check_plugins, - ) - - def run_checks(self): - # type: () -> NoneType - """Run the actual checks with the FileChecker Manager. - - This method encapsulates the logic to make a - :class:`~flake8.checker.Manger` instance run the checks it is - managing. - """ - files = None - if self.running_against_diff: - files = list(sorted(self.parsed_diff.keys())) - self.file_checker_manager.start(files) - self.file_checker_manager.run() - LOG.info('Finished running') - self.file_checker_manager.stop() - - def report_errors(self): - # type: () -> NoneType - """Report all the errors found by flake8 3.0. - - This also updates the :attr:`result_count` attribute with the total - number of errors, warnings, and other messages found. - """ - LOG.info('Reporting errors') - results = self.file_checker_manager.report() - self.total_result_count, self.result_count = results - LOG.info('Found a total of %d results and reported %d', - self.total_result_count, self.result_count) - - def initialize(self, argv): - # type: () -> NoneType - """Initialize the application to be run. - - This finds the plugins, registers their options, and parses the - command-line arguments. - """ - self.option_manager = manager.OptionManager( - prog='flake8', version=flake8.__version__ - ) - self.find_plugins() - register_default_options(self.option_manager, - self.formatting_plugins.names) - self.register_plugin_options() - self.parse_configuration_and_cli(argv) - self.make_formatter() - self.make_notifier() - self.make_guide() - self.make_file_checker_manager() - - def _run(self, argv): - # type: (Union[NoneType, List[str]]) -> NoneType - self.initialize(argv) - self.run_checks() - self.report_errors() - - def run(self, argv=None): - # type: (Union[NoneType, List[str]]) -> NoneType - """Run our application. - - This method will also handle KeyboardInterrupt exceptions for the - entirety of the flake8 application. If it sees a KeyboardInterrupt it - will forcibly clean up the :class:`~flake8.checker.Manager`. - """ - try: - self._run(argv) - except KeyboardInterrupt as exc: - LOG.critical('Caught keyboard interrupt from user') - LOG.exception(exc) - self.file_checker_manager._force_cleanup() +from flake8.main import application def main(argv=None): @@ -472,6 +12,6 @@ def main(argv=None): :param list argv: The arguments to be passed to the application for parsing. """ - app = Application() + app = application.Application() app.run(argv) app.exit() diff --git a/flake8/main/git.py b/flake8/main/git.py index c24f294..6e9aead 100644 --- a/flake8/main/git.py +++ b/flake8/main/git.py @@ -13,7 +13,6 @@ import tempfile from flake8 import defaults from flake8 import exceptions -from flake8.main import cli __all__ = ('hook', 'install') @@ -36,7 +35,9 @@ def hook(lazy=False, strict=False): :rtype: int """ - app = cli.Application() + # NOTE(sigmavirus24): Delay import of application until we need it. + from flake8.main import application + app = application.Application() with make_temporary_directory() as tempdir: filepaths = list(copy_indexed_files_to(tempdir, lazy)) app.initialize(filepaths) diff --git a/flake8/main/options.py b/flake8/main/options.py new file mode 100644 index 0000000..a12276c --- /dev/null +++ b/flake8/main/options.py @@ -0,0 +1,190 @@ +from flake8 import defaults +from flake8.main import vcs + + +def register_default_options(option_manager, formatters=None): + """Register the default options on our OptionManager. + + The default options include: + + - ``-v``/``--verbose`` + - ``-q``/``--quiet`` + - ``--count`` + - ``--diff`` + - ``--exclude`` + - ``--filename`` + - ``--format`` + - ``--hang-closing`` + - ``--ignore`` + - ``--max-line-length`` + - ``--select`` + - ``--disable-noqa`` + - ``--show-source`` + - ``--statistics`` + - ``--enable-extensions`` + - ``--exit-zero`` + - ``-j``/``--jobs`` + - ``--output-file`` + - ``--append-config`` + - ``--config`` + - ``--isolated`` + """ + add_option = option_manager.add_option + + # pep8 options + add_option( + '-v', '--verbose', default=0, action='count', + parse_from_config=True, + help='Print more information about what is happening in flake8.' + ' This option is repeatable and will increase verbosity each ' + 'time it is repeated.', + ) + add_option( + '-q', '--quiet', default=0, action='count', + parse_from_config=True, + help='Report only file names, or nothing. This option is repeatable.', + ) + + add_option( + '--count', action='store_true', parse_from_config=True, + help='Print total number of errors and warnings to standard error and' + ' set the exit code to 1 if total is not empty.', + ) + + add_option( + '--diff', action='store_true', + help='Report changes only within line number ranges in the unified ' + 'diff provided on standard in by the user.', + ) + + add_option( + '--exclude', metavar='patterns', default=defaults.EXCLUDE, + comma_separated_list=True, parse_from_config=True, + normalize_paths=True, + help='Comma-separated list of files or directories to exclude.' + ' (Default: %default)', + ) + + add_option( + '--filename', metavar='patterns', default='*.py', + parse_from_config=True, comma_separated_list=True, + help='Only check for filenames matching the patterns in this comma-' + 'separated list. (Default: %default)', + ) + + add_option( + '--stdin-display-name', default='stdin', + help='The name used when reporting errors from code passed via stdin.' + ' This is useful for editors piping the file contents to flake8.' + ' (Default: %default)', + ) + + # TODO(sigmavirus24): Figure out --first/--repeat + + add_option( + '--format', metavar='format', default='default', + parse_from_config=True, choices=(formatters or ['default']), + help='Format errors according to the chosen formatter.', + ) + + add_option( + '--hang-closing', action='store_true', parse_from_config=True, + help='Hang closing bracket instead of matching indentation of opening' + " bracket's line.", + ) + + add_option( + '--ignore', metavar='errors', default=defaults.IGNORE, + parse_from_config=True, comma_separated_list=True, + help='Comma-separated list of errors and warnings to ignore (or skip).' + ' For example, ``--ignore=E4,E51,W234``. (Default: %default)', + ) + + add_option( + '--max-line-length', type='int', metavar='n', + default=defaults.MAX_LINE_LENGTH, parse_from_config=True, + help='Maximum allowed line length for the entirety of this run. ' + '(Default: %default)', + ) + + add_option( + '--select', metavar='errors', default='', + parse_from_config=True, comma_separated_list=True, + help='Comma-separated list of errors and warnings to enable.' + ' For example, ``--select=E4,E51,W234``. (Default: %default)', + ) + + add_option( + '--disable-noqa', default=False, parse_from_config=True, + action='store_true', + help='Disable the effect of "# noqa". This will report errors on ' + 'lines with "# noqa" at the end.' + ) + + # TODO(sigmavirus24): Decide what to do about --show-pep8 + + add_option( + '--show-source', action='store_true', parse_from_config=True, + help='Show the source generate each error or warning.', + ) + + add_option( + '--statistics', action='store_true', parse_from_config=True, + help='Count errors and warnings.', + ) + + # Flake8 options + add_option( + '--enable-extensions', default='', parse_from_config=True, + comma_separated_list=True, type='string', + help='Enable plugins and extensions that are otherwise disabled ' + 'by default', + ) + + add_option( + '--exit-zero', action='store_true', + help='Exit with status code "0" even if there are errors.', + ) + + add_option( + '--install-hook', action='callback', type='choice', + choices=vcs.choices(), callback=vcs.install, + help='Install a hook that is run prior to a commit for the supported ' + 'version control systema.' + ) + + add_option( + '-j', '--jobs', type='string', default='auto', parse_from_config=True, + help='Number of subprocesses to use to run checks in parallel. ' + 'This is ignored on Windows. The default, "auto", will ' + 'auto-detect the number of processors available to use.' + ' (Default: %default)', + ) + + add_option( + '--output-file', default=None, type='string', parse_from_config=True, + # callback=callbacks.redirect_stdout, + help='Redirect report to a file.', + ) + + # Config file options + + add_option( + '--append-config', action='append', + help='Provide extra config files to parse in addition to the files ' + 'found by Flake8 by default. These files are the last ones read ' + 'and so they take the highest precedence when multiple files ' + 'provide the same option.', + ) + + add_option( + '--config', default=None, + help='Path to the config file that will be the authoritative config ' + 'source. This will cause Flake8 to ignore all other ' + 'configuration files.' + ) + + add_option( + '--isolated', default=False, action='store_true', + help='Ignore all found configuration files.', + ) From 6dca1d6fd74c61be16d95bd37dab58cc5e91d033 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 13 Jun 2016 20:36:44 -0500 Subject: [PATCH 306/364] Start fleshing out mercurial hook This also reports errors with hook installation using exceptions and includes far more exceptions for mercurial since it has more failure modes. --- flake8/exceptions.py | 44 +++++++++++++++++- flake8/main/git.py | 4 +- flake8/main/mercurial.py | 99 ++++++++++++++++++++++++++++++++++++++-- flake8/main/vcs.py | 12 ++++- 4 files changed, 152 insertions(+), 7 deletions(-) diff --git a/flake8/exceptions.py b/flake8/exceptions.py index c5f3c5a..e709ff5 100644 --- a/flake8/exceptions.py +++ b/flake8/exceptions.py @@ -43,10 +43,52 @@ class InvalidSyntax(Flake8Exception): super(InvalidSyntax, self).__init__(*args, **kwargs) -class GitHookAlreadyExists(Flake8Exception): +class HookInstallationError(Flake8Exception): + """Parent exception for all hooks errors.""" + pass + + +class GitHookAlreadyExists(HookInstallationError): """Exception raised when the git pre-commit hook file already exists.""" def __init__(self, *args, **kwargs): """Initialize the path attribute.""" self.path = kwargs.pop('path') super(GitHookAlreadyExists, self).__init__(*args, **kwargs) + + def __str__(self): + """Provide a nice message regarding the exception.""" + msg = ('The Git pre-commit hook ({0}) already exists. To convince ' + 'Flake8 to install the hook, please remove the existing ' + 'hook.') + return msg.format(self.path) + + +class MercurialHookAlreadyExists(HookInstallationError): + """Exception raised when a mercurial hook is already configured.""" + + hook_name = None + + def __init__(self, *args, **kwargs): + """Initialize the relevant attributes.""" + self.path = kwargs.pop('path') + self.value = kwargs.pop('value') + super(MercurialHookAlreadyExists, self).__init__(*args, **kwargs) + + def __str__(self): + msg = ('The Mercurial {0} hook already exists with "{1}" in {2}. ' + 'To convince Flake8 to install the hook, please remove the ' + '{0} configuration from the [hooks] section of your hgrc.') + return msg.format(self.hook_name, self.value, self.path) + + +class MercurialCommitHookAlreadyExists(MercurialHookAlreadyExists): + """Exception raised when the hg commit hook is already configured.""" + + hook_name = 'commit' + + +class MercurialQRefreshHookAlreadyExists(MercurialHookAlreadyExists): + """Exception raised when the hg commit hook is already configured.""" + + hook_name = 'qrefresh' diff --git a/flake8/main/git.py b/flake8/main/git.py index 6e9aead..7e55321 100644 --- a/flake8/main/git.py +++ b/flake8/main/git.py @@ -71,7 +71,9 @@ def install(): if not os.path.exists(hooks_directory): os.mkdir(hooks_directory) - pre_commit_file = os.path.join(hooks_directory, 'hooks', 'pre-commit') + pre_commit_file = os.path.abspath( + os.path.join(hooks_directory, 'pre-commit') + ) if os.path.exists(pre_commit_file): raise exceptions.GitHookAlreadyExists( 'File already exists', diff --git a/flake8/main/mercurial.py b/flake8/main/mercurial.py index 9767f34..54725c6 100644 --- a/flake8/main/mercurial.py +++ b/flake8/main/mercurial.py @@ -4,13 +4,106 @@ .. autofunction:: install """ +import configparser +import os +import subprocess + +from flake8 import exceptions as exc __all__ = ('hook', 'install') -def hook(lazy=False, strict=False): - pass +def hook(ui, repo, **kwargs): + """Execute Flake8 on the repository provided by Mercurial. + + To understand the parameters read more of the Mercurial documentation + around Hooks: https://www.mercurial-scm.org/wiki/Hook. + + We avoid using the ``ui`` attribute because it can cause issues with + the GPL license tha Mercurial is under. We don't import it, but we + avoid using it all the same. + """ + from flake8.main import application + hgrc = find_hgrc(create_if_missing=False) + if hgrc is None: + print('Cannot locate your root mercurial repository.') + raise SystemExit(True) + + hgconfig = configparser_for(hgrc) + strict = hgconfig.get('flake8', 'strict', fallback=True) + + app = application.Application() + app.run() + + if strict: + return app.result_count + return 0 def install(): - pass + """Ensure that the mercurial hooks are installed.""" + hgrc = find_hgrc(create_if_missing=True) + if hgrc is None: + print('Could not locate your root mercurial repository.') + raise SystemExit(True) + + hgconfig = configparser_for(hgrc) + + if not hgconfig.has_section('hooks'): + hgconfig.add_section('hooks') + + if hgconfig.has_option('hooks', 'commit'): + raise exc.MercurialCommitHookAlreadyExists( + path=hgrc, + value=hgconfig.get('hooks', 'commit'), + ) + + if hgconfig.has_option('hooks', 'qrefresh'): + raise exc.MercurialQRefreshHookAlreadyExists( + path=hgrc, + value=hgconfig.get('hooks', 'qrefresh'), + ) + + hgconfig.set('hooks', 'commit', 'python:flake8.main.mercurial.hook') + hgconfig.set('hooks', 'qrefresh', 'python:flake8.main.mercurial.hook') + + if not hgconfig.has_section('flake8'): + hgconfig.add_section('flake8') + + if not hgconfig.has_option('flake8', 'strict'): + hgconfig.set('flake8', 'strict', False) + + with open(hgrc, 'w') as fd: + hgconfig.write(fd) + + +def find_hgrc(create_if_missing=False): + root = subprocess.Popen( + ['hg', 'root'], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + (hg_directory, _) = root.communicate() + if callable(getattr(hg_directory, 'decode', None)): + hg_directory = hg_directory.decode('utf-8') + + if not os.path.isdir(hg_directory): + return None + + hgrc = os.path.abspath( + os.path.join(hg_directory, '.hg', 'hgrc') + ) + if not os.path.exists(hgrc): + if create_if_missing: + open(hgrc, 'w').close() + else: + return None + + return hgrc + + +def configparser_for(path): + parser = configparser.ConfigParser(interpolation=None) + parser.read(path) + return parser diff --git a/flake8/main/vcs.py b/flake8/main/vcs.py index 95cbfa1..10e6256 100644 --- a/flake8/main/vcs.py +++ b/flake8/main/vcs.py @@ -1,4 +1,5 @@ """Module containing some of the logic for our VCS installation logic.""" +from flake8 import exceptions as exc from flake8.main import git from flake8.main import mercurial @@ -20,8 +21,15 @@ def install(option, option_string, value, parser): https://docs.python.org/2/library/optparse.html#optparse-option-callbacks """ installer = _INSTALLERS.get(value) - installer() + errored = False + try: + installer() + except exc.HookInstallationError as hook_error: + print(str(hook_error)) + errored = True + raise SystemExit(errored) def choices(): - return _INSTALLERS.keys() + """Return the list of VCS choices.""" + return list(_INSTALLERS.keys()) From dc8fc0058d638479335b30e441973f7eab2ac2b0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 07:58:13 -0500 Subject: [PATCH 307/364] Configure bandit to ignore subprocess warnings --- .bandit.yml | 84 +++++++++++++++++++++++++++++++++++++++++++++++++++++ tox.ini | 2 +- 2 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 .bandit.yml diff --git a/.bandit.yml b/.bandit.yml new file mode 100644 index 0000000..ea868e2 --- /dev/null +++ b/.bandit.yml @@ -0,0 +1,84 @@ +tests: +skips: +- B404 # Ignore warnings about importing subprocess +- B603 # Ignore warnings about calling subprocess.Popen without shell=True +- B607 # Ignore warnings about calling subprocess.Popen without a full path to executable + +### (optional) plugin settings - some test plugins require configuration data +### that may be given here, per-plugin. All bandit test plugins have a built in +### set of sensible defaults and these will be used if no configuration is +### provided. It is not necessary to provide settings for every (or any) plugin +### if the defaults are acceptable. + +any_other_function_with_shell_equals_true: + no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv, os.execve, os.execvp, + os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, + os.spawnvp, os.spawnvpe, os.startfile] + shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, + popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] + subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, + utils.execute, utils.execute_with_timeout] +execute_with_run_as_root_equals_true: + function_names: [ceilometer.utils.execute, cinder.utils.execute, neutron.agent.linux.utils.execute, + nova.utils.execute, nova.utils.trycmd] +hardcoded_tmp_directory: + tmp_dirs: [/tmp, /var/tmp, /dev/shm] +linux_commands_wildcard_injection: + no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv, os.execve, os.execvp, + os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, + os.spawnvp, os.spawnvpe, os.startfile] + shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, + popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] + subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, + utils.execute, utils.execute_with_timeout] +password_config_option_not_marked_secret: + function_names: [oslo.config.cfg.StrOpt, oslo_config.cfg.StrOpt] +ssl_with_bad_defaults: + bad_protocol_versions: [PROTOCOL_SSLv2, SSLv2_METHOD, SSLv23_METHOD, PROTOCOL_SSLv3, + PROTOCOL_TLSv1, SSLv3_METHOD, TLSv1_METHOD] +ssl_with_bad_version: + bad_protocol_versions: [PROTOCOL_SSLv2, SSLv2_METHOD, SSLv23_METHOD, PROTOCOL_SSLv3, + PROTOCOL_TLSv1, SSLv3_METHOD, TLSv1_METHOD] +start_process_with_a_shell: + no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv, os.execve, os.execvp, + os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, + os.spawnvp, os.spawnvpe, os.startfile] + shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, + popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] + subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, + utils.execute, utils.execute_with_timeout] +start_process_with_no_shell: + no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv, os.execve, os.execvp, + os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, + os.spawnvp, os.spawnvpe, os.startfile] + shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, + popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] + subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, + utils.execute, utils.execute_with_timeout] +start_process_with_partial_path: + no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv, os.execve, os.execvp, + os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, + os.spawnvp, os.spawnvpe, os.startfile] + shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, + popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] + subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, + utils.execute, utils.execute_with_timeout] +subprocess_popen_with_shell_equals_true: + no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv, os.execve, os.execvp, + os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, + os.spawnvp, os.spawnvpe, os.startfile] + shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, + popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] + subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, + utils.execute, utils.execute_with_timeout] +subprocess_without_shell_equals_true: + no_shell: [os.execl, os.execle, os.execlp, os.execlpe, os.execv, os.execve, os.execvp, + os.execvpe, os.spawnl, os.spawnle, os.spawnlp, os.spawnlpe, os.spawnv, os.spawnve, + os.spawnvp, os.spawnvpe, os.startfile] + shell: [os.system, os.popen, os.popen2, os.popen3, os.popen4, popen2.popen2, popen2.popen3, + popen2.popen4, popen2.Popen3, popen2.Popen4, commands.getoutput, commands.getstatusoutput] + subprocess: [subprocess.Popen, subprocess.call, subprocess.check_call, subprocess.check_output, + utils.execute, utils.execute_with_timeout] +try_except_continue: {check_typed_exception: false} +try_except_pass: {check_typed_exception: false} + diff --git a/tox.ini b/tox.ini index 045abb6..0bc1795 100644 --- a/tox.ini +++ b/tox.ini @@ -66,7 +66,7 @@ use_develop = false deps = bandit commands = - bandit -r flake8/ + bandit -r flake8/ -c .bandit.yml [testenv:linters] basepython = python3 From 3cd8ef7d341fe91283420c2257ec2d43d1bd2aa4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 08:02:54 -0500 Subject: [PATCH 308/364] Add docstrings and a new-line for pydocstyle --- flake8/exceptions.py | 2 ++ flake8/main/options.py | 1 + 2 files changed, 3 insertions(+) diff --git a/flake8/exceptions.py b/flake8/exceptions.py index e709ff5..5ff55a2 100644 --- a/flake8/exceptions.py +++ b/flake8/exceptions.py @@ -45,6 +45,7 @@ class InvalidSyntax(Flake8Exception): class HookInstallationError(Flake8Exception): """Parent exception for all hooks errors.""" + pass @@ -76,6 +77,7 @@ class MercurialHookAlreadyExists(HookInstallationError): super(MercurialHookAlreadyExists, self).__init__(*args, **kwargs) def __str__(self): + """Return a nicely formatted string for these errors.""" msg = ('The Mercurial {0} hook already exists with "{1}" in {2}. ' 'To convince Flake8 to install the hook, please remove the ' '{0} configuration from the [hooks] section of your hgrc.') diff --git a/flake8/main/options.py b/flake8/main/options.py index a12276c..6ca7e77 100644 --- a/flake8/main/options.py +++ b/flake8/main/options.py @@ -1,3 +1,4 @@ +"""Contains the logic for all of the default options for Flake8.""" from flake8 import defaults from flake8.main import vcs From 84456866a5a8bbbe322ead206fe1e03309a0ad37 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 08:03:48 -0500 Subject: [PATCH 309/364] Fix integration tests --- tests/integration/test_aggregator.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/integration/test_aggregator.py b/tests/integration/test_aggregator.py index bb87be0..d0a7971 100644 --- a/tests/integration/test_aggregator.py +++ b/tests/integration/test_aggregator.py @@ -1,7 +1,7 @@ """Test aggregation of config files and command-line options.""" import os -from flake8.main import cli +from flake8.main import options from flake8.options import aggregator from flake8.options import manager @@ -17,7 +17,7 @@ def optmanager(): prog='flake8', version='3.0.0', ) - cli.register_default_options(option_manager) + options.register_default_options(option_manager) return option_manager From deb59817cf8958e7899142a3c492d1b8463cf30f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 08:04:13 -0500 Subject: [PATCH 310/364] Use the same interface for vcs installation flake8.main.git.install was already returning False if it couldn't find the directory to install into. This makes mercurial.install do the same thing and allows the vcs.install callback to understand that. --- flake8/main/mercurial.py | 5 +++-- flake8/main/vcs.py | 8 ++++++-- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/flake8/main/mercurial.py b/flake8/main/mercurial.py index 54725c6..10e5c06 100644 --- a/flake8/main/mercurial.py +++ b/flake8/main/mercurial.py @@ -44,8 +44,7 @@ def install(): """Ensure that the mercurial hooks are installed.""" hgrc = find_hgrc(create_if_missing=True) if hgrc is None: - print('Could not locate your root mercurial repository.') - raise SystemExit(True) + return False hgconfig = configparser_for(hgrc) @@ -76,6 +75,8 @@ def install(): with open(hgrc, 'w') as fd: hgconfig.write(fd) + return True + def find_hgrc(create_if_missing=False): root = subprocess.Popen( diff --git a/flake8/main/vcs.py b/flake8/main/vcs.py index 10e6256..6f7499e 100644 --- a/flake8/main/vcs.py +++ b/flake8/main/vcs.py @@ -22,12 +22,16 @@ def install(option, option_string, value, parser): """ installer = _INSTALLERS.get(value) errored = False + successful = False try: - installer() + successful = installer() except exc.HookInstallationError as hook_error: print(str(hook_error)) errored = True - raise SystemExit(errored) + + if not successful: + print('Could not find the {0} directory'.format(value)) + raise SystemExit(not successful and errored) def choices(): From 706520828d6639a7eb6aec8fd983fb0c47ce0d4e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 09:57:49 -0500 Subject: [PATCH 311/364] Fix documentation build --- docs/source/internal/cli.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/internal/cli.rst b/docs/source/internal/cli.rst index fc696bc..f3af8c1 100644 --- a/docs/source/internal/cli.rst +++ b/docs/source/internal/cli.rst @@ -20,7 +20,7 @@ API Documentation .. autofunction:: flake8.main.cli.main -.. autoclass:: flake8.main.cli.Application +.. autoclass:: flake8.main.application.Application :members: -.. autofunction:: flake8.main.cli.register_default_options +.. autofunction:: flake8.main.options.register_default_options From 472e7c95895cb00c5de6dea8cd3a1a96dd30a7f8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 09:58:24 -0500 Subject: [PATCH 312/364] Add release notes portion of documentation --- docs/source/index.rst | 8 ++++++ docs/source/release-notes/0.6.0.rst | 4 +++ docs/source/release-notes/0.7.0.rst | 6 +++++ docs/source/release-notes/0.8.0.rst | 5 ++++ docs/source/release-notes/0.9.0.rst | 5 ++++ docs/source/release-notes/1.0.0.rst | 5 ++++ docs/source/release-notes/1.1.0.rst | 8 ++++++ docs/source/release-notes/1.2.0.rst | 6 +++++ docs/source/release-notes/1.3.0.rst | 4 +++ docs/source/release-notes/1.3.1.rst | 4 +++ docs/source/release-notes/1.4.0.rst | 5 ++++ docs/source/release-notes/1.5.0.rst | 9 +++++++ docs/source/release-notes/1.6.0.rst | 14 ++++++++++ docs/source/release-notes/1.6.1.rst | 7 +++++ docs/source/release-notes/1.6.2.rst | 4 +++ docs/source/release-notes/1.7.0.rst | 9 +++++++ docs/source/release-notes/2.0.0.rst | 13 ++++++++++ docs/source/release-notes/2.1.0.rst | 12 +++++++++ docs/source/release-notes/2.2.0.rst | 12 +++++++++ docs/source/release-notes/2.2.1.rst | 5 ++++ docs/source/release-notes/2.2.2.rst | 5 ++++ docs/source/release-notes/2.2.3.rst | 4 +++ docs/source/release-notes/2.2.4.rst | 20 +++++++++++++++ docs/source/release-notes/2.2.5.rst | 6 +++++ docs/source/release-notes/2.3.0.rst | 10 ++++++++ docs/source/release-notes/2.4.0.rst | 33 ++++++++++++++++++++++++ docs/source/release-notes/2.4.1.rst | 12 +++++++++ docs/source/release-notes/2.5.0.rst | 25 ++++++++++++++++++ docs/source/release-notes/2.5.1.rst | 13 ++++++++++ docs/source/release-notes/2.5.2.rst | 7 +++++ docs/source/release-notes/2.5.3.rst | 5 ++++ docs/source/release-notes/2.5.4.rst | 4 +++ docs/source/release-notes/2.5.5.rst | 7 +++++ docs/source/release-notes/index.rst | 40 +++++++++++++++++++++++++++++ 34 files changed, 336 insertions(+) create mode 100644 docs/source/release-notes/0.6.0.rst create mode 100644 docs/source/release-notes/0.7.0.rst create mode 100644 docs/source/release-notes/0.8.0.rst create mode 100644 docs/source/release-notes/0.9.0.rst create mode 100644 docs/source/release-notes/1.0.0.rst create mode 100644 docs/source/release-notes/1.1.0.rst create mode 100644 docs/source/release-notes/1.2.0.rst create mode 100644 docs/source/release-notes/1.3.0.rst create mode 100644 docs/source/release-notes/1.3.1.rst create mode 100644 docs/source/release-notes/1.4.0.rst create mode 100644 docs/source/release-notes/1.5.0.rst create mode 100644 docs/source/release-notes/1.6.0.rst create mode 100644 docs/source/release-notes/1.6.1.rst create mode 100644 docs/source/release-notes/1.6.2.rst create mode 100644 docs/source/release-notes/1.7.0.rst create mode 100644 docs/source/release-notes/2.0.0.rst create mode 100644 docs/source/release-notes/2.1.0.rst create mode 100644 docs/source/release-notes/2.2.0.rst create mode 100644 docs/source/release-notes/2.2.1.rst create mode 100644 docs/source/release-notes/2.2.2.rst create mode 100644 docs/source/release-notes/2.2.3.rst create mode 100644 docs/source/release-notes/2.2.4.rst create mode 100644 docs/source/release-notes/2.2.5.rst create mode 100644 docs/source/release-notes/2.3.0.rst create mode 100644 docs/source/release-notes/2.4.0.rst create mode 100644 docs/source/release-notes/2.4.1.rst create mode 100644 docs/source/release-notes/2.5.0.rst create mode 100644 docs/source/release-notes/2.5.1.rst create mode 100644 docs/source/release-notes/2.5.2.rst create mode 100644 docs/source/release-notes/2.5.3.rst create mode 100644 docs/source/release-notes/2.5.4.rst create mode 100644 docs/source/release-notes/2.5.5.rst create mode 100644 docs/source/release-notes/index.rst diff --git a/docs/source/index.rst b/docs/source/index.rst index 71b3c94..3343801 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -90,6 +90,14 @@ Developer Guide internal/index +Release Notes and History +========================= + +.. toctree:: + :maxdepth: 2 + + release-notes/index + Indices and tables ================== diff --git a/docs/source/release-notes/0.6.0.rst b/docs/source/release-notes/0.6.0.rst new file mode 100644 index 0000000..16590fa --- /dev/null +++ b/docs/source/release-notes/0.6.0.rst @@ -0,0 +1,4 @@ +0.6 - 2010-02-15 +---------------- + +- Fix the McCabe metric on some loops diff --git a/docs/source/release-notes/0.7.0.rst b/docs/source/release-notes/0.7.0.rst new file mode 100644 index 0000000..f842060 --- /dev/null +++ b/docs/source/release-notes/0.7.0.rst @@ -0,0 +1,6 @@ +0.7 - 2010-02-18 +---------------- + +- Fix pep8 initialization when run through Hg +- Make pep8 short options work when run through the command line +- Skip duplicates when controlling files via Hg diff --git a/docs/source/release-notes/0.8.0.rst b/docs/source/release-notes/0.8.0.rst new file mode 100644 index 0000000..66b90b5 --- /dev/null +++ b/docs/source/release-notes/0.8.0.rst @@ -0,0 +1,5 @@ +0.8 - 2011-02-27 +---------------- + +- fixed hg hook +- discard unexisting files on hook check diff --git a/docs/source/release-notes/0.9.0.rst b/docs/source/release-notes/0.9.0.rst new file mode 100644 index 0000000..be6c41b --- /dev/null +++ b/docs/source/release-notes/0.9.0.rst @@ -0,0 +1,5 @@ +0.9 - 2011-11-09 +---------------- + +- update pep8 version to 0.6.1 +- mccabe check: gracefully handle compile failure diff --git a/docs/source/release-notes/1.0.0.rst b/docs/source/release-notes/1.0.0.rst new file mode 100644 index 0000000..6882a43 --- /dev/null +++ b/docs/source/release-notes/1.0.0.rst @@ -0,0 +1,5 @@ +1.0 - 2011-11-29 +---------------- + +- Deactivates by default the complexity checker +- Introduces the complexity option in the HG hook and the command line. diff --git a/docs/source/release-notes/1.1.0.rst b/docs/source/release-notes/1.1.0.rst new file mode 100644 index 0000000..dadbe2e --- /dev/null +++ b/docs/source/release-notes/1.1.0.rst @@ -0,0 +1,8 @@ +1.1 - 2012-02-14 +---------------- + +- fixed the value returned by --version +- allow the flake8: header to be more generic +- fixed the "hg hook raises 'physical lines'" bug +- allow three argument form of raise +- now uses setuptools if available, for 'develop' command diff --git a/docs/source/release-notes/1.2.0.rst b/docs/source/release-notes/1.2.0.rst new file mode 100644 index 0000000..de86237 --- /dev/null +++ b/docs/source/release-notes/1.2.0.rst @@ -0,0 +1,6 @@ +1.2 - 2012-02-12 +---------------- + +- added a git hook +- now Python 3 compatible +- mccabe and pyflakes have warning codes like pep8 now diff --git a/docs/source/release-notes/1.3.0.rst b/docs/source/release-notes/1.3.0.rst new file mode 100644 index 0000000..0ddfe78 --- /dev/null +++ b/docs/source/release-notes/1.3.0.rst @@ -0,0 +1,4 @@ +1.3 - 2012-03-12 +---------------- + +- fixed false W402 warning on exception blocks. diff --git a/docs/source/release-notes/1.3.1.rst b/docs/source/release-notes/1.3.1.rst new file mode 100644 index 0000000..b2e34ee --- /dev/null +++ b/docs/source/release-notes/1.3.1.rst @@ -0,0 +1,4 @@ +1.3.1 - 2012-05-19 +------------------ + +- fixed support for Python 2.5 diff --git a/docs/source/release-notes/1.4.0.rst b/docs/source/release-notes/1.4.0.rst new file mode 100644 index 0000000..51f34cf --- /dev/null +++ b/docs/source/release-notes/1.4.0.rst @@ -0,0 +1,5 @@ +1.4 - 2012-07-12 +---------------- + +- git_hook: Only check staged changes for compliance +- use pep8 1.2 diff --git a/docs/source/release-notes/1.5.0.rst b/docs/source/release-notes/1.5.0.rst new file mode 100644 index 0000000..cd0a88d --- /dev/null +++ b/docs/source/release-notes/1.5.0.rst @@ -0,0 +1,9 @@ +1.5 - 2012-10-13 +---------------- + +- fixed the stdin +- make sure mccabe catches the syntax errors as warnings +- pep8 upgrade +- added max_line_length default value +- added Flake8Command and entry points if setuptools is around +- using the setuptools console wrapper when available diff --git a/docs/source/release-notes/1.6.0.rst b/docs/source/release-notes/1.6.0.rst new file mode 100644 index 0000000..658be2f --- /dev/null +++ b/docs/source/release-notes/1.6.0.rst @@ -0,0 +1,14 @@ +1.6 - 2012-11-16 +---------------- + +- changed the signatures of the ``check_file`` function in flake8/run.py, + ``skip_warning`` in flake8/util.py and the ``check``, ``checkPath`` + functions in flake8/pyflakes.py. +- fix ``--exclude`` and ``--ignore`` command flags (#14, #19) +- fix the git hook that wasn't catching files not already added to the index + (#29) +- pre-emptively includes the addition to pep8 to ignore certain lines. + Add ``# nopep8`` to the end of a line to ignore it. (#37) +- ``check_file`` can now be used without any special prior setup (#21) +- unpacking exceptions will no longer cause an exception (#20) +- fixed crash on non-existent file (#38) diff --git a/docs/source/release-notes/1.6.1.rst b/docs/source/release-notes/1.6.1.rst new file mode 100644 index 0000000..194dfa5 --- /dev/null +++ b/docs/source/release-notes/1.6.1.rst @@ -0,0 +1,7 @@ +1.6.1 - 2012-11-24 +------------------ + +- fixed the mercurial hook, a change from a previous patch was not properly + applied +- fixed an assumption about warnings/error messages that caused an exception + to be thrown when McCabe is used diff --git a/docs/source/release-notes/1.6.2.rst b/docs/source/release-notes/1.6.2.rst new file mode 100644 index 0000000..64633bd --- /dev/null +++ b/docs/source/release-notes/1.6.2.rst @@ -0,0 +1,4 @@ +1.6.2 - 2012-11-25 +------------------ + +- fixed the NameError: global name 'message' is not defined (#46) diff --git a/docs/source/release-notes/1.7.0.rst b/docs/source/release-notes/1.7.0.rst new file mode 100644 index 0000000..a3a4725 --- /dev/null +++ b/docs/source/release-notes/1.7.0.rst @@ -0,0 +1,9 @@ +1.7.0 - 2012-12-21 +------------------ + +- Fixes part of #35: Exception for no WITHITEM being an attribute of Checker + for Python 3.3 +- Support stdin +- Incorporate @phd's builtins pull request +- Fix the git hook +- Update pep8.py to the latest version diff --git a/docs/source/release-notes/2.0.0.rst b/docs/source/release-notes/2.0.0.rst new file mode 100644 index 0000000..4c1ff04 --- /dev/null +++ b/docs/source/release-notes/2.0.0.rst @@ -0,0 +1,13 @@ +2.0.0 - 2013-02-23 +------------------ + +- Pyflakes errors are prefixed by an ``F`` instead of an ``E`` +- McCabe complexity warnings are prefixed by a ``C`` instead of a ``W`` +- Flake8 supports extensions through entry points +- Due to the above support, we **require** setuptools +- We publish the `documentation `_ +- Fixes #13: pep8, pyflakes and mccabe become external dependencies +- Split run.py into main.py, engine.py and hooks.py for better logic +- Expose our parser for our users +- New feature: Install git and hg hooks automagically +- By relying on pyflakes (0.6.1), we also fixed #45 and #35 diff --git a/docs/source/release-notes/2.1.0.rst b/docs/source/release-notes/2.1.0.rst new file mode 100644 index 0000000..c9e3c60 --- /dev/null +++ b/docs/source/release-notes/2.1.0.rst @@ -0,0 +1,12 @@ +2.1.0 - 2013-10-26 +------------------ + +- Add FLAKE8_LAZY and FLAKE8_IGNORE environment variable support to git and + mercurial hooks +- Force git and mercurial hooks to repsect configuration in setup.cfg +- Only check staged files if that is specified +- Fix hook file permissions +- Fix the git hook on python 3 +- Ignore non-python files when running the git hook +- Ignore .tox directories by default +- Flake8 now reports the column number for PyFlakes messages diff --git a/docs/source/release-notes/2.2.0.rst b/docs/source/release-notes/2.2.0.rst new file mode 100644 index 0000000..357b6d2 --- /dev/null +++ b/docs/source/release-notes/2.2.0.rst @@ -0,0 +1,12 @@ +2.2.0 - 2014-06-22 +------------------ + +- New option ``doctests`` to run Pyflakes checks on doctests too +- New option ``jobs`` to launch multiple jobs in parallel +- Turn on using multiple jobs by default using the CPU count +- Add support for ``python -m flake8`` on Python 2.7 and Python 3 +- Fix Git and Mercurial hooks: issues #88, #133, #148 and #149 +- Fix crashes with Python 3.4 by upgrading dependencies +- Fix traceback when running tests with Python 2.6 +- Fix the setuptools command ``python setup.py flake8`` to read + the project configuration diff --git a/docs/source/release-notes/2.2.1.rst b/docs/source/release-notes/2.2.1.rst new file mode 100644 index 0000000..5575f8f --- /dev/null +++ b/docs/source/release-notes/2.2.1.rst @@ -0,0 +1,5 @@ +2.2.1 - 2014-06-30 +------------------ + +- Turn off multiple jobs by default. To enable automatic use of all CPUs, use + ``--jobs=auto``. Fixes #155 and #154. diff --git a/docs/source/release-notes/2.2.2.rst b/docs/source/release-notes/2.2.2.rst new file mode 100644 index 0000000..8fcff88 --- /dev/null +++ b/docs/source/release-notes/2.2.2.rst @@ -0,0 +1,5 @@ +2.2.2 - 2014-07-04 +------------------ + +- Re-enable multiprocessing by default while fixing the issue Windows users + were seeing. diff --git a/docs/source/release-notes/2.2.3.rst b/docs/source/release-notes/2.2.3.rst new file mode 100644 index 0000000..e7430f0 --- /dev/null +++ b/docs/source/release-notes/2.2.3.rst @@ -0,0 +1,4 @@ +2.2.3 - 2014-08-25 +------------------ + +- Actually turn multiprocessing on by default diff --git a/docs/source/release-notes/2.2.4.rst b/docs/source/release-notes/2.2.4.rst new file mode 100644 index 0000000..2564948 --- /dev/null +++ b/docs/source/release-notes/2.2.4.rst @@ -0,0 +1,20 @@ +2.2.4 - 2014-10-09 +------------------ + +- Fix bugs triggered by turning multiprocessing on by default (again) + + Multiprocessing is forcibly disabled in the following cases: + + - Passing something in via stdin + + - Analyzing a diff + + - Using windows + +- Fix --install-hook when there are no config files present for pep8 or + flake8. + +- Fix how the setuptools command parses excludes in config files + +- Fix how the git hook determines which files to analyze (Thanks Chris + Buccella!) diff --git a/docs/source/release-notes/2.2.5.rst b/docs/source/release-notes/2.2.5.rst new file mode 100644 index 0000000..540278f --- /dev/null +++ b/docs/source/release-notes/2.2.5.rst @@ -0,0 +1,6 @@ +2.2.5 - 2014-10-19 +------------------ + +- Flush standard out when using multiprocessing + +- Make the check for "# flake8: noqa" more strict diff --git a/docs/source/release-notes/2.3.0.rst b/docs/source/release-notes/2.3.0.rst new file mode 100644 index 0000000..120efa9 --- /dev/null +++ b/docs/source/release-notes/2.3.0.rst @@ -0,0 +1,10 @@ +2.3.0 - 2015-01-04 +------------------ + +- **Feature**: Add ``--output-file`` option to specify a file to write to + instead of ``stdout``. + +- **Bug** Fix interleaving of output while using multiprocessing + (`GitLab#17`_) + +.. _GitLab#17: https://gitlab.com/pycqa/flake8/issues/17 diff --git a/docs/source/release-notes/2.4.0.rst b/docs/source/release-notes/2.4.0.rst new file mode 100644 index 0000000..6c470cd --- /dev/null +++ b/docs/source/release-notes/2.4.0.rst @@ -0,0 +1,33 @@ +2.4.0 - 2015-03-07 +------------------ + +- **Bug** Print filenames when using multiprocessing and ``-q`` option. + (`GitLab#31`_) + +- **Bug** Put upper cap on dependencies. The caps for 2.4.0 are: + + - ``pep8 < 1.6`` (Related to `GitLab#35`_) + + - ``mccabe < 0.4`` + + - ``pyflakes < 0.9`` + + See also `GitLab#32`_ + +- **Bug** Files excluded in a config file were not being excluded when flake8 + was run from a git hook. (`GitHub#2`_) + +- **Improvement** Print warnings for users who are providing mutually + exclusive options to flake8. (`GitLab#8`_, `GitLab!18`_) + +- **Feature** Allow git hook configuration to live in ``.git/config``. + See the updated `VCS hooks docs`_ for more details. (`GitLab!20`_) + +.. _GitHub#2: https://github.com/pycqa/flake8/pull/2 +.. _GitLab#8: https://gitlab.com/pycqa/flake8/issues/8 +.. _GitLab#31: https://gitlab.com/pycqa/flake8/issues/31 +.. _GitLab#32: https://gitlab.com/pycqa/flake8/issues/32 +.. _GitLab#35: https://gitlab.com/pycqa/flake8/issues/35 +.. _GitLab!18: https://gitlab.com/pycqa/flake8/merge_requests/18 +.. _GitLab!20: https://gitlab.com/pycqa/flake8/merge_requests/20 +.. _VCS hooks docs: https://flake8.readthedocs.org/en/latest/vcs.html diff --git a/docs/source/release-notes/2.4.1.rst b/docs/source/release-notes/2.4.1.rst new file mode 100644 index 0000000..3448bc4 --- /dev/null +++ b/docs/source/release-notes/2.4.1.rst @@ -0,0 +1,12 @@ +2.4.1 - 2015-05-18 +------------------ + +- **Bug** Do not raise a ``SystemError`` unless there were errors in the + setuptools command. (`GitLab#39`_, `GitLab!23`_) + +- **Bug** Do not verify dependencies of extensions loaded via entry-points. + +- **Improvement** Blacklist versions of pep8 we know are broken + +.. _GitLab#39: https://gitlab.com/pycqa/flake8/issues/39 +.. _GitLab!23: https://gitlab.com/pycqa/flake8/merge_requests/23 diff --git a/docs/source/release-notes/2.5.0.rst b/docs/source/release-notes/2.5.0.rst new file mode 100644 index 0000000..1558fcf --- /dev/null +++ b/docs/source/release-notes/2.5.0.rst @@ -0,0 +1,25 @@ +2.5.0 - 2015-10-26 +------------------ + +- **Improvement** Raise cap on PyFlakes for Python 3.5 support + +- **Improvement** Avoid deprecation warnings when loading extensions + (`GitLab#59`_, `GitLab#90`_) + +- **Improvement** Separate logic to enable "off-by-default" extensions + (`GitLab#67`_) + +- **Bug** Properly parse options to setuptools Flake8 command (`GitLab!41`_) + +- **Bug** Fix exceptions when output on stdout is truncated before Flake8 + finishes writing the output (`GitLab#69`_) + +- **Bug** Fix error on OS X where Flake8 can no longer acquire or create new + semaphores (`GitLab#74`_) + +.. _GitLab!41: https://gitlab.com/pycqa/flake8/merge_requests/41 +.. _GitLab#59: https://gitlab.com/pycqa/flake8/issues/59 +.. _GitLab#67: https://gitlab.com/pycqa/flake8/issues/67 +.. _GitLab#69: https://gitlab.com/pycqa/flake8/issues/69 +.. _GitLab#74: https://gitlab.com/pycqa/flake8/issues/74 +.. _GitLab#90: https://gitlab.com/pycqa/flake8/issues/90 diff --git a/docs/source/release-notes/2.5.1.rst b/docs/source/release-notes/2.5.1.rst new file mode 100644 index 0000000..8a114c8 --- /dev/null +++ b/docs/source/release-notes/2.5.1.rst @@ -0,0 +1,13 @@ +2.5.1 - 2015-12-08 +------------------ + +- **Bug** Properly look for ``.flake8`` in current working directory + (`GitLab#103`_) + +- **Bug** Monkey-patch ``pep8.stdin_get_value`` to cache the actual value in + stdin. This helps plugins relying on the function when run with + multiprocessing. (`GitLab#105`_, `GitLab#107`_) + +.. _GitLab#103: https://gitlab.com/pycqa/flake8/issues/103 +.. _GitLab#105: https://gitlab.com/pycqa/flake8/issues/105 +.. _GitLab#107: https://gitlab.com/pycqa/flake8/issues/107 diff --git a/docs/source/release-notes/2.5.2.rst b/docs/source/release-notes/2.5.2.rst new file mode 100644 index 0000000..a093c9f --- /dev/null +++ b/docs/source/release-notes/2.5.2.rst @@ -0,0 +1,7 @@ +2.5.2 - 2016-01-30 +------------------ + +- **Bug** Parse ``output_file`` and ``enable_extensions`` from config files + +- **Improvement** Raise upper bound on mccabe plugin to allow for version + 0.4.0 diff --git a/docs/source/release-notes/2.5.3.rst b/docs/source/release-notes/2.5.3.rst new file mode 100644 index 0000000..85dbf33 --- /dev/null +++ b/docs/source/release-notes/2.5.3.rst @@ -0,0 +1,5 @@ +2.5.3 - 2016-02-11 +------------------ + +- **Bug** Actually parse ``output_file`` and ``enable_extensions`` from config + files diff --git a/docs/source/release-notes/2.5.4.rst b/docs/source/release-notes/2.5.4.rst new file mode 100644 index 0000000..5ba03ba --- /dev/null +++ b/docs/source/release-notes/2.5.4.rst @@ -0,0 +1,4 @@ +2.5.4 - 2016-02-11 +------------------ + +- **Bug** Missed an attribute rename during the v2.5.3 release. diff --git a/docs/source/release-notes/2.5.5.rst b/docs/source/release-notes/2.5.5.rst new file mode 100644 index 0000000..683cfb6 --- /dev/null +++ b/docs/source/release-notes/2.5.5.rst @@ -0,0 +1,7 @@ +2.5.5 - 2016-06-14 +------------------ + +- **Bug** Fix setuptools integration when parsing config files + +- **Bug** Don't pass the user's config path as the config_file when creating a + StyleGuide diff --git a/docs/source/release-notes/index.rst b/docs/source/release-notes/index.rst new file mode 100644 index 0000000..aef67e9 --- /dev/null +++ b/docs/source/release-notes/index.rst @@ -0,0 +1,40 @@ +=========================== + Release Notes and History +=========================== + +All of the release notes that have been recorded for Flake8 are organized here +with the newest releases first. + +.. toctree:: + 2.5.5 + 2.5.4 + 2.5.3 + 2.5.2 + 2.5.1 + 2.5.0 + 2.4.1 + 2.4.0 + 2.3.0 + 2.2.5 + 2.2.4 + 2.2.3 + 2.2.2 + 2.2.1 + 2.2.0 + 2.1.0 + 2.0.0 + 1.7.0 + 1.6.2 + 1.6.1 + 1.6.0 + 1.5.0 + 1.4.0 + 1.3.1 + 1.3.0 + 1.2.0 + 1.1.0 + 1.0.0 + 0.9.0 + 0.8.0 + 0.7.0 + 0.6.0 From 18d6e6264cca194567e3e75ae0efe1df05ed57ea Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 20:04:52 -0500 Subject: [PATCH 313/364] Revert silliness around setting choices for --format If we restrict the user's ability to specify a format string, we break existing workflows. --- docs/source/internal/cli.rst | 4 ++-- flake8/main/application.py | 12 +++--------- flake8/main/options.py | 7 +++++-- 3 files changed, 10 insertions(+), 13 deletions(-) diff --git a/docs/source/internal/cli.rst b/docs/source/internal/cli.rst index f3af8c1..a1e9612 100644 --- a/docs/source/internal/cli.rst +++ b/docs/source/internal/cli.rst @@ -11,8 +11,8 @@ passed by the user as early as possible so as much logging can be produced as possible. The default flake8 options are registered by -:func:`~flake8.main.cli.register_default_options`. Trying to register these -options in plugins will result in errors. +:func:`~flake8.main.options.register_default_options`. Trying to register +these options in plugins will result in errors. API Documentation diff --git a/flake8/main/application.py b/flake8/main/application.py index d776bbf..e41a6c2 100644 --- a/flake8/main/application.py +++ b/flake8/main/application.py @@ -34,11 +34,10 @@ class Application(object): self.version = version #: The instance of :class:`flake8.options.manager.OptionManager` used #: to parse and handle the options and arguments passed by the user - self.option_manager = None - temp_option_manager = manager.OptionManager( + self.option_manager = manager.OptionManager( prog='flake8', version=flake8.__version__ ) - options.register_default_options(temp_option_manager) + options.register_default_options(self.option_manager) # We haven't found or registered our plugins yet, so let's defer # printing the version until we aggregate options from config files @@ -62,7 +61,7 @@ class Application(object): except ValueError: pass - preliminary_opts, _ = temp_option_manager.parse_args(args) + preliminary_opts, _ = self.option_manager.parse_args(args) # Set the verbosity of the program flake8.configure_logging(preliminary_opts.verbose, preliminary_opts.output_file) @@ -242,12 +241,7 @@ class Application(object): This finds the plugins, registers their options, and parses the command-line arguments. """ - self.option_manager = manager.OptionManager( - prog='flake8', version=flake8.__version__ - ) self.find_plugins() - options.register_default_options(self.option_manager, - self.formatting_plugins.names) self.register_plugin_options() self.parse_configuration_and_cli(argv) self.make_formatter() diff --git a/flake8/main/options.py b/flake8/main/options.py index 6ca7e77..9be4386 100644 --- a/flake8/main/options.py +++ b/flake8/main/options.py @@ -3,7 +3,7 @@ from flake8 import defaults from flake8.main import vcs -def register_default_options(option_manager, formatters=None): +def register_default_options(option_manager): """Register the default options on our OptionManager. The default options include: @@ -82,9 +82,12 @@ def register_default_options(option_manager, formatters=None): # TODO(sigmavirus24): Figure out --first/--repeat + # NOTE(sigmavirus24): We can't use choices for this option since users can + # freely provide a format string and that will break if we restrict their + # choices. add_option( '--format', metavar='format', default='default', - parse_from_config=True, choices=(formatters or ['default']), + parse_from_config=True, help='Format errors according to the chosen formatter.', ) From 8a18cfa4a321218a67c322dcf70fe4d11efb92ca Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 20:10:09 -0500 Subject: [PATCH 314/364] Add documentation around specifying a format string --- docs/source/user/options.rst | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index e336daa..9e998ee 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -225,17 +225,30 @@ By default, there are two formatters available: - default - - pylint Other formatters can be installed. Refer to their documentation for the - name to use to select them. + name to use to select them. Further, users can specify they're own format + string. The variables available are: + + - code + - col + - path + - row + - text + + The default formatter has a format string of: + + .. code-block:: python + + '%(path)s:%(row)d:%(col)d: %(code)s %(text)s' Command-line example: .. prompt:: bash flake8 --format=pylint dir/ + flake8 --format='%(path)s::%(row)d,%(col)d::%(code)s::%(text)s' dir/ This **can** be specified in config files. @@ -244,6 +257,7 @@ .. code-block:: ini format=pylint + format=%(path)s::%(row)d,%(col)d::%(code)s::%(text)s .. option:: --hang-closing From 8bb304877547710e8deef328beddb23a1f6b9975 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 20:43:00 -0500 Subject: [PATCH 315/364] Add documentation for --install-hook --- docs/source/user/options.rst | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index 9e998ee..69aec6e 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -492,6 +492,28 @@ This **can not** be specified in config files. +.. option:: --install-hook=VERSION_CONTROL_SYSTEM + + Install a hook for your version control system that is executed before + or during commit. + + The available options are: + + - git + - mercurial + + Command-line usage: + + .. prompt:: bash + + flake8 --install-hook=git + flake8 --install-hook=mercurial + + This **can not** be specified in config files. + +Thank you for your contribution to Flake8's documentation. + + .. option:: --jobs= Specify the number of subprocesses that Flake8 will use to run checks in From e2ef07fe80225f6b87acb6623f120a61b6cb46fb Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 20:47:35 -0500 Subject: [PATCH 316/364] Copied one too many lines from template --- docs/source/user/options.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index 69aec6e..64ecaeb 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -511,8 +511,6 @@ This **can not** be specified in config files. -Thank you for your contribution to Flake8's documentation. - .. option:: --jobs= From 29419b0ac3c5fe69439d93a1d53bc42e9d8e7166 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 20:48:04 -0500 Subject: [PATCH 317/364] Start a changelog entry for 3.0.0 --- docs/source/release-notes/3.0.0.rst | 23 +++++++++++++++++++++++ docs/source/release-notes/index.rst | 1 + 2 files changed, 24 insertions(+) create mode 100644 docs/source/release-notes/3.0.0.rst diff --git a/docs/source/release-notes/3.0.0.rst b/docs/source/release-notes/3.0.0.rst new file mode 100644 index 0000000..e729723 --- /dev/null +++ b/docs/source/release-notes/3.0.0.rst @@ -0,0 +1,23 @@ +3.0.0 -- Unreleased +------------------- + +- Rewrite our documentation from scratch! (http://flake8.pycqa.org) + +- ``--select`` and ``--ignore`` can now both be specified and try to find the + most specific rule from each. For example, if you do ``--select E --ignore + E123`` then we will report everything that starts with ``E`` except for + ``E123``. Previously, you would have had to do ``--ignore E123,F,W`` which + will also still work, but the former should be far more intuitive. + +- Remove dependence on pep8/pycodestyle for file processing, plugin + dispatching, and more. We now control all of this while keeping backwards + compatibility. + +- Add entry-point for formatters as well as a base class that new formatters + can inherit from. See the documentation for more details. + +- Enhance our usage of optparse for plugin developers by adding new parameters + to the ``add_option`` that plugins use to register new options. + +- Update ``--install-hook`` to require the name of version control system hook + you wish to install a Flake8. diff --git a/docs/source/release-notes/index.rst b/docs/source/release-notes/index.rst index aef67e9..d91324f 100644 --- a/docs/source/release-notes/index.rst +++ b/docs/source/release-notes/index.rst @@ -6,6 +6,7 @@ All of the release notes that have been recorded for Flake8 are organized here with the newest releases first. .. toctree:: + 3.0.0 2.5.5 2.5.4 2.5.3 From 3af48abac6c7221f736bbed5a3c726aca260ad97 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 14 Jun 2016 20:48:44 -0500 Subject: [PATCH 318/364] Remove CHANGES.rst --- CHANGES.rst | 331 ---------------------------------------------------- setup.py | 2 +- 2 files changed, 1 insertion(+), 332 deletions(-) delete mode 100644 CHANGES.rst diff --git a/CHANGES.rst b/CHANGES.rst deleted file mode 100644 index 47ef27f..0000000 --- a/CHANGES.rst +++ /dev/null @@ -1,331 +0,0 @@ -CHANGES -======= - -2.5.4 - 2016-02-11 ------------------- - -- **Bug** Missed an attribute rename during the v2.5.3 release. - -2.5.3 - 2016-02-11 ------------------- - -- **Bug** Actually parse ``output_file`` and ``enable_extensions`` from config - files - -2.5.2 - 2016-01-30 ------------------- - -- **Bug** Parse ``output_file`` and ``enable_extensions`` from config files - -- **Improvement** Raise upper bound on mccabe plugin to allow for version - 0.4.0 - -2.5.1 - 2015-12-08 ------------------- - -- **Bug** Properly look for ``.flake8`` in current working directory - (`GitLab#103`_) - -- **Bug** Monkey-patch ``pep8.stdin_get_value`` to cache the actual value in - stdin. This helps plugins relying on the function when run with - multiprocessing. (`GitLab#105`_, `GitLab#107`_) - -.. _GitLab#103: https://gitlab.com/pycqa/flake8/issues/103 -.. _GitLab#105: https://gitlab.com/pycqa/flake8/issues/105 -.. _GitLab#107: https://gitlab.com/pycqa/flake8/issues/107 - -2.5.0 - 2015-10-26 ------------------- - -- **Improvement** Raise cap on PyFlakes for Python 3.5 support - -- **Improvement** Avoid deprecation warnings when loading extensions - (`GitLab#59`_, `GitLab#90`_) - -- **Improvement** Separate logic to enable "off-by-default" extensions - (`GitLab#67`_) - -- **Bug** Properly parse options to setuptools Flake8 command (`GitLab!41`_) - -- **Bug** Fix exceptions when output on stdout is truncated before Flake8 - finishes writing the output (`GitLab#69`_) - -- **Bug** Fix error on OS X where Flake8 can no longer acquire or create new - semaphores (`GitLab#74`_) - -.. _GitLab!41: https://gitlab.com/pycqa/flake8/merge_requests/41 -.. _GitLab#59: https://gitlab.com/pycqa/flake8/issues/59 -.. _GitLab#67: https://gitlab.com/pycqa/flake8/issues/67 -.. _GitLab#69: https://gitlab.com/pycqa/flake8/issues/69 -.. _GitLab#74: https://gitlab.com/pycqa/flake8/issues/74 -.. _GitLab#90: https://gitlab.com/pycqa/flake8/issues/90 - -2.4.1 - 2015-05-18 ------------------- - -- **Bug** Do not raise a ``SystemError`` unless there were errors in the - setuptools command. (`GitLab#39`_, `GitLab!23`_) - -- **Bug** Do not verify dependencies of extensions loaded via entry-points. - -- **Improvement** Blacklist versions of pep8 we know are broken - -.. _GitLab#39: https://gitlab.com/pycqa/flake8/issues/39 -.. _GitLab!23: https://gitlab.com/pycqa/flake8/merge_requests/23 - -2.4.0 - 2015-03-07 ------------------- - -- **Bug** Print filenames when using multiprocessing and ``-q`` option. - (`GitLab#31`_) - -- **Bug** Put upper cap on dependencies. The caps for 2.4.0 are: - - - ``pep8 < 1.6`` (Related to `GitLab#35`_) - - - ``mccabe < 0.4`` - - - ``pyflakes < 0.9`` - - See also `GitLab#32`_ - -- **Bug** Files excluded in a config file were not being excluded when flake8 - was run from a git hook. (`GitHub#2`_) - -- **Improvement** Print warnings for users who are providing mutually - exclusive options to flake8. (`GitLab#8`_, `GitLab!18`_) - -- **Feature** Allow git hook configuration to live in ``.git/config``. - See the updated `VCS hooks docs`_ for more details. (`GitLab!20`_) - -.. _GitHub#2: https://github.com/pycqa/flake8/pull/2 -.. _GitLab#8: https://gitlab.com/pycqa/flake8/issues/8 -.. _GitLab#31: https://gitlab.com/pycqa/flake8/issues/31 -.. _GitLab#32: https://gitlab.com/pycqa/flake8/issues/32 -.. _GitLab#35: https://gitlab.com/pycqa/flake8/issues/35 -.. _GitLab!18: https://gitlab.com/pycqa/flake8/merge_requests/18 -.. _GitLab!20: https://gitlab.com/pycqa/flake8/merge_requests/20 -.. _VCS hooks docs: https://flake8.readthedocs.org/en/latest/vcs.html - -2.3.0 - 2015-01-04 ------------------- - -- **Feature**: Add ``--output-file`` option to specify a file to write to - instead of ``stdout``. - -- **Bug** Fix interleaving of output while using multiprocessing - (`GitLab#17`_) - -.. _GitLab#17: https://gitlab.com/pycqa/flake8/issues/17 - -2.2.5 - 2014-10-19 ------------------- - -- Flush standard out when using multiprocessing - -- Make the check for "# flake8: noqa" more strict - -2.2.4 - 2014-10-09 ------------------- - -- Fix bugs triggered by turning multiprocessing on by default (again) - - Multiprocessing is forcibly disabled in the following cases: - - - Passing something in via stdin - - - Analyzing a diff - - - Using windows - -- Fix --install-hook when there are no config files present for pep8 or - flake8. - -- Fix how the setuptools command parses excludes in config files - -- Fix how the git hook determines which files to analyze (Thanks Chris - Buccella!) - -2.2.3 - 2014-08-25 ------------------- - -- Actually turn multiprocessing on by default - -2.2.2 - 2014-07-04 ------------------- - -- Re-enable multiprocessing by default while fixing the issue Windows users - were seeing. - -2.2.1 - 2014-06-30 ------------------- - -- Turn off multiple jobs by default. To enable automatic use of all CPUs, use - ``--jobs=auto``. Fixes #155 and #154. - -2.2.0 - 2014-06-22 ------------------- - -- New option ``doctests`` to run Pyflakes checks on doctests too -- New option ``jobs`` to launch multiple jobs in parallel -- Turn on using multiple jobs by default using the CPU count -- Add support for ``python -m flake8`` on Python 2.7 and Python 3 -- Fix Git and Mercurial hooks: issues #88, #133, #148 and #149 -- Fix crashes with Python 3.4 by upgrading dependencies -- Fix traceback when running tests with Python 2.6 -- Fix the setuptools command ``python setup.py flake8`` to read - the project configuration - - -2.1.0 - 2013-10-26 ------------------- - -- Add FLAKE8_LAZY and FLAKE8_IGNORE environment variable support to git and - mercurial hooks -- Force git and mercurial hooks to repsect configuration in setup.cfg -- Only check staged files if that is specified -- Fix hook file permissions -- Fix the git hook on python 3 -- Ignore non-python files when running the git hook -- Ignore .tox directories by default -- Flake8 now reports the column number for PyFlakes messages - - -2.0.0 - 2013-02-23 ------------------- - -- Pyflakes errors are prefixed by an ``F`` instead of an ``E`` -- McCabe complexity warnings are prefixed by a ``C`` instead of a ``W`` -- Flake8 supports extensions through entry points -- Due to the above support, we **require** setuptools -- We publish the `documentation `_ -- Fixes #13: pep8, pyflakes and mccabe become external dependencies -- Split run.py into main.py, engine.py and hooks.py for better logic -- Expose our parser for our users -- New feature: Install git and hg hooks automagically -- By relying on pyflakes (0.6.1), we also fixed #45 and #35 - - -1.7.0 - 2012-12-21 ------------------- - -- Fixes part of #35: Exception for no WITHITEM being an attribute of Checker - for Python 3.3 -- Support stdin -- Incorporate @phd's builtins pull request -- Fix the git hook -- Update pep8.py to the latest version - - -1.6.2 - 2012-11-25 ------------------- - -- fixed the NameError: global name 'message' is not defined (#46) - - -1.6.1 - 2012-11-24 ------------------- - -- fixed the mercurial hook, a change from a previous patch was not properly - applied -- fixed an assumption about warnings/error messages that caused an exception - to be thrown when McCabe is used - - -1.6 - 2012-11-16 ----------------- - -- changed the signatures of the ``check_file`` function in flake8/run.py, - ``skip_warning`` in flake8/util.py and the ``check``, ``checkPath`` - functions in flake8/pyflakes.py. -- fix ``--exclude`` and ``--ignore`` command flags (#14, #19) -- fix the git hook that wasn't catching files not already added to the index - (#29) -- pre-emptively includes the addition to pep8 to ignore certain lines. - Add ``# nopep8`` to the end of a line to ignore it. (#37) -- ``check_file`` can now be used without any special prior setup (#21) -- unpacking exceptions will no longer cause an exception (#20) -- fixed crash on non-existent file (#38) - - -1.5 - 2012-10-13 ----------------- - -- fixed the stdin -- make sure mccabe catches the syntax errors as warnings -- pep8 upgrade -- added max_line_length default value -- added Flake8Command and entry points if setuptools is around -- using the setuptools console wrapper when available - - -1.4 - 2012-07-12 ----------------- - -- git_hook: Only check staged changes for compliance -- use pep8 1.2 - - -1.3.1 - 2012-05-19 ------------------- - -- fixed support for Python 2.5 - - -1.3 - 2012-03-12 ----------------- - -- fixed false W402 warning on exception blocks. - - -1.2 - 2012-02-12 ----------------- - -- added a git hook -- now Python 3 compatible -- mccabe and pyflakes have warning codes like pep8 now - - -1.1 - 2012-02-14 ----------------- - -- fixed the value returned by --version -- allow the flake8: header to be more generic -- fixed the "hg hook raises 'physical lines'" bug -- allow three argument form of raise -- now uses setuptools if available, for 'develop' command - - -1.0 - 2011-11-29 ----------------- - -- Deactivates by default the complexity checker -- Introduces the complexity option in the HG hook and the command line. - - -0.9 - 2011-11-09 ----------------- - -- update pep8 version to 0.6.1 -- mccabe check: gracefully handle compile failure - - -0.8 - 2011-02-27 ----------------- - -- fixed hg hook -- discard unexisting files on hook check - - -0.7 - 2010-02-18 ----------------- - -- Fix pep8 initialization when run through Hg -- Make pep8 short options work when run through the command line -- Skip duplicates when controlling files via Hg - - -0.6 - 2010-02-15 ----------------- - -- Fix the McCabe metric on some loops diff --git a/setup.py b/setup.py index 6c3ccdc..4a8e2a8 100644 --- a/setup.py +++ b/setup.py @@ -45,7 +45,7 @@ if sys.version_info < (3, 2): def get_long_description(): """Generate a long description from the README and CHANGES files.""" descr = [] - for fname in ('README.rst', 'CHANGES.rst'): + for fname in ('README.rst',): with open(fname) as f: descr.append(f.read()) return '\n\n'.join(descr) From 25502f6ade5bef3fb9fc5fda0e13c7a16b93f332 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 15 Jun 2016 07:48:46 -0500 Subject: [PATCH 319/364] Add note about # noqa: --- docs/source/release-notes/3.0.0.rst | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/source/release-notes/3.0.0.rst b/docs/source/release-notes/3.0.0.rst index e729723..2969c0d 100644 --- a/docs/source/release-notes/3.0.0.rst +++ b/docs/source/release-notes/3.0.0.rst @@ -3,16 +3,21 @@ - Rewrite our documentation from scratch! (http://flake8.pycqa.org) -- ``--select`` and ``--ignore`` can now both be specified and try to find the - most specific rule from each. For example, if you do ``--select E --ignore - E123`` then we will report everything that starts with ``E`` except for - ``E123``. Previously, you would have had to do ``--ignore E123,F,W`` which - will also still work, but the former should be far more intuitive. +- Drop explicit support for Pythons 2.6, 3.2, and 3.3. - Remove dependence on pep8/pycodestyle for file processing, plugin dispatching, and more. We now control all of this while keeping backwards compatibility. +- ``--select`` and ``--ignore`` can now both be specified and try to find the + most specific rule from each. For example, if you do ``--select E --ignore + E123`` then we will report everything that starts with ``E`` except for + ``E123``. Previously, you would have had to do ``--ignore E123,F,W`` which + will also still work, but the former should be far more intuitive. + +- Add support for in-line ``# noqa`` comments to specify **only** the error + codes to be ignored, e.g., ``# noqa: E123,W503`` + - Add entry-point for formatters as well as a base class that new formatters can inherit from. See the documentation for more details. From b1f039a4768c9289cd38db7d8184f8ab3b53b29e Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 15 Jun 2016 09:20:26 -0500 Subject: [PATCH 320/364] Add information about new --verbose behaviour --- docs/source/release-notes/3.0.0.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/release-notes/3.0.0.rst b/docs/source/release-notes/3.0.0.rst index 2969c0d..605f8a7 100644 --- a/docs/source/release-notes/3.0.0.rst +++ b/docs/source/release-notes/3.0.0.rst @@ -21,6 +21,8 @@ - Add entry-point for formatters as well as a base class that new formatters can inherit from. See the documentation for more details. +- Add detailed verbose output using the standard library logging module. + - Enhance our usage of optparse for plugin developers by adding new parameters to the ``add_option`` that plugins use to register new options. From a4b0793bf79e1649f8382e0ee66d8b9932e553e0 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 06:26:02 -0500 Subject: [PATCH 321/364] Use correct stat module constant Previously we were using stat.IROTH instead of stat.S_IROTH. I must have been thinking about retirement. ;-) --- flake8/main/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/main/git.py b/flake8/main/git.py index 7e55321..3160a26 100644 --- a/flake8/main/git.py +++ b/flake8/main/git.py @@ -89,7 +89,7 @@ def install(): # - read permissions for other people # The owner needs the file to be readable, writable, and executable # so that git can actually execute it as a hook. - pre_commit_permissions = stat.S_IRWXU | stat.S_IRGRP | stat.IROTH + pre_commit_permissions = stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH os.chmod(pre_commit_file, pre_commit_permissions) return True From 27eb975411618bd173db9969a3a11ae9d32fab2b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 06:27:45 -0500 Subject: [PATCH 322/364] Properly unpack stdout and stdin subprocess.Popen#communicate returns (stdout, stdin) but we were only unpacking stdout from it causing an exception. --- flake8/main/git.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/flake8/main/git.py b/flake8/main/git.py index 3160a26..c9b7dad 100644 --- a/flake8/main/git.py +++ b/flake8/main/git.py @@ -169,7 +169,7 @@ def piped_process(command): def git_config_for(parameter): config = piped_process(['git', 'config', '--get', '--bool', parameter]) - (stdout, ) = config.communicate() + (stdout, _) = config.communicate() return to_text(stdout) From 07101231d9aeeaa8d593218cd856945e91e73e31 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 06:29:04 -0500 Subject: [PATCH 323/364] Ensure that the directories exist If we create our temporary directory structure for our files, we need to ensure that all the directories (including the ones in the filename) exist before we copy the contents to that location. --- flake8/main/git.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/flake8/main/git.py b/flake8/main/git.py index c9b7dad..39c2ad1 100644 --- a/flake8/main/git.py +++ b/flake8/main/git.py @@ -116,6 +116,8 @@ def copy_file_to(destination_directory, filepath, contents): directory, filename = os.path.split(os.path.abspath(filepath)) temporary_directory = make_temporary_directory_from(destination_directory, directory) + if not os.path.exists(temporary_directory): + os.makedirs(temporary_directory) temporary_filepath = os.path.join(temporary_directory, filename) with open(temporary_filepath, 'wb') as fd: fd.write(contents) From 29896e648a67a4195429047621b10511a9312d3c Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 06:30:07 -0500 Subject: [PATCH 324/364] Use a template for git pre-commit hook Previously, we forced Flake8 to be installed in whatever Python environment that the user was using. This allows someone to use Flake8 from a virtual environment, e.g., if you use the tox.ini from this commit, you can do: tox -e venv -- flake8 --install-hook git And that will allow you to use the Python and Flake8 from ./.tox/venv/bin/python. This means that you can avoid installing Flake8 globally and still have a working commit hook. --- flake8/main/git.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/flake8/main/git.py b/flake8/main/git.py index 39c2ad1..bae0233 100644 --- a/flake8/main/git.py +++ b/flake8/main/git.py @@ -9,6 +9,7 @@ import os import shutil import stat import subprocess +import sys import tempfile from flake8 import defaults @@ -80,8 +81,10 @@ def install(): path=pre_commit_file, ) + executable = get_executable() + with open(pre_commit_file, 'w') as fd: - fd.write(_HOOK_TEMPLATE) + fd.write(_HOOK_TEMPLATE.format(executable=executable)) # NOTE(sigmavirus24): The following sets: # - read, write, and execute permissions for the owner @@ -94,6 +97,12 @@ def install(): return True +def get_executable(): + if sys.executable is not None: + return sys.executable + return '/usr/bin/env python' + + def find_git_directory(): rev_parse = piped_process(['git', 'rev-parse', '--git-dir']) @@ -182,7 +191,7 @@ def config_for(parameter): return value.lower() in defaults.TRUTHY_VALUES -_HOOK_TEMPLATE = """#!/usr/bin/env python +_HOOK_TEMPLATE = """#!{executable} import os import sys From eff77e2bc2ab2c061e7b71398e1e5e25d44721d2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 06:35:51 -0500 Subject: [PATCH 325/364] Set a lower-limit on flake8-docstrings flake8-docstrings is broken on 0.2.6 with flake8 2.6.0, 0.2.7 was released today which fixes the problem. Let's avoid that version if at all possible. --- tox.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tox.ini b/tox.ini index 0bc1795..9ba3f2c 100644 --- a/tox.ini +++ b/tox.ini @@ -21,7 +21,7 @@ skip_install = true use_develop = false deps = flake8 - flake8-docstrings + flake8-docstrings>=0.2.7 flake8-import-order commands = flake8 From 023de21fe209818282a188edc38c0d4e97f97e6b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 07:20:09 -0500 Subject: [PATCH 326/364] Find filenames for mercurial hook Extract the files changed in a particular commit or set of commits for the commit and qrefresh mercurial hooks. --- flake8/main/mercurial.py | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/flake8/main/mercurial.py b/flake8/main/mercurial.py index 10e5c06..d067612 100644 --- a/flake8/main/mercurial.py +++ b/flake8/main/mercurial.py @@ -32,8 +32,10 @@ def hook(ui, repo, **kwargs): hgconfig = configparser_for(hgrc) strict = hgconfig.get('flake8', 'strict', fallback=True) + filenames = list(get_filenames_from(repo, kwargs)) + app = application.Application() - app.run() + app.run(filenames) if strict: return app.result_count @@ -78,6 +80,22 @@ def install(): return True +def get_filenames_from(repository, kwargs): + seen_filenames = set() + node = kwargs['node'] + for revision in range(repository[node], len(repository)): + for filename in repository[revision].files(): + full_filename = os.path.join(repository.root, filename) + have_seen_filename = full_filename in seen_filenames + filename_does_not_exist = not os.path.exists(full_filename) + if have_seen_filename or filename_does_not_exist: + continue + + seen_filenames.add(full_filename) + if full_filename.endswith('.py'): + yield full_filename + + def find_hgrc(create_if_missing=False): root = subprocess.Popen( ['hg', 'root'], From 1de71fbe89e4e3cd9cb4b2b43a1dfd57e7bc4da2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 08:27:28 -0500 Subject: [PATCH 327/364] Set a default value for --select This allows us to add --append-to-select and --append-to-ignore at a later date that will work as expected and in a consistent manner. --- docs/source/user/options.rst | 2 +- flake8/defaults.py | 1 + flake8/main/options.py | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index 64ecaeb..2c41bde 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -354,7 +354,7 @@ that start with that string. For example, you can use ``E``, ``E4``, ``E43``, and ``E431``. - This has no default value. + This defaults to: E,F,W,C Command-line example: diff --git a/flake8/defaults.py b/flake8/defaults.py index 2007eed..76829a5 100644 --- a/flake8/defaults.py +++ b/flake8/defaults.py @@ -2,6 +2,7 @@ EXCLUDE = '.svn,CVS,.bzr,.hg,.git,__pycache__,.tox' IGNORE = 'E121,E123,E126,E226,E24,E704,W503,W504' +SELECT = 'E,F,W,C' MAX_LINE_LENGTH = 79 TRUTHY_VALUES = set(['true', '1', 't']) diff --git a/flake8/main/options.py b/flake8/main/options.py index 9be4386..51db34e 100644 --- a/flake8/main/options.py +++ b/flake8/main/options.py @@ -112,7 +112,7 @@ def register_default_options(option_manager): ) add_option( - '--select', metavar='errors', default='', + '--select', metavar='errors', default=defaults.SELECT, parse_from_config=True, comma_separated_list=True, help='Comma-separated list of errors and warnings to enable.' ' For example, ``--select=E4,E51,W234``. (Default: %default)', From 9a9bcdfb5252aef2fb4985a7769fd263c6d2a651 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 10:59:16 -0500 Subject: [PATCH 328/364] Add setuptools integration This proved simpler because I realized something important: Most of the code that the old integration was using was in fact doing nothing of value. Since we can't meaningfully allow users to use command-line options as parameters to `python setup.py flake8`, we just remove the work that we do to set up those attributes and parse them. --- flake8/main/setuptools_command.py | 77 +++++++++++++++++++++++++++++++ setup.py | 8 +++- 2 files changed, 83 insertions(+), 2 deletions(-) create mode 100644 flake8/main/setuptools_command.py diff --git a/flake8/main/setuptools_command.py b/flake8/main/setuptools_command.py new file mode 100644 index 0000000..1c27bf6 --- /dev/null +++ b/flake8/main/setuptools_command.py @@ -0,0 +1,77 @@ +"""The logic for Flake8's integration with setuptools.""" +import os + +import setuptools + +from flake8.main import application as app + + +class Flake8(setuptools.Command): + """Run Flake8 via setuptools/distutils for registered modules.""" + + description = 'Run Flake8 on modules registered in setup.py' + # NOTE(sigmavirus24): If we populated this with a list of tuples, users + # could do something like ``python setup.py flake8 --ignore=E123,E234`` + # but we would have to redefine it and we can't define it dynamically. + # Since I refuse to copy-and-paste the options here or maintain two lists + # of options, and since this will break when users use plugins that + # provide command-line options, we are leaving this empty. If users want + # to configure this command, they can do so through config files. + user_options = [] + + def initialize_options(self): + """Override this method to initialize our application.""" + pass + + def finalize_options(self): + """Override this to parse the parameters.""" + pass + + def package_files(self): + """Collect the files/dirs included in the registered modules.""" + seen_package_directories = () + directories = self.distribution.package_dir or {} + empty_directory_exists = '' in directories + packages = self.distribution.packages or [] + for package in packages: + package_directory = package + if package in directories: + package_directory = directories[package] + elif empty_directory_exists: + package_directory = os.path.join(directories[''], + package_directory) + + # NOTE(sigmavirus24): Do not collect submodules, e.g., + # if we have: + # - flake8/ + # - flake8/plugins/ + # Flake8 only needs ``flake8/`` to be provided. It will + # recurse on its own. + if package_directory.startswith(seen_package_directories): + continue + + seen_package_directories += (package_directory,) + yield package_directory + + def module_files(self): + """Collect the files listed as py_modules.""" + modules = self.distribution.py_modules or [] + filename_from = '{0}.py'.format + for module in modules: + yield filename_from(module) + + def distribution_files(self): + """Collect package and module files.""" + for package in self.package_files(): + yield package + + for module in self.module_files(): + yield module + + yield 'setup.py' + + def run(self): + """Run the Flake8 application.""" + flake8 = app.Application() + flake8.run(list(self.distribution_files())) + flake8.exit() diff --git a/setup.py b/setup.py index 4a8e2a8..b31fb19 100644 --- a/setup.py +++ b/setup.py @@ -76,8 +76,12 @@ setuptools.setup( ], install_requires=requires, entry_points={ - 'distutils.commands': ['flake8 = flake8.main:Flake8Command'], - 'console_scripts': ['flake8 = flake8.main.cli:main'], + 'distutils.commands': [ + 'flake8 = flake8.main.setuptools_command:Flake8' + ], + 'console_scripts': [ + 'flake8 = flake8.main.cli:main' + ], 'flake8.extension': [ 'F = flake8.plugins.pyflakes:FlakesChecker', # PEP-0008 checks provied by PyCQA/pycodestyle From 3f434f7d1cf3f7a8e48456410db94376bae4bc24 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Thu, 16 Jun 2016 16:19:09 -0500 Subject: [PATCH 329/364] Add broken config file to test error handling ConfigFileFinder should absolutely handle broken/invalid config files by refusing to try to parse them. Here we catch the ParsingError, log the exception, and then return normally. The RawConfigParser instance is perfectly valid still and will behave as if nothing had been read and we just need to indicate that we didn't find any files worthy of reading. Related to: https://github.com/PyCQA/pycodestyle/issues/506 --- flake8/options/config.py | 8 +++++++- tests/fixtures/config_files/broken.ini | 9 +++++++++ tests/unit/test_config_file_finder.py | 10 ++++++++++ 3 files changed, 26 insertions(+), 1 deletion(-) create mode 100644 tests/fixtures/config_files/broken.ini diff --git a/flake8/options/config.py b/flake8/options/config.py index 9bd7e19..48719a8 100644 --- a/flake8/options/config.py +++ b/flake8/options/config.py @@ -52,7 +52,13 @@ class ConfigFileFinder(object): @staticmethod def _read_config(files): config = configparser.RawConfigParser() - found_files = config.read(files) + try: + found_files = config.read(files) + except configparser.ParsingError: + LOG.exception("There was an error trying to parse a config " + "file. The files we were attempting to parse " + "were: %r", files) + found_files = [] return (config, found_files) def cli_config(self, files): diff --git a/tests/fixtures/config_files/broken.ini b/tests/fixtures/config_files/broken.ini new file mode 100644 index 0000000..33986ae --- /dev/null +++ b/tests/fixtures/config_files/broken.ini @@ -0,0 +1,9 @@ +[flake8] +exclude = +<<<<<<< 642f88cb1b6027e184d9a662b255f7fea4d9eacc + tests/fixtures/, +======= + tests/, +>>>>>>> HEAD + docs/ +ignore = D203 diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 8d7e920..2c56685 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -10,6 +10,7 @@ import mock import pytest CLI_SPECIFIED_FILEPATH = 'tests/fixtures/config_files/cli-specified.ini' +BROKEN_CONFIG_PATH = 'tests/fixtures/config_files/broken.ini' def test_uses_default_args(): @@ -115,3 +116,12 @@ def test_local_configs(): finder = config.ConfigFileFinder('flake8', None, []) assert isinstance(finder.local_configs(), configparser.RawConfigParser) + + +@pytest.mark.parametrize('files', [ + [BROKEN_CONFIG_PATH], + [CLI_SPECIFIED_FILEPATH, BROKEN_CONFIG_PATH], +]) +def test_read_config_catches_broken_config_files(files): + """Verify that we do not allow the exception to bubble up.""" + assert config.ConfigFileFinder._read_config(files)[1] == [] From 8bc76f79debddb80833767075118a63712048704 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 17 Jun 2016 10:26:36 -0500 Subject: [PATCH 330/364] Configure flake8-import-order to use Google Style This relies on two things: 1. Properly configuring flake8-import-order to use that style 2. Properly configuring flake8-import-order to know that flake8 is our application name. --- flake8/main/application.py | 6 ++++++ flake8/plugins/manager.py | 4 ++-- flake8/plugins/pyflakes.py | 4 ++-- tests/integration/test_aggregator.py | 4 ++-- tests/unit/test_base_formatter.py | 7 +++---- tests/unit/test_checker_manager.py | 5 ++--- tests/unit/test_config_file_finder.py | 5 ++--- tests/unit/test_merged_config_parser.py | 7 +++---- tests/unit/test_notifier.py | 4 ++-- tests/unit/test_option.py | 5 ++--- tests/unit/test_option_manager.py | 4 ++-- tests/unit/test_plugin.py | 7 +++---- tests/unit/test_plugin_manager.py | 4 ++-- tests/unit/test_plugin_type_manager.py | 7 +++---- tests/unit/test_style_guide.py | 7 +++---- tests/unit/test_utils.py | 6 ++---- tox.ini | 2 ++ 17 files changed, 43 insertions(+), 45 deletions(-) diff --git a/flake8/main/application.py b/flake8/main/application.py index e41a6c2..6739216 100644 --- a/flake8/main/application.py +++ b/flake8/main/application.py @@ -3,6 +3,7 @@ from __future__ import print_function import logging import sys +import time import flake8 from flake8 import checker @@ -28,6 +29,10 @@ class Application(object): :param str version: The version of the program/application we're executing. """ + #: The timestamp when the Application instance was instantiated. + self.start_time = time.time() + #: The timestamp when the Application finished reported errors. + self.end_time = None #: The name of the program being run self.program = program #: The version of the program being run @@ -254,6 +259,7 @@ class Application(object): self.initialize(argv) self.run_checks() self.report_errors() + self.end_time = time.time() def run(self, argv=None): # type: (Union[NoneType, List[str]]) -> NoneType diff --git a/flake8/plugins/manager.py b/flake8/plugins/manager.py index dffc2f6..d08d542 100644 --- a/flake8/plugins/manager.py +++ b/flake8/plugins/manager.py @@ -2,12 +2,12 @@ import collections import logging +import pkg_resources + from flake8 import exceptions from flake8 import utils from flake8.plugins import notifier -import pkg_resources - LOG = logging.getLogger(__name__) __all__ = ( diff --git a/flake8/plugins/pyflakes.py b/flake8/plugins/pyflakes.py index 7025baa..72d45fa 100644 --- a/flake8/plugins/pyflakes.py +++ b/flake8/plugins/pyflakes.py @@ -11,11 +11,11 @@ else: demandimport.disable() import os -from flake8 import utils - import pyflakes import pyflakes.checker +from flake8 import utils + def patch_pyflakes(): """Add error codes to Pyflakes messages.""" diff --git a/tests/integration/test_aggregator.py b/tests/integration/test_aggregator.py index d0a7971..929bdbf 100644 --- a/tests/integration/test_aggregator.py +++ b/tests/integration/test_aggregator.py @@ -1,12 +1,12 @@ """Test aggregation of config files and command-line options.""" import os +import pytest + from flake8.main import options from flake8.options import aggregator from flake8.options import manager -import pytest - CLI_SPECIFIED_CONFIG = 'tests/fixtures/config_files/cli-specified.ini' diff --git a/tests/unit/test_base_formatter.py b/tests/unit/test_base_formatter.py index a23a9cb..dc4a95c 100644 --- a/tests/unit/test_base_formatter.py +++ b/tests/unit/test_base_formatter.py @@ -1,13 +1,12 @@ """Tests for the BaseFormatter object.""" import optparse +import mock +import pytest + from flake8 import style_guide from flake8.formatting import base -import mock - -import pytest - def options(**kwargs): """Create an optparse.Values instance.""" diff --git a/tests/unit/test_checker_manager.py b/tests/unit/test_checker_manager.py index 5b91204..8240723 100644 --- a/tests/unit/test_checker_manager.py +++ b/tests/unit/test_checker_manager.py @@ -1,12 +1,11 @@ """Tests for the Manager object for FileCheckers.""" import errno -from flake8 import checker - import mock - import pytest +from flake8 import checker + def style_guide_mock(**kwargs): """Create a mock StyleGuide object.""" diff --git a/tests/unit/test_config_file_finder.py b/tests/unit/test_config_file_finder.py index 2c56685..2e1a1e5 100644 --- a/tests/unit/test_config_file_finder.py +++ b/tests/unit/test_config_file_finder.py @@ -3,12 +3,11 @@ import configparser import os import sys -from flake8.options import config - import mock - import pytest +from flake8.options import config + CLI_SPECIFIED_FILEPATH = 'tests/fixtures/config_files/cli-specified.ini' BROKEN_CONFIG_PATH = 'tests/fixtures/config_files/broken.ini' diff --git a/tests/unit/test_merged_config_parser.py b/tests/unit/test_merged_config_parser.py index baaa57a..eb57802 100644 --- a/tests/unit/test_merged_config_parser.py +++ b/tests/unit/test_merged_config_parser.py @@ -1,13 +1,12 @@ """Unit tests for flake8.options.config.MergedConfigParser.""" import os +import mock +import pytest + from flake8.options import config from flake8.options import manager -import mock - -import pytest - @pytest.fixture def optmanager(): diff --git a/tests/unit/test_notifier.py b/tests/unit/test_notifier.py index 8c001da..6a162cf 100644 --- a/tests/unit/test_notifier.py +++ b/tests/unit/test_notifier.py @@ -1,8 +1,8 @@ """Unit tests for the Notifier object.""" -from flake8.plugins import notifier - import pytest +from flake8.plugins import notifier + class _Listener(object): def __init__(self, error_code): diff --git a/tests/unit/test_option.py b/tests/unit/test_option.py index 45f9be7..67e2255 100644 --- a/tests/unit/test_option.py +++ b/tests/unit/test_option.py @@ -1,10 +1,9 @@ """Unit tests for flake8.options.manager.Option.""" -from flake8.options import manager - import mock - import pytest +from flake8.options import manager + def test_to_optparse(): """Test conversion to an optparse.Option class.""" diff --git a/tests/unit/test_option_manager.py b/tests/unit/test_option_manager.py index 0086585..53e8bf1 100644 --- a/tests/unit/test_option_manager.py +++ b/tests/unit/test_option_manager.py @@ -2,10 +2,10 @@ import optparse import os -from flake8.options import manager - import pytest +from flake8.options import manager + TEST_VERSION = '3.0.0b1' diff --git a/tests/unit/test_plugin.py b/tests/unit/test_plugin.py index d3d1185..f69bc05 100644 --- a/tests/unit/test_plugin.py +++ b/tests/unit/test_plugin.py @@ -1,13 +1,12 @@ """Tests for flake8.plugins.manager.Plugin.""" import optparse +import mock +import pytest + from flake8 import exceptions from flake8.plugins import manager -import mock - -import pytest - def test_load_plugin_fallsback_on_old_setuptools(): """Verify we fallback gracefully to on old versions of setuptools.""" diff --git a/tests/unit/test_plugin_manager.py b/tests/unit/test_plugin_manager.py index 5a50386..8991b96 100644 --- a/tests/unit/test_plugin_manager.py +++ b/tests/unit/test_plugin_manager.py @@ -1,8 +1,8 @@ """Tests for flake8.plugins.manager.PluginManager.""" -from flake8.plugins import manager - import mock +from flake8.plugins import manager + def create_entry_point_mock(name): """Create a mocked EntryPoint.""" diff --git a/tests/unit/test_plugin_type_manager.py b/tests/unit/test_plugin_type_manager.py index fb17a37..186c2e3 100644 --- a/tests/unit/test_plugin_type_manager.py +++ b/tests/unit/test_plugin_type_manager.py @@ -1,13 +1,12 @@ """Tests for flake8.plugins.manager.PluginTypeManager.""" import collections +import mock +import pytest + from flake8 import exceptions from flake8.plugins import manager -import mock - -import pytest - TEST_NAMESPACE = "testing.plugin-type-manager" diff --git a/tests/unit/test_style_guide.py b/tests/unit/test_style_guide.py index 4aa3c54..3d05528 100644 --- a/tests/unit/test_style_guide.py +++ b/tests/unit/test_style_guide.py @@ -1,14 +1,13 @@ """Tests for the flake8.style_guide.StyleGuide class.""" import optparse +import mock +import pytest + from flake8 import style_guide from flake8.formatting import base from flake8.plugins import notifier -import mock - -import pytest - def create_options(**kwargs): """Create and return an instance of optparse.Values.""" diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 54aa3a7..062f7f9 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -1,13 +1,11 @@ """Tests for flake8's utils module.""" import os -from flake8 import utils -from flake8.plugins import manager as plugin_manager - import mock - import pytest +from flake8 import utils +from flake8.plugins import manager as plugin_manager RELATIVE_PATHS = ["flake8", "pep8", "pyflakes", "mccabe"] diff --git a/tox.ini b/tox.ini index 9ba3f2c..c17f025 100644 --- a/tox.ini +++ b/tox.ini @@ -121,3 +121,5 @@ ignore = D203 # :-( exclude = .git,__pycache__,docs/source/conf.py,old,build,dist,tests/fixtures/ max-complexity = 10 +import-order-style = google +application-import-names = flake8 From fd5c7d009df0a40377a7cf0826195d72c4521f07 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 17 Jun 2016 11:10:16 -0500 Subject: [PATCH 331/364] Update Travis to run the linters individually --- .travis.yml | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 889f406..c9a85f4 100644 --- a/.travis.yml +++ b/.travis.yml @@ -24,6 +24,14 @@ matrix: - python: 2.7 env: TOXENV=readme - python: 3.4 - env: TOXENV=linters + env: TOXENV=flake8 + - python: 3.4 + env: TOXENV=pylint + - python: 3.4 + env: TOXENV=doc8 + - python: 3.4 + env: TOXENV=mypy + - python: 3.4 + env: TOXENV=bandit - python: 3.4 env: TOXENV=docs From 330fdb5b5623e09151a4c6576cb3738563c13769 Mon Sep 17 00:00:00 2001 From: Cea Stapleton Date: Sun, 19 Jun 2016 14:06:26 -0500 Subject: [PATCH 332/364] Grammar and clarity improvements. --- docs/source/dev/formatters.rst | 2 +- docs/source/dev/index.rst | 4 ++-- docs/source/dev/registering_plugins.rst | 6 +++--- docs/source/index.rst | 2 +- docs/source/internal/checker.rst | 18 +++++++++--------- docs/source/internal/cli.rst | 4 ++-- docs/source/internal/contributing.rst | 10 +++++----- docs/source/internal/formatters.rst | 2 +- docs/source/internal/option_handling.rst | 14 +++++++------- docs/source/internal/plugin_handling.rst | 8 ++++---- docs/source/internal/utils.rst | 2 +- docs/source/user/configuration.rst | 4 ++-- docs/source/user/ignoring-errors.rst | 4 ++-- docs/source/user/index.rst | 8 ++++---- docs/source/user/options.rst | 2 +- 15 files changed, 45 insertions(+), 45 deletions(-) diff --git a/docs/source/dev/formatters.rst b/docs/source/dev/formatters.rst index 5307e75..5beafda 100644 --- a/docs/source/dev/formatters.rst +++ b/docs/source/dev/formatters.rst @@ -4,7 +4,7 @@ Developing a Formatting Plugin for Flake8 =========================================== -Flake8 added the ability to develop custom formatting plugins in version +Flake8 allowed for custom formatting plugins in version 3.0.0. Let's write a plugin together: .. code-block:: python diff --git a/docs/source/dev/index.rst b/docs/source/dev/index.rst index 43f01c1..4969395 100644 --- a/docs/source/dev/index.rst +++ b/docs/source/dev/index.rst @@ -4,9 +4,9 @@ Since Flake8 2.0, the Flake8 tool has allowed for extensions and custom plugins. In Flake8 3.0, we're expanding that ability to customize and -extend **and** we're attempting to thoroughly document it too. Some of the +extend **and** we're attempting to thoroughly document it. Some of the documentation in this section will reference third-party documentation -in order to reduce duplication and to point you, the developer, towards +to reduce duplication and to point you, the developer, towards the authoritative documentation for those pieces. .. toctree:: diff --git a/docs/source/dev/registering_plugins.rst b/docs/source/dev/registering_plugins.rst index 0cc18d9..53b6dc4 100644 --- a/docs/source/dev/registering_plugins.rst +++ b/docs/source/dev/registering_plugins.rst @@ -4,9 +4,9 @@ Registering a Plugin with Flake8 ================================== -To register any kind of plugin with Flake8, you need a few things: +To register any kind of plugin with Flake8, you need: -#. You need a way to install the plugin (whether it is packaged on its own or +#. A way to install the plugin (whether it is packaged on its own or as part of something else). In this section, we will use a ``setup.py`` written for an example plugin. @@ -15,7 +15,7 @@ To register any kind of plugin with Flake8, you need a few things: #. A somewhat recent version of setuptools (newer than 0.7.0 but preferably as recent as you can attain). -Flake8 presently relies on a functionality provided by setuptools called +Flake8 relies on functionality provided by setuptools called `Entry Points`_. These allow any package to register a plugin with Flake8 via that package's ``setup.py`` file. diff --git a/docs/source/index.rst b/docs/source/index.rst index 3343801..da383c6 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -30,7 +30,7 @@ can instead use: It is **very** important to install Flake8 on the *correct* version of Python for your needs. If you want Flake8 to properly parse new language features in Python 3.5 (for example), you need it to be installed on 3.5 - for those features to be understandable. In many ways, Flake8 is tied to + for flake8 to understand those features. In many ways, Flake8 is tied to the version of Python on which it runs. Quickstart diff --git a/docs/source/internal/checker.rst b/docs/source/internal/checker.rst index 4c2789f..e82e62f 100644 --- a/docs/source/internal/checker.rst +++ b/docs/source/internal/checker.rst @@ -2,17 +2,17 @@ How Checks are Run ==================== -In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 we have taken -that responsibility upon ourselves. This has allowed us to simplify our -handling of the ``--jobs`` parameter (using :mod:`multiprocessing`) as well as -simplifying our fallback in the event something goes awry with concurency. +In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 flake8 takes +on that responsibility. This has allowed for simpler +handling of the ``--jobs`` parameter (using :mod:`multiprocessing`) and +simplified our fallback if something goes awry with concurency. At the lowest level we have a |FileChecker|. Instances of |FileChecker| are created for *each* file to be analyzed by Flake8. Each instance, has a copy of all of the plugins registered with setuptools in the ``flake8.extension`` entry-point group. The |FileChecker| instances are managed by an instance of |Manager|. The -|Manager| instance is what handles creating sub-processes with +|Manager| instance handles creating sub-processes with :mod:`multiprocessing` module and falling back to running checks in serial if an operating system level error arises. When creating |FileChecker| instances, the |Manager| is responsible for determining if a particular file has been @@ -23,10 +23,10 @@ Processing Files ---------------- Unfortunately, since Flake8 took over check running from pep8/pycodestyle, it -also was required to take over parsing and processing files for the checkers -to use. Since we couldn't reuse pycodestyle's functionality (since it did not -separate cleanly the processing from check running) we isolated that function -into the :class:`~flake8.processor.FileProcessor` class. Further, we moved +also had to take over parsing and processing files for the checkers +to use. Since it couldn't reuse pycodestyle's functionality (since it did not +separate cleanly the processing from check running) that function was isolated +into the :class:`~flake8.processor.FileProcessor` class. We moved several helper functions into the :mod:`flake8.processor` module (see also :ref:`Processor Utility Functions `). diff --git a/docs/source/internal/cli.rst b/docs/source/internal/cli.rst index a1e9612..bb8f774 100644 --- a/docs/source/internal/cli.rst +++ b/docs/source/internal/cli.rst @@ -7,8 +7,8 @@ command line, :func:`~flake8.main.cli.main` is run which handles management of the application. User input is parsed *twice* to accomodate logging and verbosity options -passed by the user as early as possible so as much logging can be produced as -possible. +passed by the user as early as possible. +This is so as much logging can be produced as possible. The default flake8 options are registered by :func:`~flake8.main.options.register_default_options`. Trying to register diff --git a/docs/source/internal/contributing.rst b/docs/source/internal/contributing.rst index 255929b..2406e9a 100644 --- a/docs/source/internal/contributing.rst +++ b/docs/source/internal/contributing.rst @@ -2,7 +2,7 @@ Contributing to Flake8 ======================== -We encourage multiple methods of participation in Flake8: +There are many ways to contriubte to Flake8, and we encourage them all: - contributing bug reports and feature requests @@ -44,7 +44,7 @@ To contribute to Flake8's development, you simply need: python -m pip install --user tox -- and your favorite editor +- your favorite editor Filing a Bug @@ -166,7 +166,7 @@ Reviewing and Triaging Issues and Merge Requests ================================================ When reviewing other people's merge requests and issues, please be -**especially** mindful of how the words you choose can be ready by someone +**especially** mindful of how the words you choose can be read by someone else. We strive for professional code reviews that do not insult the contributor's intelligence or impugn their character. The code review should be focused on the code, it's effectiveness, and whether it is @@ -175,8 +175,8 @@ appropriate for Flake8. If you have the ability to edit an issue or merge request's labels, please do so to make search and prioritization easier. -Flake8 uses milestones with both issues and merge requests to provide -other contributors direction about when an issue or merge request will be +Flake8 uses milestones with both issues and merge requests. This provides +direction for other contributors about when an issue or merge request will be delivered. diff --git a/docs/source/internal/formatters.rst b/docs/source/internal/formatters.rst index d54cf87..b371307 100644 --- a/docs/source/internal/formatters.rst +++ b/docs/source/internal/formatters.rst @@ -20,7 +20,7 @@ Default Formatter The |DefaultFormatter| continues to use the same default format string as pep8: ``'%(path)s:%(row)d:%(col)d: %(code)s %(text)s'``. -In order to provide the default functionality it overrides two methods: +To provide the default functionality it overrides two methods: #. ``after_init`` diff --git a/docs/source/internal/option_handling.rst b/docs/source/internal/option_handling.rst index 8ab1911..1a5b31f 100644 --- a/docs/source/internal/option_handling.rst +++ b/docs/source/internal/option_handling.rst @@ -6,8 +6,8 @@ Option Management Command-line options are often also set in configuration files for Flake8. While not all options are meant to be parsed from configuration files, many -default options are also parsed from configuration files as are most plugin -options. +default options are also parsed from configuration files as well as +most plugin options. In Flake8 2, plugins received a :class:`optparse.OptionParser` instance and called :meth:`optparse.OptionParser.add_option` to register options. If the @@ -19,7 +19,7 @@ also had to do something like: parser.config_options.append('my_config_option') parser.config_options.extend(['config_opt1', 'config_opt2']) -This was previously undocumented and led to a lot of confusion as to why +This was previously undocumented and led to a lot of confusion about why registered options were not automatically parsed from configuration files. Since Flake8 3 was rewritten from scratch, we decided to take a different @@ -40,8 +40,8 @@ three new parameters: - ``normalize_paths`` The last two are not specifically for configuration file handling, but they -do improve that dramatically. We found that there were options that when -specified in a configuration file, lended themselves to being split across +do improve that dramatically. We found that there were options that, when +specified in a configuration file, often necessitated being spit multiple lines and those options were almost always comma-separated. For example, let's consider a user's list of ignored error codes for a project: @@ -82,7 +82,7 @@ Presently OpenStack's Nova project has this line in their `tox.ini`_: exclude = .venv,.git,.tox,dist,doc,*openstack/common/*,*lib/python*,*egg,build,tools/xenserver*,releasenotes -I think we can all agree that this would be easier to read like this: +We think we can all agree that this would be easier to read like this: .. code-block:: ini @@ -104,7 +104,7 @@ both ``comma_separated_list=True`` and ``normalize_paths=True`` because we want the paths to be provided to us with some consistency (either all absolute paths or not). -Now let's look at how this would actually be utilized. Most plugin developers +Now let's look at how this will actually be used. Most plugin developers will receive an instance of :class:`~flake8.options.manager.OptionManager` so to ease the transition we kept the same API as the :class:`optparse.OptionParser` object. The only difference is that diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst index d4692ce..65b2fa1 100644 --- a/docs/source/internal/plugin_handling.rst +++ b/docs/source/internal/plugin_handling.rst @@ -13,8 +13,8 @@ checks. It now supports: - listeners to auto-correct violations of checks -To facilitate this, Flake8 needed a more mature way of managing plugins. As -such, we developed the |PluginManager| which accepts a namespace and will load +To facilitate this, Flake8 needed a more mature way of managing plugins. +Thus, we developed the |PluginManager| which accepts a namespace and will load the plugins for that namespace. A |PluginManager| creates and manages many |Plugin| instances. @@ -24,7 +24,7 @@ The entry-point will be loaded either by calling attribute. We also use this abstraction to retrieve options that the plugin wishes to register and parse. -The only public method that the |PluginManager| provides is +The only public method the |PluginManager| provides is :meth:`~flake8.plugins.manager.PluginManager.map`. This will accept a function (or other callable) and call it with each plugin as the first parameter. @@ -59,7 +59,7 @@ Notifying Listener Plugins One of the interesting challenges with allowing plugins to be notified each time an error or warning is emitted by a checker is finding listeners quickly and efficiently. It makes sense to allow a listener to listen for a certain -class of warnings or just a specific warning. As such, we need to allow all +class of warnings or just a specific warning. Hence, we need to allow all plugins that listen to a specific warning or class to be notified. For example, someone might register a listener for ``E1`` and another for ``E111`` if ``E111`` is triggered by the code, both listeners should be notified. diff --git a/docs/source/internal/utils.rst b/docs/source/internal/utils.rst index 98b2f27..573938c 100644 --- a/docs/source/internal/utils.rst +++ b/docs/source/internal/utils.rst @@ -120,7 +120,7 @@ argument so we can check the parameters of the plugin consistently. .. autofunction:: flake8.utils.parse_unified_diff -In order to handle usage of :option:`flake8 --diff`, Flake8 needs to be able +To handle usage of :option:`flake8 --diff`, Flake8 needs to be able to parse the name of the files in the diff as well as the ranges indicated the sections that have been changed. This function either accepts the diff as an argument or reads the diff from standard-in. It then returns a dictionary with diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst index ddf85f7..6085bcd 100644 --- a/docs/source/user/configuration.rst +++ b/docs/source/user/configuration.rst @@ -24,7 +24,7 @@ Remember that you want to specify certain options without writing Configuration Locations ======================= -Presently, Flake8 supports storing its configuration in the following places: +Flake8 supports storing its configuration in the following places: - Your top-level user directory @@ -163,7 +163,7 @@ This would allow us to add comments for why we're excluding items, e.g., :mod:`configparser` backport from PyPI. That backport enables us to support this behaviour on all supported versions of Python. - Please do **not** open issues about this dependency. + Please do **not** open issues about this dependency to Flake8. .. note:: diff --git a/docs/source/user/ignoring-errors.rst b/docs/source/user/ignoring-errors.rst index 1630c10..b9af8a8 100644 --- a/docs/source/user/ignoring-errors.rst +++ b/docs/source/user/ignoring-errors.rst @@ -4,7 +4,7 @@ By default, |Flake8| has a list of error codes that it ignores. The list used by a version of |Flake8| may be different than the list used by a different -version. To determine the default list, using :option:`flake8 --help` will +version. To see the default list, :option:`flake8 --help` will show the output with the current default list. @@ -42,7 +42,7 @@ specific error code on a specific line. Let's take for example a line like example = lambda: 'example' Sometimes we genuinely need something this simple. We could instead define -a function like we normally would but in some contexts that distracts from +a function like we normally would. Note, in some contexts this distracts from what is actually happening. In those cases, we can also do: .. code-block:: python diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 2ed779e..4644efc 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -2,13 +2,13 @@ Using Flake8 ============== -Flake8 can be used in many ways: +Flake8 can be used in many ways. A few: -- invoking it on the command-line +- invoked on the command-line -- invoking it via Python +- invoked via Python -- having it called by Git or Mercurial on or around committing +- called by Git or Mercurial on or around committing This guide will cover all of these and the nuances for using Flake8. diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index 2c41bde..ecbf52d 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -228,7 +228,7 @@ - pylint Other formatters can be installed. Refer to their documentation for the - name to use to select them. Further, users can specify they're own format + name to use to select them. Further, users can specify their own format string. The variables available are: - code From adedd6c5cfd705d01e79a0f673ed95ee4bb26e08 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 19 Jun 2016 09:25:27 -0500 Subject: [PATCH 333/364] Add --benchmark and formatting for its values --- flake8/checker.py | 60 ++++++++++++++++++++++++++++++++++++-- flake8/defaults.py | 8 ++++- flake8/formatting/base.py | 19 +++++++++++- flake8/main/application.py | 19 +++++++++++- flake8/main/options.py | 7 +++++ flake8/processor.py | 5 ++++ 6 files changed, 112 insertions(+), 6 deletions(-) diff --git a/flake8/checker.py b/flake8/checker.py index 655387e..b875f44 100644 --- a/flake8/checker.py +++ b/flake8/checker.py @@ -10,6 +10,12 @@ try: except ImportError: multiprocessing = None +try: + import Queue as queue +except ImportError: + import queue + +from flake8 import defaults from flake8 import exceptions from flake8 import processor from flake8 import utils @@ -72,14 +78,22 @@ class Manager(object): self.jobs = self._job_count() self.process_queue = None self.results_queue = None + self.statistics_queue = None self.using_multiprocessing = self.jobs > 1 self.processes = [] self.checkers = [] + self.statistics = { + 'files': 0, + 'logical lines': 0, + 'physical lines': 0, + 'tokens': 0, + } if self.using_multiprocessing: try: self.process_queue = multiprocessing.Queue() self.results_queue = multiprocessing.Queue() + self.statistics_queue = multiprocessing.Queue() except OSError as oserr: if oserr.errno not in SERIAL_RETRY_ERRNOS: raise @@ -96,6 +110,29 @@ class Manager(object): proc.join(0.2) self._cleanup_queue(self.process_queue) self._cleanup_queue(self.results_queue) + self._cleanup_queue(self.statistics_queue) + + def _process_statistics(self): + all_statistics = self.statistics + if self.using_multiprocessing: + total_number_of_checkers = len(self.checkers) + statistics_gathered = 0 + while statistics_gathered < total_number_of_checkers: + try: + statistics = self.statistics_queue.get(block=False) + statistics_gathered += 1 + except queue.Empty: + break + + for statistic in defaults.STATISTIC_NAMES: + all_statistics[statistic] += statistics[statistic] + else: + statistics_generator = (checker.statistics + for checker in self.checkers) + for statistics in statistics_generator: + for statistic in defaults.STATISTIC_NAMES: + all_statistics[statistic] += statistics[statistic] + all_statistics['files'] += len(self.checkers) def _job_count(self): # type: () -> Union[int, NoneType] @@ -182,7 +219,7 @@ class Manager(object): LOG.info('Running checks in parallel') for checker in iter(self.process_queue.get, 'DONE'): LOG.debug('Running checker for file "%s"', checker.filename) - checker.run_checks(self.results_queue) + checker.run_checks(self.results_queue, self.statistics_queue) self.results_queue.put('DONE') def is_path_excluded(self, path): @@ -280,7 +317,7 @@ class Manager(object): def run_serial(self): """Run the checkers in serial.""" for checker in self.checkers: - checker.run_checks(self.results_queue) + checker.run_checks(self.results_queue, self.statistics_queue) def run(self): """Run all the checkers. @@ -325,6 +362,7 @@ class Manager(object): def stop(self): """Stop checking files.""" + self._process_statistics() for proc in self.processes: LOG.info('Joining %s to the main process', proc.name) proc.join() @@ -342,12 +380,21 @@ class FileChecker(object): The plugins registered to check the file. :type checks: flake8.plugins.manager.Checkers + :param style_guide: + The initialized StyleGuide for this particular run. + :type style_guide: + flake8.style_guide.StyleGuide """ self.filename = filename self.checks = checks self.style_guide = style_guide self.results = [] self.processor = self._make_processor() + self.statistics = { + 'tokens': 0, + 'logical lines': 0, + 'physical lines': len(self.processor.lines), + } def _make_processor(self): try: @@ -466,8 +513,10 @@ class FileChecker(object): :meth:`flake8.checker.FileChecker.run_checks`. """ parens = 0 + statistics = self.statistics file_processor = self.processor for token in file_processor.generate_tokens(): + statistics['tokens'] += 1 self.check_physical_eol(token) token_type, text = token[0:2] processor.log_token(LOG, token) @@ -485,7 +534,7 @@ class FileChecker(object): self.run_physical_checks(file_processor.lines[-1]) self.run_logical_checks() - def run_checks(self, results_queue): + def run_checks(self, results_queue, statistics_queue): """Run checks against the file.""" if self.processor.should_ignore_file(): return @@ -501,6 +550,11 @@ class FileChecker(object): if results_queue is not None: results_queue.put((self.filename, self.results)) + logical_lines = self.processor.statistics['logical lines'] + self.statistics['logical lines'] = logical_lines + if statistics_queue is not None: + statistics_queue.put(self.statistics) + def handle_comment(self, token, token_text): """Handle the logic when encountering a comment token.""" # The comment also ends a physical line diff --git a/flake8/defaults.py b/flake8/defaults.py index 76829a5..d9f5a0b 100644 --- a/flake8/defaults.py +++ b/flake8/defaults.py @@ -7,5 +7,11 @@ MAX_LINE_LENGTH = 79 TRUTHY_VALUES = set(['true', '1', 't']) -# Other consants +# Other constants WHITESPACE = frozenset(' \t') + +STATISTIC_NAMES = ( + 'logical lines', + 'physical lines', + 'tokens', +) diff --git a/flake8/formatting/base.py b/flake8/formatting/base.py index 19e21e4..4fda6f4 100644 --- a/flake8/formatting/base.py +++ b/flake8/formatting/base.py @@ -87,7 +87,24 @@ class BaseFormatter(object): def show_benchmarks(self, benchmarks): """Format and print the benchmarks.""" - pass + # NOTE(sigmavirus24): The format strings are a little confusing, even + # to me, so here's a quick explanation: + # We specify the named value first followed by a ':' to indicate we're + # formatting the value. + # Next we use '<' to indicate we want the value left aligned. + # Then '10' is the width of the area. + # For floats, finally, we only want only want at most 3 digits after + # the decimal point to be displayed. This is the precision and it + # can not be specified for integers which is why we need two separate + # format strings. + float_format = '{value:<10.3} {statistic}'.format + int_format = '{value:<10} {statistic}'.format + for statistic, value in benchmarks: + if isinstance(value, int): + benchmark = int_format(statistic=statistic, value=value) + else: + benchmark = float_format(statistic=statistic, value=value) + self._write(benchmark) def show_source(self, error): """Show the physical line generating the error. diff --git a/flake8/main/application.py b/flake8/main/application.py index 6739216..efb1049 100644 --- a/flake8/main/application.py +++ b/flake8/main/application.py @@ -7,6 +7,7 @@ import time import flake8 from flake8 import checker +from flake8 import defaults from flake8 import style_guide from flake8 import utils from flake8.main import options @@ -225,6 +226,22 @@ class Application(object): self.file_checker_manager.run() LOG.info('Finished running') self.file_checker_manager.stop() + self.end_time = time.time() + + def report_benchmarks(self): + if not self.options.benchmark: + return + time_elapsed = self.end_time - self.start_time + statistics = [('seconds elapsed', time_elapsed)] + add_statistic = statistics.append + for statistic in (defaults.STATISTIC_NAMES + ('files',)): + value = self.file_checker_manager.statistics[statistic] + total_description = 'total ' + statistic + ' processed' + add_statistic((total_description, value)) + per_second_description = statistic + ' processed per second' + add_statistic((per_second_description, int(value / time_elapsed))) + + self.formatter.show_benchmarks(statistics) def report_errors(self): # type: () -> NoneType @@ -259,7 +276,7 @@ class Application(object): self.initialize(argv) self.run_checks() self.report_errors() - self.end_time = time.time() + self.report_benchmarks() def run(self, argv=None): # type: (Union[NoneType, List[str]]) -> NoneType diff --git a/flake8/main/options.py b/flake8/main/options.py index 51db34e..c725c38 100644 --- a/flake8/main/options.py +++ b/flake8/main/options.py @@ -192,3 +192,10 @@ def register_default_options(option_manager): '--isolated', default=False, action='store_true', help='Ignore all found configuration files.', ) + + # Benchmarking + + add_option( + '--benchmark', default=False, action='store_true', + help='Print benchmark information about this run of Flake8', + ) diff --git a/flake8/processor.py b/flake8/processor.py index 8e6d897..0c33cc2 100644 --- a/flake8/processor.py +++ b/flake8/processor.py @@ -97,6 +97,10 @@ class FileProcessor(object): self.total_lines = len(self.lines) #: Verbosity level of Flake8 self.verbose = options.verbose + #: Statistics dictionary + self.statistics = { + 'logical lines': 0, + } @contextlib.contextmanager def inside_multiline(self, line_number): @@ -186,6 +190,7 @@ class FileProcessor(object): """Build a logical line from the current tokens list.""" comments, logical, mapping_list = self.build_logical_line_tokens() self.logical_line = ''.join(logical) + self.statistics['logical lines'] += 1 return ''.join(comments), self.logical_line, mapping_list def split_line(self, token): From 16f58005bf09100b4ffb2a9337287ca30667c4d3 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sun, 19 Jun 2016 20:41:19 -0500 Subject: [PATCH 334/364] Add --benchmark option documentation --- docs/source/user/options.rst | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index ecbf52d..7f79e61 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -706,3 +706,24 @@ exclude_in_doctest = dir/subdir/file.py, dir/other/file.py + + +.. option:: --benchmark + + Collect and print benchmarks for this run of Flake8. This aggregates the + total number of: + + - tokens + - physical lines + - logical lines + - files + + and the number of elapsed seconds. + + Command-line usage: + + .. prompt:: bash + + flake8 --benchmark dir/ + + This **can not** be specified in config files. From 57ac6ab6994dc609a3514f49d77dac228328d308 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 08:04:33 -0500 Subject: [PATCH 335/364] Add |Flake8| substitution to the epilog The rst_epilog config value in Sphinx allows you to maintain a global list of replaces or includes that are appended to the text of each document in your documentation. This allows us to use |Flake8| everywhere without redefining the replace in every document. --- docs/source/conf.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/source/conf.py b/docs/source/conf.py index a44bd97..04b2ea8 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -67,6 +67,10 @@ version = flake8.__version__ # The full version, including alpha/beta/rc tags. release = flake8.__version__ +rst_epilog = """ +.. |Flake8| replace:: :program:`Flake8` +""" + # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # From 41277ff965b9083d898ad8c2c990f7c119a8a1ae Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 08:13:50 -0500 Subject: [PATCH 336/364] Use |Flake8| consistently throughout documentation --- docs/source/dev/formatters.rst | 11 ++--- docs/source/dev/index.rst | 4 +- docs/source/dev/plugin_parameters.rst | 22 ++++----- docs/source/dev/registering_plugins.rst | 12 ++--- docs/source/index.rst | 53 ++++++++++++++-------- docs/source/internal/checker.rst | 12 ++--- docs/source/internal/cli.rst | 4 +- docs/source/internal/contributing.rst | 42 ++++++++--------- docs/source/internal/formatters.rst | 6 +-- docs/source/internal/index.rst | 2 +- docs/source/internal/option_handling.rst | 57 ++++++++++++++---------- docs/source/internal/plugin_handling.rst | 20 ++++----- docs/source/internal/utils.rst | 14 +++--- docs/source/user/configuration.rst | 35 ++++++++------- docs/source/user/ignoring-errors.rst | 8 +--- docs/source/user/index.rst | 8 ++-- docs/source/user/invocation.rst | 16 +++---- docs/source/user/options.rst | 31 ++++++------- docs/source/user/python-api.rst | 2 +- 19 files changed, 192 insertions(+), 167 deletions(-) diff --git a/docs/source/dev/formatters.rst b/docs/source/dev/formatters.rst index 5beafda..480ada0 100644 --- a/docs/source/dev/formatters.rst +++ b/docs/source/dev/formatters.rst @@ -4,7 +4,7 @@ Developing a Formatting Plugin for Flake8 =========================================== -Flake8 allowed for custom formatting plugins in version +|Flake8| allowed for custom formatting plugins in version 3.0.0. Let's write a plugin together: .. code-block:: python @@ -17,11 +17,12 @@ Flake8 allowed for custom formatting plugins in version pass -We notice, as soon as we start, that we inherit from Flake8's +We notice, as soon as we start, that we inherit from |Flake8|'s :class:`~flake8.formatting.base.BaseFormatter` class. If we follow the :ref:`instructions to register a plugin ` and try to use -our example formatter, e.g., ``flake8 --format=example`` then Flake8 will fail -because we did not implement the ``format`` method. Let's do that next. +our example formatter, e.g., ``flake8 --format=example`` then +|Flake8| will fail because we did not implement the ``format`` method. +Let's do that next. .. code-block:: python @@ -34,7 +35,7 @@ because we did not implement the ``format`` method. Let's do that next. With that we're done. Obviously this isn't a very useful formatter, but it should highlight the simplicitly of creating a formatter with Flake8. If we wanted to instead create a formatter that aggregated the results and returned -XML, JSON, or subunit we could also do that. Flake8 interacts with the +XML, JSON, or subunit we could also do that. |Flake8| interacts with the formatter in two ways: #. It creates the formatter and provides it the options parsed from the diff --git a/docs/source/dev/index.rst b/docs/source/dev/index.rst index 4969395..9ef8138 100644 --- a/docs/source/dev/index.rst +++ b/docs/source/dev/index.rst @@ -2,8 +2,8 @@ Writing Plugins for Flake8 ============================ -Since Flake8 2.0, the Flake8 tool has allowed for extensions and custom -plugins. In Flake8 3.0, we're expanding that ability to customize and +Since |Flake8| 2.0, the |Flake8| tool has allowed for extensions and custom +plugins. In |Flake8| 3.0, we're expanding that ability to customize and extend **and** we're attempting to thoroughly document it. Some of the documentation in this section will reference third-party documentation to reduce duplication and to point you, the developer, towards diff --git a/docs/source/dev/plugin_parameters.rst b/docs/source/dev/plugin_parameters.rst index 9b074ec..527950c 100644 --- a/docs/source/dev/plugin_parameters.rst +++ b/docs/source/dev/plugin_parameters.rst @@ -4,15 +4,15 @@ Receiving Information For A Check Plugin ========================================== -Plugins to Flake8 have a great deal of information that they can request from -a :class:`~flake8.processor.FileProcessor` instance. Historically, Flake8 has -supported two types of plugins: +Plugins to |Flake8| have a great deal of information that they can request +from a :class:`~flake8.processor.FileProcessor` instance. Historically, +|Flake8| has supported two types of plugins: #. classes that accept parsed abstract syntax trees (ASTs) #. functions that accept a range of arguments -Flake8 now does not distinguish between the two types of plugins. Any plugin +|Flake8| now does not distinguish between the two types of plugins. Any plugin can accept either an AST or a range of arguments. Further, any plugin that has certain callable attributes can also register options and receive parsed options. @@ -21,8 +21,8 @@ options. Indicating Desired Data ======================= -Flake8 inspects the plugin's signature to determine what parameters it expects -using :func:`flake8.utils.parameters_for`. +|Flake8| inspects the plugin's signature to determine what parameters it +expects using :func:`flake8.utils.parameters_for`. :attr:`flake8.plugins.manager.Plugin.parameters` caches the values so that each plugin makes that fairly expensive call once per plugin. When processing a file, a plugin can ask for any of the following: @@ -56,7 +56,7 @@ Any plugin that has callable attributes ``provide_options`` and Your ``register_options`` function should expect to receive an instance of |OptionManager|. An |OptionManager| instance behaves very similarly to -:class:`optparse.OptionParser`. It, however, uses the layer that Flake8 has +:class:`optparse.OptionParser`. It, however, uses the layer that |Flake8| has developed on top of :mod:`optparse` to also handle configuration file parsing. :meth:`~flake8.options.manager.OptionManager.add_option` creates an |Option| which accepts the same parameters as :mod:`optparse` as well as three extra @@ -65,7 +65,7 @@ boolean parameters: - ``parse_from_config`` The command-line option should also be parsed from config files discovered - by Flake8. + by |Flake8|. .. note:: @@ -91,8 +91,8 @@ boolean parameters: allow a comma-separated list of paths. Each of these options works individually or can be combined. Let's look at a -couple examples from Flake8. In each example, we will have ``option_manager`` -which is an instance of |OptionManager|. +couple examples from |Flake8|. In each example, we will have +``option_manager`` which is an instance of |OptionManager|. .. code-block:: python @@ -143,7 +143,7 @@ documentation of :mod:`optparse`. Accessing Parsed Options ======================== -When a plugin has a callable ``provide_options`` attribute, Flake8 will call +When a plugin has a callable ``provide_options`` attribute, |Flake8| will call it and attempt to provide the |OptionManager| instance, the parsed options which will be an instance of :class:`optparse.Values`, and the extra arguments that were not parsed by the |OptionManager|. If that fails, we will just pass diff --git a/docs/source/dev/registering_plugins.rst b/docs/source/dev/registering_plugins.rst index 53b6dc4..5d01f99 100644 --- a/docs/source/dev/registering_plugins.rst +++ b/docs/source/dev/registering_plugins.rst @@ -4,7 +4,7 @@ Registering a Plugin with Flake8 ================================== -To register any kind of plugin with Flake8, you need: +To register any kind of plugin with |Flake8|, you need: #. A way to install the plugin (whether it is packaged on its own or as part of something else). In this section, we will use a ``setup.py`` @@ -15,9 +15,9 @@ To register any kind of plugin with Flake8, you need: #. A somewhat recent version of setuptools (newer than 0.7.0 but preferably as recent as you can attain). -Flake8 relies on functionality provided by setuptools called -`Entry Points`_. These allow any package to register a plugin with Flake8 via -that package's ``setup.py`` file. +|Flake8| relies on functionality provided by setuptools called +`Entry Points`_. These allow any package to register a plugin with |Flake8| +via that package's ``setup.py`` file. Let's presume that we already have our plugin written and it's in a module called ``flake8_example``. We might have a ``setup.py`` that looks something @@ -82,7 +82,7 @@ Note specifically these lines: We tell setuptools to register our entry point "X" inside the specific grouping of entry-points that flake8 should look in. -Flake8 presently looks at three groups: +|Flake8| presently looks at three groups: - ``flake8.extension`` @@ -90,7 +90,7 @@ Flake8 presently looks at three groups: - ``flake8.report`` -If your plugin is one that adds checks to Flake8, you will use +If your plugin is one that adds checks to |Flake8|, you will use ``flake8.extension``. If your plugin automatically fixes errors in code, you will use ``flake8.listen``. Finally, if your plugin performs extra report handling (formatting, filtering, etc.) it will use ``flake8.report``. diff --git a/docs/source/index.rst b/docs/source/index.rst index da383c6..68dc3fa 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -7,18 +7,21 @@ Flake8: Your Tool For Style Guide Enforcement =============================================== +Quickstart +========== + .. _installation-guide: Installation -============ +------------ -To install Flake8, open an interactive shell and run: +To install |Flake8|, open an interactive shell and run: .. code:: python -m pip install flake8 -If you want Flake8 to be installed for your default Python installation, you +If you want |Flake8| to be installed for your default Python installation, you can instead use: .. code:: @@ -27,16 +30,16 @@ can instead use: .. note:: - It is **very** important to install Flake8 on the *correct* version of - Python for your needs. If you want Flake8 to properly parse new language + It is **very** important to install |Flake8| on the *correct* version of + Python for your needs. If you want |Flake8| to properly parse new language features in Python 3.5 (for example), you need it to be installed on 3.5 - for flake8 to understand those features. In many ways, Flake8 is tied to + for |Flake8| to understand those features. In many ways, Flake8 is tied to the version of Python on which it runs. -Quickstart -========== +Using Flake8 +------------ -To start using Flake8, open an interactive shell and run: +To start using |Flake8|, open an interactive shell and run: .. code:: @@ -46,7 +49,7 @@ To start using Flake8, open an interactive shell and run: .. note:: - If you have installed Flake8 on a particular version of Python (or on + If you have installed |Flake8| on a particular version of Python (or on several versions), it may be best to instead run ``python -m flake8``. @@ -55,20 +58,24 @@ If you only want to see the instances of a specific warning or error, you can .. code:: - flake8 --select path/to/code/ + flake8 --select E123,W503 path/to/code/ Alternatively, if you want to *ignore* only one specific warning or error: .. code:: - flake8 --ignore path/to/code/ + flake8 --ignore E24,W504 path/to/code/ Please read our user guide for more information about how to use and configure -Flake8. +|Flake8|. User Guide ========== +All users of |Flake8| should read this portion of the documentation. This +provides examples and documentation around |Flake8|'s assortment of options +and how to specify them on the command-line or in configuration files. + .. toctree:: :maxdepth: 2 @@ -77,13 +84,22 @@ User Guide Plugin Developer Guide ====================== +If you're maintaining a plugin for |Flake8| or creating a new one, you should +read this section of the documentation. It explains how you can write your +plugins and distribute them to others. + .. toctree:: :maxdepth: 2 dev/index -Developer Guide -=============== +Contributor Guide +================= + +If you are reading |Flake8|'s source code for fun or looking to contribute, +you should read this portion of the documentation. This is a mix of documenting +the internal-only interfaces |Flake8| and documenting reasoning for Flake8's +design. .. toctree:: :maxdepth: 2 @@ -98,9 +114,8 @@ Release Notes and History release-notes/index -Indices and tables -================== +General Indices +=============== * :ref:`genindex` -* :ref:`modindex` -* :ref:`search` +* :ref:`Index of Documented Public Modules ` diff --git a/docs/source/internal/checker.rst b/docs/source/internal/checker.rst index e82e62f..19bcf7d 100644 --- a/docs/source/internal/checker.rst +++ b/docs/source/internal/checker.rst @@ -2,13 +2,13 @@ How Checks are Run ==================== -In Flake8 2.x, Flake8 delegated check running to pep8. In 3.0 flake8 takes -on that responsibility. This has allowed for simpler +In |Flake8| 2.x, |Flake8| delegated check running to pep8. In 3.0 |Flake8| +takes on that responsibility. This has allowed for simpler handling of the ``--jobs`` parameter (using :mod:`multiprocessing`) and simplified our fallback if something goes awry with concurency. At the lowest level we have a |FileChecker|. Instances of |FileChecker| are -created for *each* file to be analyzed by Flake8. Each instance, has a copy of -all of the plugins registered with setuptools in the ``flake8.extension`` +created for *each* file to be analyzed by |Flake8|. Each instance, has a copy +of all of the plugins registered with setuptools in the ``flake8.extension`` entry-point group. The |FileChecker| instances are managed by an instance of |Manager|. The @@ -22,8 +22,8 @@ excluded. Processing Files ---------------- -Unfortunately, since Flake8 took over check running from pep8/pycodestyle, it -also had to take over parsing and processing files for the checkers +Unfortunately, since |Flake8| took over check running from pep8/pycodestyle, +it also had to take over parsing and processing files for the checkers to use. Since it couldn't reuse pycodestyle's functionality (since it did not separate cleanly the processing from check running) that function was isolated into the :class:`~flake8.processor.FileProcessor` class. We moved diff --git a/docs/source/internal/cli.rst b/docs/source/internal/cli.rst index bb8f774..f203125 100644 --- a/docs/source/internal/cli.rst +++ b/docs/source/internal/cli.rst @@ -1,7 +1,7 @@ Command Line Interface ====================== -The command line interface of Flake8 is modeled as an application via +The command line interface of |Flake8| is modeled as an application via :class:`~flake8.main.cli.Application`. When a user runs ``flake8`` at their command line, :func:`~flake8.main.cli.main` is run which handles management of the application. @@ -10,7 +10,7 @@ User input is parsed *twice* to accomodate logging and verbosity options passed by the user as early as possible. This is so as much logging can be produced as possible. -The default flake8 options are registered by +The default |Flake8| options are registered by :func:`~flake8.main.options.register_default_options`. Trying to register these options in plugins will result in errors. diff --git a/docs/source/internal/contributing.rst b/docs/source/internal/contributing.rst index 2406e9a..06e6ec3 100644 --- a/docs/source/internal/contributing.rst +++ b/docs/source/internal/contributing.rst @@ -2,7 +2,7 @@ Contributing to Flake8 ======================== -There are many ways to contriubte to Flake8, and we encourage them all: +There are many ways to contriubte to |Flake8|, and we encourage them all: - contributing bug reports and feature requests @@ -18,7 +18,7 @@ reassure you that any help you can provide *is* valuable. Code of Conduct =============== -Flake8 adheres to the `Python Code Quality Authority's Code of Conduct`_. +|Flake8| adheres to the `Python Code Quality Authority's Code of Conduct`_. Any violations of the Code of Conduct should be reported to Ian Cordasco (graffatcolmingov [at] gmail [dot] com). @@ -26,7 +26,7 @@ Any violations of the Code of Conduct should be reported to Ian Cordasco Setting Up A Development Environment ==================================== -To contribute to Flake8's development, you simply need: +To contribute to |Flake8|'s development, you simply need: - Python (one of the versions we support) @@ -50,29 +50,29 @@ To contribute to Flake8's development, you simply need: Filing a Bug ============ -When filing a bug against Flake8, please fill out the issue template as it is -provided to you by `GitLab`_. If your bug is in reference to one of the -checks that Flake8 reports by default, please do not report them to Flake8 -unless Flake8 is doing something to prevent the check from running or you -have some reason to believe Flake8 is inhibiting the effectiveness of the +When filing a bug against |Flake8|, please fill out the issue template as it +is provided to you by `GitLab`_. If your bug is in reference to one of the +checks that |Flake8| reports by default, please do not report them to |Flake8| +unless |Flake8| is doing something to prevent the check from running or you +have some reason to believe |Flake8| is inhibiting the effectiveness of the check. **Please search for closed and open bug reports before opening new ones.** All bug reports about checks should go to their respective projects: -- Check codes starting with ``E`` and ``W`` should be reported to +- Error codes starting with ``E`` and ``W`` should be reported to `pycodestyle`_. -- Check codes starting with ``F`` should be reported to `pyflakes`_ +- Error codes starting with ``F`` should be reported to `pyflakes`_ -- Check codes starting with ``C`` should be reported to `mccabe`_ +- Error codes starting with ``C`` should be reported to `mccabe`_ Requesting a New Feature ======================== -When requesting a new feature in Flake8, please fill out the issue template. +When requesting a new feature in |Flake8|, please fill out the issue template. Please also note if there are any existing alternatives to your new feature either via plugins, or combining command-line options. Please provide example use cases. For example, do not ask for a feature like this: @@ -81,8 +81,8 @@ use cases. For example, do not ask for a feature like this: Instead ask: - I need Flake8 to frobulate these files because my team expects them to - frobulated but Flake8 currently does not frobulate them. We tried using + I need |Flake8| to frobulate these files because my team expects them to + frobulated but |Flake8| currently does not frobulate them. We tried using ``--filename`` but we could not create a pattern that worked. The more you explain about *why* you need a feature, the more likely we are to @@ -92,11 +92,11 @@ understand your needs and help you to the best of our ability. Contributing Documentation ========================== -To contribute to Flake8's documentation, you should first familiarize yourself -with reStructuredText and Sphinx. For the most part, you should be fine -following the structure and style of the rest of Flake8's documentation. +To contribute to |Flake8|'s documentation, you might want to first read a +little about reStructuredText or Sphinx. For the most part, you should be fine +following the structure and style of the rest of |Flake8|'s documentation. -All of Flake8's documentation is written in reStructuredText and rendered by +All of |Flake8|'s documentation is written in reStructuredText and rendered by Sphinx. The source (reStructuredText) lives in ``docs/source/``. To build the documentation the way our Continuous Integration does, run: @@ -129,7 +129,7 @@ documentation generation and refresh the documentation you're working on. Contributing Code ================= -Flake8 development happens on `GitLab`_. Code contributions should be +|Flake8| development happens on `GitLab`_. Code contributions should be submitted there. Merge requests should: @@ -170,12 +170,12 @@ When reviewing other people's merge requests and issues, please be else. We strive for professional code reviews that do not insult the contributor's intelligence or impugn their character. The code review should be focused on the code, it's effectiveness, and whether it is -appropriate for Flake8. +appropriate for |Flake8|. If you have the ability to edit an issue or merge request's labels, please do so to make search and prioritization easier. -Flake8 uses milestones with both issues and merge requests. This provides +|Flake8| uses milestones with both issues and merge requests. This provides direction for other contributors about when an issue or merge request will be delivered. diff --git a/docs/source/internal/formatters.rst b/docs/source/internal/formatters.rst index b371307..c58189b 100644 --- a/docs/source/internal/formatters.rst +++ b/docs/source/internal/formatters.rst @@ -2,12 +2,12 @@ Built-in Formatters ===================== -By default Flake8 has two formatters built-in, ``default`` and ``pylint``. +By default |Flake8| has two formatters built-in, ``default`` and ``pylint``. These correspond to two classes |DefaultFormatter| and |PylintFormatter|. -In Flake8 2.0, pep8 handled formatting of errors and also allowed users to +In |Flake8| 2.0, pep8 handled formatting of errors and also allowed users to specify an arbitrary format string as a parameter to ``--format``. In order -to allow for this backwards compatibility, Flake8 3.0 made two choices: +to allow for this backwards compatibility, |Flake8| 3.0 made two choices: #. To not limit a user's choices for ``--format`` to the format class names diff --git a/docs/source/internal/index.rst b/docs/source/internal/index.rst index 81efa2c..d4d05ac 100644 --- a/docs/source/internal/index.rst +++ b/docs/source/internal/index.rst @@ -2,7 +2,7 @@ Exploring Flake8's Internals ============================== -While writing Flake8 3.0, the developers attempted to capture some reasoning +While writing |Flake8| 3.0, the developers attempted to capture some reasoning and decision information in internal documentation meant for future developers and maintaners. Most of this information is unnecessary for users and plugin developers. Some of it, however, is linked to from the plugin development diff --git a/docs/source/internal/option_handling.rst b/docs/source/internal/option_handling.rst index 1a5b31f..74ecb76 100644 --- a/docs/source/internal/option_handling.rst +++ b/docs/source/internal/option_handling.rst @@ -4,12 +4,12 @@ Option and Configuration Handling Option Management ----------------- -Command-line options are often also set in configuration files for Flake8. +Command-line options are often also set in configuration files for |Flake8|. While not all options are meant to be parsed from configuration files, many default options are also parsed from configuration files as well as most plugin options. -In Flake8 2, plugins received a :class:`optparse.OptionParser` instance and +In |Flake8| 2, plugins received a :class:`optparse.OptionParser` instance and called :meth:`optparse.OptionParser.add_option` to register options. If the plugin author also wanted to have that option parsed from config files they also had to do something like: @@ -22,13 +22,13 @@ also had to do something like: This was previously undocumented and led to a lot of confusion about why registered options were not automatically parsed from configuration files. -Since Flake8 3 was rewritten from scratch, we decided to take a different +Since |Flake8| 3 was rewritten from scratch, we decided to take a different approach to configuration file parsing. Instead of needing to know about an -undocumented attribute that pep8 looks for, Flake8 3 now accepts a parameter +undocumented attribute that pep8 looks for, |Flake8| 3 now accepts a parameter to ``add_option``, specifically ``parse_from_config`` which is a boolean value. -Flake8 does this by creating its own abstractions on top of :mod:`optparse`. +|Flake8| does this by creating its own abstractions on top of :mod:`optparse`. The first abstraction is the :class:`flake8.options.manager.Option` class. The second is the :class:`flake8.options.manager.OptionManager`. In fact, we add three new parameters: @@ -49,14 +49,22 @@ example, let's consider a user's list of ignored error codes for a project: [flake8] ignore = - E111, # Reasoning - E711, # Reasoning - E712, # Reasoning - E121, # Reasoning - E122, # Reasoning - E123, # Reasoning - E131, # Reasoning - E251 # Reasoning + # Reasoning + E111, + # Reasoning + E711, + # Reasoning + E712, + # Reasoning + E121, + # Reasoning + E122, + # Reasoning + E123, + # Reasoning + E131, + # Reasoning + E251 It makes sense here to allow users to specify the value this way, but, the standard libary's :class:`configparser.RawConfigParser` class does returns a @@ -67,8 +75,8 @@ string that looks like "\nE111, \nE711, \nE712, \nE121, \nE122, \nE123, \nE131, \nE251 " This means that a typical call to :meth:`str.split` with ``','`` will not be -sufficient here. Telling Flake8 that something is a comma-separated list -(e.g., ``comma_separated_list=True``) will handle this for you. Flake8 will +sufficient here. Telling |Flake8| that something is a comma-separated list +(e.g., ``comma_separated_list=True``) will handle this for you. |Flake8| will return: .. code-block:: python @@ -117,10 +125,11 @@ extra arguments we highlighted above. Configuration File Management ----------------------------- -In Flake8 2, configuration file discovery and management was handled by pep8. -In pep8's 1.6 release series, it drastically broke how discovery and merging -worked (as a result of trying to improve it). To avoid a dependency breaking -Flake8 again in the future, we have created our own discovery and management. +In |Flake8| 2, configuration file discovery and management was handled by +pep8. In pep8's 1.6 release series, it drastically broke how discovery and +merging worked (as a result of trying to improve it). To avoid a dependency +breaking |Flake8| again in the future, we have created our own discovery and +management. As part of managing this ourselves, we decided to change management/discovery for 3.0.0. We have done the following: @@ -137,7 +146,7 @@ for 3.0.0. We have done the following: take precedence over user and project files. - **New in 3.0.0** The user can specify ``--config `` to so this - file is the only configuration file used. This is a change from Flake8 2 + file is the only configuration file used. This is a change from |Flake8| 2 where pep8 would simply merge this configuration file into the configuration generated by user and project files (where this takes precedence). @@ -148,10 +157,10 @@ To facilitate the configuration file management, we've taken a different approach to discovery and management of files than pep8. In pep8 1.5, 1.6, and 1.7 configuration discovery and management was centralized in `66 lines of very terse python`_ which was confusing and not very explicit. The terseness -of this function (Flake8's authors believe) caused the confusion and problems -with pep8's 1.6 series. As such, Flake8 has separated out discovery, -management, and merging into a module to make reasoning about each of these -pieces easier and more explicit (as well as easier to test). +of this function (|Flake8|'s authors believe) caused the confusion and +problems with pep8's 1.6 series. As such, |Flake8| has separated out +discovery, management, and merging into a module to make reasoning about each +of these pieces easier and more explicit (as well as easier to test). Configuration file discovery is managed by the :class:`~flake8.options.config.ConfigFileFinder` object. This object needs to diff --git a/docs/source/internal/plugin_handling.rst b/docs/source/internal/plugin_handling.rst index 65b2fa1..9af3182 100644 --- a/docs/source/internal/plugin_handling.rst +++ b/docs/source/internal/plugin_handling.rst @@ -4,8 +4,8 @@ Plugin Handling Plugin Management ----------------- -Flake8 3.0 added support for two other plugins besides those which define new -checks. It now supports: +|Flake8| 3.0 added support for two other plugins besides those which define +new checks. It now supports: - extra checks @@ -13,7 +13,7 @@ checks. It now supports: - listeners to auto-correct violations of checks -To facilitate this, Flake8 needed a more mature way of managing plugins. +To facilitate this, |Flake8| needed a more mature way of managing plugins. Thus, we developed the |PluginManager| which accepts a namespace and will load the plugins for that namespace. A |PluginManager| creates and manages many |Plugin| instances. @@ -72,21 +72,21 @@ existing packages on PyPI allowed for storing data on each node of the trie, it was left up to write our own as :class:`~flake8.plugins._trie.Trie`. On top of that we layer our :class:`~flake8.plugins.notifier.Notifier` class. -Now when Flake8 receives an error or warning, we can easily call the +Now when |Flake8| receives an error or warning, we can easily call the :meth:`~flake8.plugins.notifier.Notifier.notify` method and let plugins act on that knowledge. Default Plugins --------------- -Finally, Flake8 has always provided its own plugin shim for Pyflakes. As part -of that we carry our own shim in-tree and now store that in +Finally, |Flake8| has always provided its own plugin shim for Pyflakes. As +part of that we carry our own shim in-tree and now store that in :mod:`flake8.plugins.pyflakes`. -Flake8 also registers plugins for pep8. Each check in pep8 requires different -parameters and it cannot easily be shimmed together like Pyflakes was. As -such, plugins have a concept of a "group". If you look at our :file:`setup.py` -you will see that we register pep8 checks roughly like so: +|Flake8| also registers plugins for pep8. Each check in pep8 requires +different parameters and it cannot easily be shimmed together like Pyflakes +was. As such, plugins have a concept of a "group". If you look at our +:file:`setup.py` you will see that we register pep8 checks roughly like so: .. code:: diff --git a/docs/source/internal/utils.rst b/docs/source/internal/utils.rst index 573938c..1b2bb1c 100644 --- a/docs/source/internal/utils.rst +++ b/docs/source/internal/utils.rst @@ -2,7 +2,7 @@ Utility Functions =================== -Flake8 has a few utility functions that it uses internally. +|Flake8| has a few utility functions that it uses internally. .. warning:: @@ -39,8 +39,8 @@ And converts it to a list that looks as follows ["E121", "W123", "F904"] -This function helps normalize any kind of comma-separated input you or Flake8 -might receive. This is most helpful when taking advantage of Flake8's +This function helps normalize any kind of comma-separated input you or |Flake8| +might receive. This is most helpful when taking advantage of |Flake8|'s additional parameters to :class:`~flake8.options.manager.Option`. .. autofunction:: flake8.utils.normalize_path @@ -74,11 +74,11 @@ Python we're using can actually use multiprocessing on Windows. Another helpful function that is named only to be explicit given it is a very trivial check, this checks if the user specified ``-`` in their arguments to -Flake8 to indicate we should read from stdin. +|Flake8| to indicate we should read from stdin. .. autofunction:: flake8.utils.filenames_from -When provided an argument to Flake8, we need to be able to traverse +When provided an argument to |Flake8|, we need to be able to traverse directories in a convenient manner. For example, if someone runs .. code:: @@ -103,7 +103,7 @@ function abstracts that logic away with a little extra logic. .. autofunction:: flake8.utils.parameters_for -Flake8 analyzes the parameters to plugins to determine what input they are +|Flake8| analyzes the parameters to plugins to determine what input they are expecting. Plugins may expect one of the following: - ``physical_line`` to receive the line as it appears in the file @@ -120,7 +120,7 @@ argument so we can check the parameters of the plugin consistently. .. autofunction:: flake8.utils.parse_unified_diff -To handle usage of :option:`flake8 --diff`, Flake8 needs to be able +To handle usage of :option:`flake8 --diff`, |Flake8| needs to be able to parse the name of the files in the diff as well as the ranges indicated the sections that have been changed. This function either accepts the diff as an argument or reads the diff from standard-in. It then returns a dictionary with diff --git a/docs/source/user/configuration.rst b/docs/source/user/configuration.rst index 6085bcd..439aeae 100644 --- a/docs/source/user/configuration.rst +++ b/docs/source/user/configuration.rst @@ -4,7 +4,7 @@ Configuring Flake8 ==================== -Once you have learned how to :ref:`invoke ` Flake8, you will soon +Once you have learned how to :ref:`invoke ` |Flake8|, you will soon want to learn how to configure it so you do not have to specify the same options every time you use it. @@ -24,7 +24,7 @@ Remember that you want to specify certain options without writing Configuration Locations ======================= -Flake8 supports storing its configuration in the following places: +|Flake8| supports storing its configuration in the following places: - Your top-level user directory @@ -34,7 +34,7 @@ Flake8 supports storing its configuration in the following places: "User" Configuration -------------------- -Flake8 allows a user to use "global" configuration file to store preferences. +|Flake8| allows a user to use "global" configuration file to store preferences. The user configuration file is expected to be stored somewhere in the user's "home" directory. @@ -44,7 +44,7 @@ The user configuration file is expected to be stored somewhere in the user's - On Linux and other Unix like systems (including OS X) we will look in ``~/``. -Note that Flake8 looks for ``~\.flake8`` on Windows and ``~/.config/flake8`` +Note that |Flake8| looks for ``~\.flake8`` on Windows and ``~/.config/flake8`` on Linux and other Unix systems. User configuration files use the same syntax as Project Configuration files. @@ -54,8 +54,8 @@ Keep reading to see that syntax. Project Configuration --------------------- -Flake8 is written with the understanding that people organize projects into -sub-directories. Let's take for example Flake8's own project structure +|Flake8| is written with the understanding that people organize projects into +sub-directories. Let's take for example |Flake8|'s own project structure .. code:: @@ -81,10 +81,10 @@ sub-directories. Let's take for example Flake8's own project structure In the top-level ``flake8`` directory (which contains ``docs``, ``flake8``, and ``tests``) there's also ``tox.ini`` and ``setup.cfg`` files. In our case, -we keep our Flake8 configuration in ``tox.ini``. Regardless of whether you +we keep our |Flake8| configuration in ``tox.ini``. Regardless of whether you keep your config in ``.flake8``, ``setup.cfg``, or ``tox.ini`` we expect you -to use INI to configure Flake8 (since each of these files already uses INI -as a format). This means that any Flake8 configuration you wish to set needs +to use INI to configure |Flake8| (since each of these files already uses INI +as a format). This means that any |Flake8| configuration you wish to set needs to be in the ``flake8`` section, which means it needs to start like so: .. code-block:: ini @@ -100,11 +100,11 @@ be named in either of two ways: .. note:: - Not every Flake8 command-line option can be specified in the configuration - file. See :ref:`our list of options ` to determine which - options will be parsed from the configuration files. + Not every |Flake8| command-line option can be specified in the + configuration file. See :ref:`our list of options ` to + determine which options will be parsed from the configuration files. -Let's actually look at Flake8's own configuration section: +Let's actually look at |Flake8|'s own configuration section: .. code-block:: ini @@ -163,14 +163,14 @@ This would allow us to add comments for why we're excluding items, e.g., :mod:`configparser` backport from PyPI. That backport enables us to support this behaviour on all supported versions of Python. - Please do **not** open issues about this dependency to Flake8. + Please do **not** open issues about this dependency to |Flake8|. .. note:: You can also specify ``--max-complexity`` as ``max_complexity = 10``. This is also useful if you have a long list of error codes to ignore. Let's -look at a portion of OpenStack's Swift project configuration: +look at a portion of OpenStack's Swift `project configuration`_: .. code-block:: ini @@ -220,5 +220,8 @@ They use the comments to describe the check but they could also write this as: H501 Or they could use each comment to describe **why** they've ignored the check. -:program:`Flake8` knows how to parse these lists and will appropriatey handle +|Flake8| knows how to parse these lists and will appropriatey handle these situations. + +.. _project configuration: + https://github.com/openstack/swift/blob/3944d820387f08372c1a29444f4af7d8e6090ae9/tox.ini#L66..L81 diff --git a/docs/source/user/ignoring-errors.rst b/docs/source/user/ignoring-errors.rst index b9af8a8..ff37c4a 100644 --- a/docs/source/user/ignoring-errors.rst +++ b/docs/source/user/ignoring-errors.rst @@ -55,7 +55,7 @@ then those will be reported. .. note:: - If we ever want to disable Flake8 respecting ``# noqa`` comments, we can + If we ever want to disable |Flake8| respecting ``# noqa`` comments, we can can refer to :option:`flake8 --disable-noqa`. If we instead had more than one error that we wished to ignore, we could @@ -84,11 +84,7 @@ file: The former is the **recommended** way of ignoring entire files. By using our exclude list, we can include it in our configuration file and have one central -place to find what files aren't included in Flake8 checks. The latter has the +place to find what files aren't included in |Flake8| checks. The latter has the benefit that when we run |Flake8| with :option:`flake8 --disable-noqa` all of the errors in that file will show up without having to modify our configuration. Both exist so we can choose which is better for us. - - -.. replacements -.. |Flake8| replace:: :program:`Flake8` diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 4644efc..255ec27 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -2,7 +2,7 @@ Using Flake8 ============== -Flake8 can be used in many ways. A few: +|Flake8| can be used in many ways. A few: - invoked on the command-line @@ -10,12 +10,12 @@ Flake8 can be used in many ways. A few: - called by Git or Mercurial on or around committing -This guide will cover all of these and the nuances for using Flake8. +This guide will cover all of these and the nuances for using |Flake8|. .. note:: - This portion of Flake8's documentation does not cover installation. See - the :ref:`installation-guide` section for how to install Flake8. + This portion of |Flake8|'s documentation does not cover installation. See + the :ref:`installation-guide` section for how to install |Flake8|. .. toctree:: :maxdepth: 2 diff --git a/docs/source/user/invocation.rst b/docs/source/user/invocation.rst index ef763ae..5c92ca3 100644 --- a/docs/source/user/invocation.rst +++ b/docs/source/user/invocation.rst @@ -4,16 +4,16 @@ Invoking Flake8 ================= -Once you have :ref:`installed ` Flake8, you can begin -using it. Most of the time, you will be able to generically invoke Flake8 +Once you have :ref:`installed ` |Flake8|, you can begin +using it. Most of the time, you will be able to generically invoke |Flake8| like so: .. prompt:: bash flake8 ... -Where you simply allow the shell running in your terminal to locate Flake8. -In some cases, though, you may have installed Flake8 for multiple versions +Where you simply allow the shell running in your terminal to locate |Flake8|. +In some cases, though, you may have installed |Flake8| for multiple versions of Python (e.g., Python 2.7 and Python 3.5) and you need to call a specific version. In that case, you will have much better results using: @@ -27,15 +27,15 @@ Or python3.5 -m flake8 -Since that will tell the correct version of Python to run Flake8. +Since that will tell the correct version of Python to run |Flake8|. .. note:: - Installing Flake8 once will not install it on both Python 2.7 and + Installing |Flake8| once will not install it on both Python 2.7 and Python 3.5. It will only install it for the version of Python that is running pip. -It is also possible to specify command-line options directly to Flake8: +It is also possible to specify command-line options directly to |Flake8|: .. prompt:: bash @@ -53,7 +53,7 @@ Or From now on, we'll simply use ``flake8`` and assume that the user knows they can instead use ``python -m flake8`` instead. -It's also possible to narrow what Flake8 will try to check by specifying +It's also possible to narrow what |Flake8| will try to check by specifying exactly the paths and directories you want it to check. Let's assume that we have a directory with python files and subdirectories which have python files (and may have more sub-directories) called ``my_project``. Then if diff --git a/docs/source/user/options.rst b/docs/source/user/options.rst index 7f79e61..acaa67c 100644 --- a/docs/source/user/options.rst +++ b/docs/source/user/options.rst @@ -29,7 +29,7 @@ .. option:: --version - Show :program:`Flake8`\ 's version as well as the versions of all plugins + Show |Flake8|'s version as well as the versions of all plugins installed. Command-line usage: @@ -43,7 +43,7 @@ .. option:: -h, --help - Show a description of how to use :program:`Flake8` and its options. + Show a description of how to use |Flake8| and its options. Command-line usage: @@ -57,7 +57,7 @@ .. option:: -v, --verbose - Increase the verbosity of Flake8's output. Each time you specify + Increase the verbosity of |Flake8|'s output. Each time you specify it, it will print more and more information. Command-line example: @@ -77,7 +77,7 @@ .. option:: -q, --quiet - Decrease the verbosity of Flake8's output. Each time you specify it, + Decrease the verbosity of |Flake8|'s output. Each time you specify it, it will print less and less information. Command-line example: @@ -286,7 +286,7 @@ Specify a list of codes to ignore. The list is expected to be comma-separated, and does not need to specify an error code exactly. - Since Flake8 3.0, this **can** be combined with :option:`--select`. See + Since |Flake8| 3.0, this **can** be combined with :option:`--select`. See :option:`--select` for more information. For example, if you wish to only ignore ``W234``, then you can specify @@ -349,7 +349,7 @@ .. option:: --select= - Specify the list of error codes you wish Flake8 to report. Similarly to + Specify the list of error codes you wish |Flake8| to report. Similarly to :option:`--ignore`. You can specify a portion of an error code to get all that start with that string. For example, you can use ``E``, ``E4``, ``E43``, and ``E431``. @@ -370,7 +370,7 @@ flake8 --select=E --ignore=E432 dir/ This will report all codes that start with ``E``, but ignore ``E432`` - specifically. This is more flexibly than the Flake8 2.x and 1.x used + specifically. This is more flexibly than the |Flake8| 2.x and 1.x used to be. This **can** be specified in config files. @@ -453,7 +453,7 @@ Enable off-by-default extensions. - Plugins to Flake8 have the option of registering themselves as + Plugins to |Flake8| have the option of registering themselves as off-by-default. These plugins effectively add themselves to the default ignore list. @@ -479,9 +479,9 @@ .. option:: --exit-zero - Force Flake8 to use the exit status code 0 even if there are errors. + Force |Flake8| to use the exit status code 0 even if there are errors. - By default Flake8 will exit with a non-zero integer if there are errors. + By default |Flake8| will exit with a non-zero integer if there are errors. Command-line example: @@ -514,7 +514,7 @@ .. option:: --jobs= - Specify the number of subprocesses that Flake8 will use to run checks in + Specify the number of subprocesses that |Flake8| will use to run checks in parallel. .. note:: @@ -566,7 +566,7 @@ .. option:: --append-config= Provide extra config files to parse in after and in addition to the files - that Flake8 found on its own. Since these files are the last ones read + that |Flake8| found on its own. Since these files are the last ones read into the Configuration Parser, so it has the highest precedence if it provides an option specified in another config file. @@ -582,7 +582,8 @@ .. option:: --config= Provide a path to a config file that will be the only config file read and - used. This will cause Flake8 to ignore all other config files that exist. + used. This will cause |Flake8| to ignore all other config files that + exist. Command-line example: @@ -595,7 +596,7 @@ .. option:: --isolated - Ignore any config files and use Flake8 as if there were no config files + Ignore any config files and use |Flake8| as if there were no config files found. Command-line example: @@ -710,7 +711,7 @@ .. option:: --benchmark - Collect and print benchmarks for this run of Flake8. This aggregates the + Collect and print benchmarks for this run of |Flake8|. This aggregates the total number of: - tokens diff --git a/docs/source/user/python-api.rst b/docs/source/user/python-api.rst index 2e2a16e..214565d 100644 --- a/docs/source/user/python-api.rst +++ b/docs/source/user/python-api.rst @@ -2,7 +2,7 @@ Public Python API =================== -Flake8 3.0.0 presently does not have a public, stable Python API. +|Flake8| 3.0.0 presently does not have a public, stable Python API. When it does it will be located in :mod:`flake8.api` and that will be documented here. From 3e8bbc50dc76e42a25909e502ef378188dc67345 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 09:26:01 -0500 Subject: [PATCH 337/364] Add documentation ... for writing documentation --- docs/source/internal/contributing.rst | 6 +- docs/source/internal/index.rst | 1 + .../source/internal/writing-documentation.rst | 183 ++++++++++++++++++ 3 files changed, 188 insertions(+), 2 deletions(-) create mode 100644 docs/source/internal/writing-documentation.rst diff --git a/docs/source/internal/contributing.rst b/docs/source/internal/contributing.rst index 06e6ec3..29afc6a 100644 --- a/docs/source/internal/contributing.rst +++ b/docs/source/internal/contributing.rst @@ -93,8 +93,10 @@ Contributing Documentation ========================== To contribute to |Flake8|'s documentation, you might want to first read a -little about reStructuredText or Sphinx. For the most part, you should be fine -following the structure and style of the rest of |Flake8|'s documentation. +little about reStructuredText or Sphinx. |Flake8| has a :ref:`guide of best +practices ` when contributing to our documentation. For the most +part, you should be fine following the structure and style of the rest of +|Flake8|'s documentation. All of |Flake8|'s documentation is written in reStructuredText and rendered by Sphinx. The source (reStructuredText) lives in ``docs/source/``. To build diff --git a/docs/source/internal/index.rst b/docs/source/internal/index.rst index d4d05ac..796a3ba 100644 --- a/docs/source/internal/index.rst +++ b/docs/source/internal/index.rst @@ -16,6 +16,7 @@ pull gently. :maxdepth: 2 contributing + writing-documentation checker cli formatters diff --git a/docs/source/internal/writing-documentation.rst b/docs/source/internal/writing-documentation.rst new file mode 100644 index 0000000..920b411 --- /dev/null +++ b/docs/source/internal/writing-documentation.rst @@ -0,0 +1,183 @@ +.. _docs-style: + +================================== + Writing Documentation for Flake8 +================================== + +The maintainers of |Flake8| believe strongly in benefit of style guides. +Hence, for all contributors who wish to work on our documentation, we've +put together a loose set of guidelines and best practices when adding to +our documentation. + + +View the docs locally before submitting +======================================= + +You can and should generate the docs locally before you submit a pull request +with your changes. You can build the docs by running: + +.. prompt:: bash + + tox -e docs + +From the directory containing the ``tox.ini`` file (which also contains the +``docs/`` directory that this file lives in). + +.. note:: + + If the docs don't build locally, they will not build in our continuous + integration system. We will generally not merge any pull request that + fails continuous integration. + + +Run the docs linter tests before submitting +=========================================== + +You should run the ``doc8`` linter job before you're ready to commit and fix +any errors found. + + +Capitalize Flake8 in prose +========================== + +We believe that by capitalizing |Flake8| in prose, we can help reduce +confusion between the command-line usage of ``flake8`` and the project. + +We also have defined a global replacement ``|Flake8|`` that should be used +and will replace each instance with ``:program:`Flake8```. + + +Use the prompt directive for command-line examples +================================================== + +When documenting something on the command-line, use the ``.. prompt::`` +directive to make it easier for users to copy and paste into their terminal. + +Example: + +.. code-block:: restructuredtext + + .. prompt:: bash + + flake8 --select E123,W503 dir/ + flake8 --ignore E24,W504 dir + + +Wrap lines around 79 characters +=============================== + +We use a maximum line-length in our documentation that is similar to the +default in |Flake8|. Please wrap lines at 79 characters (or less). + + +Use two new-lines before new sections +===================================== + +After the final paragraph of a section and before the next section title, +use two new-lines to separate them. This makes reading the plain-text +document a little nicer. Sphinx ignores these when rendering so they have +no semantic meaning. + +Example: + +.. code-block:: restructuredtext + + Section Header + ============== + + Paragraph. + + + Next Section Header + =================== + + Paragraph. + + +Surround document titles with equal symbols +=========================================== + +To indicate the title of a document, we place an equal number of ``=`` symbols +on the lines before and after the title. For example: + +.. code-block:: restructuredtext + + ================================== + Writing Documentation for Flake8 + ================================== + +Note also that we "center" the title by adding a leading space and having +extra ``=`` symbols at the end of those lines. + + +Use the option template for new options +======================================= + +All of |Flake8|'s command-line options are documented in the User Guide. Each +option is documented individually using the ``.. option::`` directive provided +by Sphinx. At the top of the document, in a reStructuredText comment, is a +template that should be copied and pasted into place when documening new +options. + +.. note:: + + The ordering of the options page is the order that options are printed + in the output of: + + .. prompt:: bash + + flake8 --help + + Please insert your option documentation according to that order. + + +Use anchors for easy reference linking +====================================== + +Use link anchors to allow for other areas of the documentation to use the +``:ref:`` role for intralinking documentation. Example: + +.. code-block:: restructuredtext + + .. _use-anchors: + + Use anchors for easy reference linking + ====================================== + +.. code-block:: restructuredtext + + Somewhere in this paragraph we will :ref:`reference anchors + `. + +.. note:: + + You do not need to provide custom text for the ``:ref:`` if the title of + the section has a title that is sufficient. + + +Keep your audience in mind +========================== + +|Flake8|'s documentation has three distinct (but not separate) audiences: + +#. Users + +#. Plugin Developers + +#. Flake8 Developers and Contributors + +At the moment, you're one of the third group (because you're contributing +or thinking of contributing). + +Consider that most Users aren't very interested in the internal working of +|Flake8|. When writing for Users, focus on how to do something or the +behaviour of a certain piece of configuration or invocation. + +Plugin developers will only care about the internals of |Flake8| as much as +they will have to interact with that. Keep discussions of internal to the +mininmum required. + +Finally, Flake8 Developers and Contributors need to know how everything fits +together. We don't need detail about every line of code, but cogent +explanations and design specifications will help future developers understand +the Hows and Whys of |Flake8|'s internal design. From 3cbad557ea131870235ba518d60341550e7cfcf9 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 13:29:53 -0500 Subject: [PATCH 338/364] Remove mypy from list of tox environments --- .travis.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index c9a85f4..20c7c29 100644 --- a/.travis.yml +++ b/.travis.yml @@ -29,8 +29,6 @@ matrix: env: TOXENV=pylint - python: 3.4 env: TOXENV=doc8 - - python: 3.4 - env: TOXENV=mypy - python: 3.4 env: TOXENV=bandit - python: 3.4 From 590e15cb4cefb4bb6355069e846920eadc2618c8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 13:42:48 -0500 Subject: [PATCH 339/364] Trim trailing whitespace in docs --- docs/source/internal/writing-documentation.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/internal/writing-documentation.rst b/docs/source/internal/writing-documentation.rst index 920b411..ee37517 100644 --- a/docs/source/internal/writing-documentation.rst +++ b/docs/source/internal/writing-documentation.rst @@ -44,7 +44,7 @@ We believe that by capitalizing |Flake8| in prose, we can help reduce confusion between the command-line usage of ``flake8`` and the project. We also have defined a global replacement ``|Flake8|`` that should be used -and will replace each instance with ``:program:`Flake8```. +and will replace each instance with ``:program:`Flake8```. Use the prompt directive for command-line examples @@ -178,6 +178,6 @@ they will have to interact with that. Keep discussions of internal to the mininmum required. Finally, Flake8 Developers and Contributors need to know how everything fits -together. We don't need detail about every line of code, but cogent +together. We don't need detail about every line of code, but cogent explanations and design specifications will help future developers understand the Hows and Whys of |Flake8|'s internal design. From ba5bf0c57a13df92f444a8b29ec5114ea8e35d78 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 13:49:17 -0500 Subject: [PATCH 340/364] Add missing docstring to Application#report_benchmarks --- flake8/main/application.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/flake8/main/application.py b/flake8/main/application.py index efb1049..225c701 100644 --- a/flake8/main/application.py +++ b/flake8/main/application.py @@ -229,8 +229,10 @@ class Application(object): self.end_time = time.time() def report_benchmarks(self): + """Aggregate, calculate, and report benchmarks for this run.""" if not self.options.benchmark: return + time_elapsed = self.end_time - self.start_time statistics = [('seconds elapsed', time_elapsed)] add_statistic = statistics.append From 6ac4d1652277022267f4e7f97db9d1388ba0105f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 20:20:48 -0500 Subject: [PATCH 341/364] Switch to pycodestyle and start constraining versions again --- setup.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/setup.py b/setup.py index b31fb19..8b9617d 100644 --- a/setup.py +++ b/setup.py @@ -30,9 +30,9 @@ if mock is None: requires = [ - "pyflakes >= 0.8.1, != 1.2.0, != 1.2.1, != 1.2.2", - "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", - "mccabe >= 0.5.0", + "pyflakes >= 0.8.1, != 1.2.0, != 1.2.1, != 1.2.2, < 1.3.0", + "pycodestyle >= 2.0.0, < 2.1.0", + "mccabe >= 0.5.0, < 0.6.0", ] if sys.version_info < (3, 4): @@ -50,7 +50,7 @@ def get_long_description(): descr.append(f.read()) return '\n\n'.join(descr) -PEP8 = 'pep8' +PEP8 = 'pycodestyle' _FORMAT = '{0}.{1} = {0}:{1}' PEP8_PLUGIN = functools.partial(_FORMAT.format, PEP8) @@ -93,6 +93,7 @@ setuptools.setup( PEP8_PLUGIN('blank_lines'), PEP8_PLUGIN('extraneous_whitespace'), PEP8_PLUGIN('whitespace_around_keywords'), + PEP8_PLUGIN('missing_whitespace_after_import_keyword'), PEP8_PLUGIN('missing_whitespace'), PEP8_PLUGIN('indentation'), PEP8_PLUGIN('continued_indentation'), @@ -110,6 +111,7 @@ setuptools.setup( PEP8_PLUGIN('comparison_to_singleton'), PEP8_PLUGIN('comparison_negative'), PEP8_PLUGIN('comparison_type'), + PEP8_PLUGIN('ambiguous_identifier'), PEP8_PLUGIN('python_3000_has_key'), PEP8_PLUGIN('python_3000_raise_comma'), PEP8_PLUGIN('python_3000_not_equal'), From b67d1d633e13595e85e78b29af63605a80f17a63 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 20:53:47 -0500 Subject: [PATCH 342/364] Add documentation about release versioning and process --- docs/source/internal/index.rst | 1 + docs/source/internal/releases.rst | 43 +++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) create mode 100644 docs/source/internal/releases.rst diff --git a/docs/source/internal/index.rst b/docs/source/internal/index.rst index 796a3ba..482b898 100644 --- a/docs/source/internal/index.rst +++ b/docs/source/internal/index.rst @@ -17,6 +17,7 @@ pull gently. contributing writing-documentation + releases checker cli formatters diff --git a/docs/source/internal/releases.rst b/docs/source/internal/releases.rst new file mode 100644 index 0000000..2af8950 --- /dev/null +++ b/docs/source/internal/releases.rst @@ -0,0 +1,43 @@ +================== + Releasing Flake8 +================== + +There is not much that is hard to find about how |Flake8| is released. + +- We use **major** releases (e.g., 2.0.0, 3.0.0, etc.) for big releases (e.g., + large scale refactors). This can also contain dependency version changes. + +- We use **minor** releases (e.g., 2.1.0, 2.2.0, 3.1.0, 3.2.0, etc.) for + releases that contain features and dependency version changes. + +- We use **patch** releases (e.g., 2.1.1, 2.1.2, 3.0.1, 3.0.10, etc.) for + releases that contain *only* bug fixes. These *never* contain changes to + dependency version constraints. + + +Process +======= + +To prepare a release, we create a file in :file:`docs/source/releases/` named: +``{{ release_number }}.rst`` (e.g., ``3.0.0.rst``). We note bug fixes, +improvements, and dependency version changes as well as other items of note +for users. + +Before releasing, the following tox test environments must pass: + +- Python 2.7 (a.k.a., ``tox -e py27``) + +- Python 3.4 (a.k.a., ``tox -e py34``) + +- Python 3.5 (a.k.a., ``tox -e py35``) + +- PyPy (a.k.a., ``tox -e pypy``) + +- Linters (a.k.a., ``tox -e linters``) + +We tag the most recent commit that passes those items and contains our release +notes. + +Finally, we run ``tox -e release`` to build source distributions (e.g., +``flake8-3.0.0.tar.gz``), universal wheels, and upload them to PyPI with +Twine. From a4c1edb03efadb9cb8ef72bf6e16853d831429f6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Mon, 20 Jun 2016 20:54:31 -0500 Subject: [PATCH 343/364] Add FAQ --- docs/source/faq.rst | 44 +++++++++++++++++++++++++++++++++++++++++++ docs/source/index.rst | 8 ++++++++ 2 files changed, 52 insertions(+) create mode 100644 docs/source/faq.rst diff --git a/docs/source/faq.rst b/docs/source/faq.rst new file mode 100644 index 0000000..877dae1 --- /dev/null +++ b/docs/source/faq.rst @@ -0,0 +1,44 @@ +============================ + Frequently Asked Questions +============================ + +When is Flake8 released? +======================== + +|Flake8| is released *as necessary*. Sometimes there are specific goals and +drives to get to a release. Usually, we release as users report and fix +bugs. + + +How can I help Flake8 release faster? +===================================== + +Look at the next milestone. If there's work you can help us complete, that +will help us get to the next milestone. If there's a show-stopping bug that +needs to be released, let us know but please be kind. |Flake8| is developed +and released entirely on volunteer time. + + +What is the next version of Flake8? +=================================== + +In general we try to use milestones to indicate this. If the last release +on PyPI is 3.1.5 and you see a milestone for 3.2.0 in GitLab, there's a +good chance that 3.2.0 is the next release. + + +Why does Flake8 use ranges for its dependencies? +================================================ + +|Flake8| uses ranges for mccabe, pyflakes, and pycodestyle because each of +those projects tend to add *new* checks between minor releases. |Flake8| +does not restrict you from using patch versions, but |Flake8| likes to +ensure that if you install |Flake8| 2.6.x repeatedly you will not be +surprised at a later time by a new error code suddenly breaking your +linting. Instead, we use minor versions of |Flake8| to add new checks from +dependencies intentionally. + +**Please do not file issues to tell the Flake8 team that a new version is +available on PyPI.** The current Flake8 core team (of one person) is also +a core developer of pycodestyle, pyflakes, and mccabe. They are aware of +these releases. diff --git a/docs/source/index.rst b/docs/source/index.rst index 68dc3fa..55dfe3d 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -69,6 +69,14 @@ Alternatively, if you want to *ignore* only one specific warning or error: Please read our user guide for more information about how to use and configure |Flake8|. +FAQ +=== + +.. toctree:: + :maxdepth: 2 + + faq + User Guide ========== From f3cd7ee6fb346bd46d428fce53a27ba70b23de2b Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 21 Jun 2016 07:36:40 -0500 Subject: [PATCH 344/364] Update FAQ to be a bit clearer --- docs/source/faq.rst | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/docs/source/faq.rst b/docs/source/faq.rst index 877dae1..3f2bc04 100644 --- a/docs/source/faq.rst +++ b/docs/source/faq.rst @@ -31,14 +31,26 @@ Why does Flake8 use ranges for its dependencies? ================================================ |Flake8| uses ranges for mccabe, pyflakes, and pycodestyle because each of -those projects tend to add *new* checks between minor releases. |Flake8| -does not restrict you from using patch versions, but |Flake8| likes to -ensure that if you install |Flake8| 2.6.x repeatedly you will not be -surprised at a later time by a new error code suddenly breaking your -linting. Instead, we use minor versions of |Flake8| to add new checks from -dependencies intentionally. +those projects tend to add *new* checks in minor releases. It has been an +implicit design goal of |Flake8|'s to make the list of error codes stable in +its own minor releases. That way if you install something from the 2.5 +series today, you will not find new checks in the same series in a month +from now when you install it again. -**Please do not file issues to tell the Flake8 team that a new version is -available on PyPI.** The current Flake8 core team (of one person) is also +|Flake8|'s dependencies tend to avoid new checks in patch versions which is +why |Flake8| expresses its dependencies roughly as:: + + pycodestyle >= 2.0.0, < 2.1.0 + pyflakes >= 0.8.0, != 1.2.0, != 1.2.1, != 1.2.2, < 1.3.0 + mccabe >= 0.5.0, < 0.6.0 + +This allows those projects to release patch versions that fix bugs and for +|Flake8| users to consume those fixes. + + +Should I file an issue when a new version of a dependency is available? +======================================================================= + +**No.** The current Flake8 core team (of one person) is also a core developer of pycodestyle, pyflakes, and mccabe. They are aware of these releases. From 9fe82d7be00d28d1005174f8833cc542af8d9e90 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Tue, 21 Jun 2016 07:37:08 -0500 Subject: [PATCH 345/364] Update version string to beta Add a version info tuple for plugins to use --- flake8/__init__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/flake8/__init__.py b/flake8/__init__.py index 40336ad..d6c6915 100644 --- a/flake8/__init__.py +++ b/flake8/__init__.py @@ -27,7 +27,8 @@ LOG.addHandler(NullHandler()) # Clean up after LOG config del NullHandler -__version__ = '3.0.0a1' +__version__ = '3.0.0b1' +__version_info__ = tuple(int(i) for i in __version__.split('.') if i.isdigit()) # There is nothing lower than logging.DEBUG (10) in the logging library, From 60b67f4ef3facf946814d16f480a4287a5c9126f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 08:05:47 -0500 Subject: [PATCH 346/364] Trim trailing whitespace I need to get better at following my own contrib documentation --- docs/source/internal/releases.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/internal/releases.rst b/docs/source/internal/releases.rst index 2af8950..0d27d15 100644 --- a/docs/source/internal/releases.rst +++ b/docs/source/internal/releases.rst @@ -38,6 +38,6 @@ Before releasing, the following tox test environments must pass: We tag the most recent commit that passes those items and contains our release notes. -Finally, we run ``tox -e release`` to build source distributions (e.g., +Finally, we run ``tox -e release`` to build source distributions (e.g., ``flake8-3.0.0.tar.gz``), universal wheels, and upload them to PyPI with Twine. From 9526e061a7fa5468bdc072f57eeaaca9ed016aab Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 08:10:07 -0500 Subject: [PATCH 347/364] Add more detail about types of releases --- docs/source/internal/releases.rst | 64 +++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 4 deletions(-) diff --git a/docs/source/internal/releases.rst b/docs/source/internal/releases.rst index 0d27d15..a624930 100644 --- a/docs/source/internal/releases.rst +++ b/docs/source/internal/releases.rst @@ -4,15 +4,71 @@ There is not much that is hard to find about how |Flake8| is released. -- We use **major** releases (e.g., 2.0.0, 3.0.0, etc.) for big releases (e.g., - large scale refactors). This can also contain dependency version changes. +- We use **major** releases (e.g., 2.0.0, 3.0.0, etc.) for big, potentially + backwards incompatible, releases. - We use **minor** releases (e.g., 2.1.0, 2.2.0, 3.1.0, 3.2.0, etc.) for releases that contain features and dependency version changes. - We use **patch** releases (e.g., 2.1.1, 2.1.2, 3.0.1, 3.0.10, etc.) for - releases that contain *only* bug fixes. These *never* contain changes to - dependency version constraints. + releases that contain *only* bug fixes. + +In this sense we follow semantic versioning. But we follow it as more of a set +of guidelines. We're also not perfect, so we may make mistakes, and that's +fine. + + +Major Releases +============== + +Major releases are often associated with backwards incompatibility. |Flake8| +hopes to avoid those, but will occasionally need them. + +Historically, |Flake8| has generated major releases for: + +- Unvendoring dependencies (2.0) + +- Large scale refactoring (2.0, 3.0) + +- Subtly breaking CLI changes (3.0) + +- Breaking changes to its plugin interface (3.0) + +Major releases can also contain: + +- Bug fixes (which may have backwards incompatible solutions) + +- New features + +- Dependency changes + + +Minor Releases +============== + +Minor releases often have new features in them, which we define roughly as: + +- New command-line flags + +- New behaviour that does not break backwards compatibility + +- New errors detected by dependencies, e.g., by raising the upper limit on + PyFlakes we introduce F405 + +- Bug fixes + + +Patch Releases +============== + +Patch releases should only ever have bug fixes in them. + +We do not update dependency constraints in patch releases. If you do not +install |Flake8| from PyPI, there is a chance that your packager is using +different requirements. Some downstream redistributors have been known to +force a new version of PyFlakes, pep8/PyCodestyle, or McCabe into place. +Occasionally this will cause breakage when using |Flake8|. There is little +we can do to help you in those cases. Process From 6c0d900a381c21ce678b6391215fa1a48583310a Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 08:40:53 -0500 Subject: [PATCH 348/364] Write plugin cross-compatibility documentation --- docs/source/dev/cross_compatibility.rst | 149 ++++++++++++++++++++++++ docs/source/dev/index.rst | 1 + 2 files changed, 150 insertions(+) create mode 100644 docs/source/dev/cross_compatibility.rst diff --git a/docs/source/dev/cross_compatibility.rst b/docs/source/dev/cross_compatibility.rst new file mode 100644 index 0000000..c546f03 --- /dev/null +++ b/docs/source/dev/cross_compatibility.rst @@ -0,0 +1,149 @@ +==================================== + Writing Plugins For Flake8 2 and 3 +==================================== + +Plugins have existed for |Flake8| 2.x for a few years. There are a number of +these on PyPI already. While it did not seem reasonable for |Flake8| to attempt +to provide a backwards compatible shim for them, we did decide to try to +document the easiest way to write a plugin that's compatible across both +versions. + +.. note:: + + If your plugin does not register options, it *should* Just Work. + +The **only** breaking change in |Flake8| 3.0 is the fact that we no longer +check the option parser for a list of strings to parse from a config file. On +|Flake8| 2.x, to have an option parsed from the configuration files that +|Flake8| finds and parses you would have to do something like: + +.. code-block:: python + + parser.add_option('-X', '--example-flag', type='string', + help='...') + parser.config_options.append('example-flag') + +For |Flake8| 3.0, we have added *three* arguments to the +:meth:`~flake8.options.manager.OptionManager.add_option` method you will call +on the parser you receive: + +- ``parse_from_config`` which expects ``True`` or ``False`` + + When ``True``, |Flake8| will parse the option from the config files |Flake8| + finds. + +- ``comma_separated_list`` which expects ``True`` or ``False`` + + When ``True``, |Flake8| will split the string intelligently and handle + extra whitespace. The parsed value will be a list. + +- ``normalize_paths`` which expects ``True`` or ``False`` + + When ``True``, |Flake8| will: + + * remove trailing path separators (i.e., ``os.path.sep``) + + * return the absolute path for values that have the separator in them + +All three of these options can be combined or used separately. + + +Parsing Options from Configuration Files +======================================== + +The example from |Flake8| 2.x now looks like: + +.. code-block:: python + + parser.add_option('-X', '--example-flag', type='string', + parse_from_config=True, + help='...') + + +Parsing Comma-Separated Lists +============================= + +Now let's imagine that the option we want to add is expecting a comma-separatd +list of values from the user (e.g., ``--select E123,W503,F405``). |Flake8| 2.x +often forced users to parse these lists themselves since pep8 special-cased +certain flags and left others on their own. |Flake8| 3.0 adds +``comma_separated_list`` so that the parsed option is already a list for +plugin authors. When combined with ``parse_from_config`` this means that users +can also do something like: + +.. code-block:: ini + + example-flag = + first, + second, + third, + fourth, + fifth + +And |Flake8| will just return the list: + +.. code-block:: python + + ["first", "second", "third", "fourth", "fifth"] + + +Normalizing Values that Are Paths +================================= + +Finally, let's imagine that our new option wants a path or list of paths. To +ensure that these paths are semi-normalized (the way |Flake8| 2.x used to +work) we need only pass ``normalize_paths=True``. If you have specified +``comma_separated_list=True`` then this will parse the value as a list of +paths that have been normalized. Otherwise, this will parse the value +as a single path. + + +Option Handling on Flake8 2 and 3 +================================= + +So, in conclusion, we can now write our plugin that relies on registering +options with |Flake8| and have it work on |Flake8| 2.x and 3.x. + +.. code-block:: python + + option_args = ('-X', '--example-flag') + option_kwargs = { + 'type': 'string', + 'parse_from_config': True, + 'help': '...', + } + try: + # Flake8 3.x registration + parser.add_option(*option_args, **option_kwargs) + except TypeError: + # Flake8 2.x registration + parse_from_config = option_kwargs.pop('parse_from_config', False) + parser.add_option(*option_args, **option_kwargs) + if parse_from_config: + parser.config_options.append(option_args[-1].lstrip('-')) + + +Or, you can write a tiny helper function: + +.. code-block:: python + + def register_opt(parser, *args, **kwargs): + try: + # Flake8 3.x registration + parser.add_option(*args, **kwargs) + except TypeError: + # Flake8 2.x registration + parse_from_config = kwargs.pop('parse_from_config', False) + parser.add_option(*args, **kwargs) + if parse_from_config: + parser.config_options.append(args[-1].lstrip('-')) + +.. code-block:: python + + def register_options(self, parser): + register_opt(parser, '-X', '--example-flag', type='string', + parse_from_config=True, help='...') + +The transition period is admittedly not fantastic, but we believe that this +is a worthwhile change for plugin developers going forward. We also hope to +help with the transition phase for as many plugins as we can manage. diff --git a/docs/source/dev/index.rst b/docs/source/dev/index.rst index 9ef8138..858ec83 100644 --- a/docs/source/dev/index.rst +++ b/docs/source/dev/index.rst @@ -12,6 +12,7 @@ the authoritative documentation for those pieces. .. toctree:: :maxdepth: 2 + cross_compatibility plugin_parameters registering_plugins formatters From 04bceeceb5424f1a970b349656e31e0c8f9a5fb7 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 08:42:30 -0500 Subject: [PATCH 349/364] Remove historical files from the repository The old/ directory was the archive of flake8 2.0. DESIGN.rst was the place I kept my initial 3.0 notes for design purposes. --- DESIGN.rst | 185 ---------------- old/docs/Makefile | 130 ----------- old/docs/api.rst | 38 ---- old/docs/buildout.rst | 17 -- old/docs/changes.rst | 5 - old/docs/conf.py | 240 -------------------- old/docs/config.rst | 55 ----- old/docs/extensions.rst | 149 ------------- old/docs/index.rst | 26 --- old/docs/setuptools.rst | 25 --- old/docs/vcs.rst | 50 ----- old/docs/warnings.rst | 51 ----- old/flake8/__init__.py | 1 - old/flake8/__main__.py | 4 - old/flake8/_pyflakes.py | 120 ---------- old/flake8/callbacks.py | 27 --- old/flake8/compat.py | 12 - old/flake8/engine.py | 316 --------------------------- old/flake8/hooks.py | 297 ------------------------- old/flake8/main.py | 142 ------------ old/flake8/reporter.py | 152 ------------- old/flake8/run.py | 11 - old/flake8/tests/__init__.py | 1 - old/flake8/tests/_test_warnings.py | 309 -------------------------- old/flake8/tests/test_engine.py | 236 -------------------- old/flake8/tests/test_hooks.py | 59 ----- old/flake8/tests/test_integration.py | 79 ------- old/flake8/tests/test_main.py | 18 -- old/flake8/tests/test_pyflakes.py | 73 ------- old/flake8/tests/test_reporter.py | 36 --- old/flake8/tests/test_util.py | 120 ---------- old/flake8/util.py | 77 ------- old/setup.py | 75 ------- old/tox.ini | 40 ---- 34 files changed, 3176 deletions(-) delete mode 100644 DESIGN.rst delete mode 100644 old/docs/Makefile delete mode 100644 old/docs/api.rst delete mode 100644 old/docs/buildout.rst delete mode 100644 old/docs/changes.rst delete mode 100644 old/docs/conf.py delete mode 100644 old/docs/config.rst delete mode 100644 old/docs/extensions.rst delete mode 100644 old/docs/index.rst delete mode 100644 old/docs/setuptools.rst delete mode 100644 old/docs/vcs.rst delete mode 100644 old/docs/warnings.rst delete mode 100644 old/flake8/__init__.py delete mode 100644 old/flake8/__main__.py delete mode 100644 old/flake8/_pyflakes.py delete mode 100644 old/flake8/callbacks.py delete mode 100644 old/flake8/compat.py delete mode 100644 old/flake8/engine.py delete mode 100644 old/flake8/hooks.py delete mode 100644 old/flake8/main.py delete mode 100644 old/flake8/reporter.py delete mode 100644 old/flake8/run.py delete mode 100644 old/flake8/tests/__init__.py delete mode 100644 old/flake8/tests/_test_warnings.py delete mode 100644 old/flake8/tests/test_engine.py delete mode 100644 old/flake8/tests/test_hooks.py delete mode 100644 old/flake8/tests/test_integration.py delete mode 100644 old/flake8/tests/test_main.py delete mode 100644 old/flake8/tests/test_pyflakes.py delete mode 100644 old/flake8/tests/test_reporter.py delete mode 100644 old/flake8/tests/test_util.py delete mode 100644 old/flake8/util.py delete mode 100644 old/setup.py delete mode 100644 old/tox.ini diff --git a/DESIGN.rst b/DESIGN.rst deleted file mode 100644 index cac20f9..0000000 --- a/DESIGN.rst +++ /dev/null @@ -1,185 +0,0 @@ -============== - Design Goals -============== - -Outline -------- - -#. :ref:`plugins` - - #. :ref:`checking` - - #. :ref:`autofixing` - - #. :ref:`reporter-plugins` - - #. :ref:`options-passing` - - #. :ref:`plugin-default-ignore` - - #. :ref:`report-generation` - -#. :ref:`options` - - #. :ref:`better-select-ignore` - -#. :ref:`standard-in` - -#. :ref:`multiprocessing` - -.. _plugins: - -Better Plugins Support ----------------------- - -Currently, Flake8 has some rather excellent support for plugins. It currently -allows for the following: - -- Third-party packages to register checks - -- Checks to be disabled by default - -- Checks to accept an AST compiled tree, physical lines, or logical lines. - -- Flake8 handles running those checks in separate subprocesses as necessary - -That said, plugins cannot access the options passed on the command-line, or -options parsed from config files (without parsing them, themselves) and all -reporting is handled by pep8 instead of flake8 which reduces the flexibility -users have in aggregating reports. - -.. _checking: - -Support for Plugins that Only Run Checks -++++++++++++++++++++++++++++++++++++++++ - -Flake8 currently already supports plugins that only run checks. This support -needs to continue and should be trivial to continue. - -.. _autofixing: - -Support for Plugins that Autofix Errors -+++++++++++++++++++++++++++++++++++++++ - -Flake8 should enable people writing plugins for both core Flake8 checkers and -third-party checkers that allow the code to be automatically fixed. The trick -is in how to do this. - -Once Flake8 has control over running plugins and treats pep8, flake8, and -mccabe as "plugins", it will aggregate the errors returned by all of the -plugins and be able to "notify" other plugins that have chosen to listen for -errors so those plugins can auto-fix the problems in the file. - -We should also be considerate of allowing these plugins to be composable. Each -plugin should have a way of defining its capabilities. - -See https://gitlab.com/pycqa/flake8/issues/84 - -.. note:: Will probably need a Trie implementation for this - -What we *might* want is for a autofix plugin to register something like - -:: - - 'flake8.listen': [ - 'E1 = my_fixer.E1Listener', - 'E2 = my_fixer.E2Listener', - ] - -This means that the notifer would need to take an error code like ``E111`` and -then notify anything listening for ``E111``, ``E11``, ``E1``, and ``E``. - -.. _reporter-plugins: - -Support for Plugins that Format Output -++++++++++++++++++++++++++++++++++++++ - -Flake8 currently supports formatting output via pep8's ``--format`` option. -This works but is fundamentally a bit limiting. Allowing users to replace or -compose formatters would allow for certain formatters to highlight more -important information over less important information as the user deems -necessary. - -:: - - 'flake8.format': [ - 'json = my_formatter.JsonFormatter', - 'xml = my_formatter.XMLFormatter', - ] - -See https://gitlab.com/pycqa/flake8/issues/66 - -.. _options-passing: - -Support for Plugins Require Parsed Options -++++++++++++++++++++++++++++++++++++++++++ - -Plugins currently are able to use ``add_options`` and ``parse_options`` -classmethods to register and retrieve options information. This is admittedly -a little awkward and could be improved, but should at least be preserved in -this rewrite. - -See potential improvements as a result of -https://gitlab.com/pycqa/flake8/issues/88 - -.. _plugin-default-ignore: - -Support for Plugins Specifying Default Ignore list -++++++++++++++++++++++++++++++++++++++++++++++++++ - -Plugins currently have no way of extending the default ignore list. This means -they have to hard-code checks to auto-ignore errors. - -.. _options: - -Better Options Support ----------------------- - -Currently there are some options handled by pep8 that are handled poorly. -Further, the way the options work is confusing to some, e.g., when specifying -``--ignore``, users do not expect it to override the ``DEFAULT_IGNORE`` list. -Users also don't expect ``--ignore`` and ``--select`` to step on each other's -toes. - -.. _better-select-ignore: - -Support for Better Select/Ignore Handling -+++++++++++++++++++++++++++++++++++++++++ - -Currently ``--select`` and ``--ignore`` cause one or the other to be ignored. -Users presently cannot specify both for granularity. This should be -significantly improved. - -Further, new tools have developed ``--add-select`` and ``--add-ignore`` which -allows an add-only interface. This seems to be a good direction to follow. -Flake8 should support this. - -See https://github.com/PyCQA/pep8/issues/390 - -.. _standard-in: - -Better stdin support --------------------- - -Currently, flake8 accepts input from standard-in to check. It also currently -monkey-patches pep8 to cache that value. It would be better if there was one -way to retrieve the stdin input for plugins. Flake8 should provide this -directly instead of pep8 providing it. - -See -https://gitlab.com/pycqa/flake8/commit/41393c9b6de513ea169b61c175b71018e8a12336 - -.. _multiprocessing: - -Multiprocessing Support ------------------------ - -Flake8's existing multiprocessing support (and handling for different error -cases needs to persist through this redesign). - -See: - -- https://gitlab.com/pycqa/flake8/issues/8 -- https://gitlab.com/pycqa/flake8/issues/17 -- https://gitlab.com/pycqa/flake8/issues/44 -- https://gitlab.com/pycqa/flake8/issues/74 diff --git a/old/docs/Makefile b/old/docs/Makefile deleted file mode 100644 index bf49b54..0000000 --- a/old/docs/Makefile +++ /dev/null @@ -1,130 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = _build - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - -clean: - -rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/Raclette.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/Raclette.qhc" - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/Raclette" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/Raclette" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - make -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." diff --git a/old/docs/api.rst b/old/docs/api.rst deleted file mode 100644 index fd746f7..0000000 --- a/old/docs/api.rst +++ /dev/null @@ -1,38 +0,0 @@ -========== -Flake8 API -========== - -.. module:: flake8 - -flake8.engine -============= - -.. autofunction:: flake8.engine.get_parser - -.. autofunction:: flake8.engine.get_style_guide - -flake8.hooks -============ - -.. autofunction:: flake8.hooks.git_hook - -.. autofunction:: flake8.hooks.hg_hook - -flake8.main -=========== - -.. autofunction:: flake8.main.main - -.. autofunction:: flake8.main.check_file - -.. autofunction:: flake8.main.check_code - -.. autoclass:: flake8.main.Flake8Command - -flake8.util -=========== - -For AST checkers, this module has the ``iter_child_nodes`` function and -handles compatibility for all versions of Python between 2.5 and 3.3. The -function was added to the ``ast`` module in Python 2.6 but is redefined in the -case where the user is running Python 2.5 diff --git a/old/docs/buildout.rst b/old/docs/buildout.rst deleted file mode 100644 index da9c58a..0000000 --- a/old/docs/buildout.rst +++ /dev/null @@ -1,17 +0,0 @@ -Buildout integration -===================== - -In order to use Flake8 inside a buildout, edit your buildout.cfg and add this:: - - [buildout] - - parts += - ... - flake8 - - [flake8] - recipe = zc.recipe.egg - eggs = flake8 - ${buildout:eggs} - entry-points = - flake8=flake8.main:main diff --git a/old/docs/changes.rst b/old/docs/changes.rst deleted file mode 100644 index 43731a7..0000000 --- a/old/docs/changes.rst +++ /dev/null @@ -1,5 +0,0 @@ -Changes -======= - -.. include:: ../CHANGES.rst - :start-line: 3 diff --git a/old/docs/conf.py b/old/docs/conf.py deleted file mode 100644 index 247e44b..0000000 --- a/old/docs/conf.py +++ /dev/null @@ -1,240 +0,0 @@ -# -*- coding: utf-8 -*- -# -# This file is execfile()d with the current directory set to its -# containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys -import os - -# This environment variable makes decorators not decorate functions, so their -# signatures in the generated documentation are still correct -os.environ['GENERATING_DOCUMENTATION'] = "flake8" - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath('..')) -import flake8 - -# -- General configuration ---------------------------------------------------- - -# If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.autodoc'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix of source filenames. -source_suffix = '.rst' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'flake8' -copyright = u'2012-2013 - Tarek Ziade, Ian Cordasco, Florent Xicluna' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = flake8.__version__ -# The full version, including alpha/beta/rc tags. -release = version - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -#language = None - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build'] - -# The reST default role (used for this markup: `text`) to use for -# all documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -# pygments_style = 'flask_theme_support.FlaskyStyle' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - - -# -- Options for HTML output -------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -#html_theme = 'nature' - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = None - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -#html_static_path = ['_static'] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = False - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = False - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Output file base name for HTML help builder. -htmlhelp_basename = 'flake8_doc' - - -# -- Options for LaTeX output ------------------------------------------------- - -# The paper size ('letter' or 'a4'). -#latex_paper_size = 'letter' - -# The font size ('10pt', '11pt' or '12pt'). -#latex_font_size = '10pt' - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, -# author, documentclass [howto/manual]). -latex_documents = [ - ('index', 'flake8.tex', u'flake8 Documentation', - u'Tarek Ziade', 'manual'), -] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Additional stuff for the LaTeX preamble. -#latex_preamble = '' - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output ------------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -man_pages = [ - ('index', 'flake8', u'flake8 Documentation', - [u'Tarek Ziade', u'Ian Cordasco', u'Florent Xicluna'], 1) -] - -# If true, show URL addresses after external links. -#man_show_urls = False - - -# -- Options for Texinfo output ----------------------------------------------- - -# Grouping the document tree into Texinfo files. List of tuples -# (source start file, target name, title, author, -# dir menu entry, description, category) -texinfo_documents = [ - ('index', 'flake8', u'flake8 Documentation', u'Tarek Ziade', - 'flake8', 'Code checking using pep8, pyflakes and mccabe', - 'Miscellaneous'), -] - -# Documents to append as an appendix to all manuals. -texinfo_appendices = [] diff --git a/old/docs/config.rst b/old/docs/config.rst deleted file mode 100644 index 59ba3ec..0000000 --- a/old/docs/config.rst +++ /dev/null @@ -1,55 +0,0 @@ -Configuration -============= - -The behaviour may be configured at two levels. - -Global ------- - -The user settings are read from the ``~/.config/flake8`` file (or the ``~/.flake8`` file on Windows). -Example:: - - [flake8] - ignore = E226,E302,E41 - max-line-length = 160 - exclude = tests/* - max-complexity = 10 - -Per-Project ------------ - -At the project level, the ``tox.ini``, ``setup.cfg``, ``.pep8`` or ``.flake8`` -files are read if present. Only the first file is considered. If this file -does not have a ``[flake8]`` section, no project specific configuration is -loaded. - -Default -------- - -If the ``ignore`` option is not in the configuration and not in the arguments, -only the error codes ``E123/E133``, ``E226`` and ``E241/E242`` are ignored -(see the :ref:`warning and error codes `). - -Settings --------- - -This is a (likely incomplete) list of settings that can be used in your config -file. In general, any settings that pep8 supports we also support and we add -the ability to set ``max-complexity`` as well. - -- ``exclude``: comma-separated filename and glob patterns - default: ``.svn,CVS,.bzr,.hg,.git,__pycache`` - -- ``filename``: comma-separated filename and glob patterns - default: ``*.py`` - -- ``select``: select errors and warnings to enable which are off by default - -- ``ignore``: skip errors or warnings - -- ``max-line-length``: set maximum allowed line length - default: 79 - -- ``format``: set the error format - -- ``max-complexity``: McCabe complexity threshold diff --git a/old/docs/extensions.rst b/old/docs/extensions.rst deleted file mode 100644 index 9e89fb1..0000000 --- a/old/docs/extensions.rst +++ /dev/null @@ -1,149 +0,0 @@ -Writing an Extension for Flake8 -=============================== - -Since Flake8 is now adding support for extensions, we require ``setuptools`` -so we can manage extensions through entry points. If you are making an -existing tool compatible with Flake8 but do not already require -``setuptools``, you should probably add it to your list of requirements. Next, -you'll need to edit your ``setup.py`` file so that upon installation, your -extension is registered. If you define a class called ``PackageEntryClass`` -then this would look something like the following:: - - - setup( - # ... - entry_points={ - 'flake8.extension': ['P10 = package.PackageEntryClass'], - } - # ... - ) - - -If you intend to publish your extension, choose a unique code prefix -following the convention for :ref:`error codes `. -In addition, you can open a request in the `issue tracker -`_ to register the prefix in the -documentation. - -.. TODO: describe the API required for the 3 kind of extensions: - * physical line checkers - * logical line checkers - * AST checkers - - -A real example: McCabe ----------------------- - -Below is an example from mccabe_ for how to write your ``setup.py`` file for -your Flake8 extension. - -.. code-block:: python - - # https://github.com/flintwork/mccabe/blob/0.2/setup.py#L38:L42 - # -*- coding: utf-8 -*- - from setuptools import setup - - # ... - - setup( - name='mccabe', - - # ... - - install_requires=[ - 'setuptools', - ], - entry_points={ - 'flake8.extension': [ - 'C90 = mccabe:McCabeChecker', - ], - }, - - # ... - - ) - -In ``mccabe.py`` you can see that extra options are added to the parser when -flake8 registers the extension: - -.. code-block:: python - - # https://github.com/flintwork/mccabe/blob/0.2/mccabe.py#L225:L254 - class McCabeChecker(object): - """McCabe cyclomatic complexity checker.""" - name = 'mccabe' - version = __version__ - _code = 'C901' - _error_tmpl = "C901 %r is too complex (%d)" - max_complexity = 0 - - def __init__(self, tree, filename): - self.tree = tree - - @classmethod - def add_options(cls, parser): - parser.add_option('--max-complexity', default=-1, action='store', - type='int', help="McCabe complexity threshold") - parser.config_options.append('max-complexity') - - @classmethod - def parse_options(cls, options): - cls.max_complexity = options.max_complexity - - def run(self): - if self.max_complexity < 0: - return - visitor = PathGraphingAstVisitor() - visitor.preorder(self.tree, visitor) - for graph in visitor.graphs.values(): - if graph.complexity() >= self.max_complexity: - text = self._error_tmpl % (graph.entity, graph.complexity()) - yield graph.lineno, 0, text, type(self) - -Since that is the defined entry point in the above ``setup.py``, flake8 finds -it and uses it to register the extension. - -If we wanted the extension or a check to be optional, you can add -``off_by_default = True`` to our entry point. For example, we could -update ``mccabe.py`` with this variable as shown below: - -.. code-block:: python - - # https://github.com/flintwork/mccabe/blob/0.2/mccabe.py#L225:L254 - class McCabeChecker(object): - """McCabe cyclomatic complexity checker.""" - name = 'mccabe' - version = __version__ - off_by_default = True - -If we wanted to run the optional extension or check, we need to specify the -error and warnings via the ``--enable-extension`` command line argument. In our -case, we could run ``flake8 --enable-extension=C90`` which would enable our -off_by_default example version of the mccabe extension. - - -Existing Extensions -=================== - -This is not at all a comprehensive listing of existing extensions but simply a -listing of the ones we are aware of: - -* `flake8-debugger `_ - -* `flake8-immediate `_ - -* `flake8-print `_ - -* `flake8-todo `_ - -* `pep8-naming `_ - -* `radon `_ - -* `flake8-import-order `_ - -* `flake8-respect-noqa `_ - -.. links -.. _mccabe: https://github.com/flintwork/mccabe -.. _PyPI: https://pypi.python.org/pypi/ diff --git a/old/docs/index.rst b/old/docs/index.rst deleted file mode 100644 index e2dc514..0000000 --- a/old/docs/index.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. include:: ../README.rst - - -Documentation -============= - -.. toctree:: - - config - warnings - vcs - buildout - setuptools - api - extensions - changes - -Original Projects -================= - -Flake8 is just a glue project, all the merits go to the creators of the original -projects: - -- pep8: https://github.com/jcrocholl/pep8 -- PyFlakes: https://launchpad.net/pyflakes -- McCabe: http://nedbatchelder.com/blog/200803/python_code_complexity_microtool.html diff --git a/old/docs/setuptools.rst b/old/docs/setuptools.rst deleted file mode 100644 index 8bc080d..0000000 --- a/old/docs/setuptools.rst +++ /dev/null @@ -1,25 +0,0 @@ -Setuptools integration -====================== - -Upon installation, Flake8 enables a setuptools command that checks Python -files declared by your project. - -Running ``python setup.py flake8`` on the command line will check the files -listed in your ``py_modules`` and ``packages``. If any warning is found, -the command will exit with an error code:: - - $ python setup.py flake8 - -Also, to allow users to be able to use the command without having to install -flake8 themselves, add flake8 to the setup_requires of your setup() like so:: - - setup( - name="project", - packages=["project"], - - setup_requires=[ - "flake8" - ] - ) - - diff --git a/old/docs/vcs.rst b/old/docs/vcs.rst deleted file mode 100644 index 66538b4..0000000 --- a/old/docs/vcs.rst +++ /dev/null @@ -1,50 +0,0 @@ -VCS Hooks -========= - -flake8 can install hooks for Mercurial and Git so that flake8 is run -automatically before commits. The commit will fail if there are any -flake8 issues. - -You can install the hook by issuing this command in the root of your -project:: - - $ flake8 --install-hook - -In the case of Git, the hook won't be installed if a custom -``pre-commit`` hook file is already present in -the ``.git/hooks`` directory. - -You can control the behavior of the pre-commit hook using configuration file -settings or environment variables: - -``flake8.complexity`` or ``FLAKE8_COMPLEXITY`` - Any value > 0 enables complexity checking with McCabe. (defaults - to 10) - -``flake8.strict`` or ``FLAKE8_STRICT`` - If True, this causes the commit to fail in case of any errors at - all. (defaults to False) - -``flake8.ignore`` or ``FLAKE8_IGNORE`` - Comma-separated list of errors and warnings to ignore. (defaults to - empty) - -``flake8.lazy`` or ``FLAKE8_LAZY`` - If True, also scans those files not added to the index before - commit. (defaults to False) - -You can set these either through the git command line - -.. code-block:: bash-session - - $ git config flake8.complexity 10 - $ git config flake8.strict true - -Or by directly editing ``.git/config`` and adding a section like - -.. code-block:: ini - - [flake8] - complexity = 10 - strict = true - lazy = false diff --git a/old/docs/warnings.rst b/old/docs/warnings.rst deleted file mode 100644 index b4b7fb1..0000000 --- a/old/docs/warnings.rst +++ /dev/null @@ -1,51 +0,0 @@ -.. _error-codes: - -Warning / Error codes -===================== - -The convention of Flake8 is to assign a code to each error or warning, like -the ``pep8`` tool. These codes are used to configure the list of errors -which are selected or ignored. - -Each code consists of an upper case ASCII letter followed by three digits. -The recommendation is to use a different prefix for each plugin. A list of the -known prefixes is published below: - -- ``E***``/``W***``: `pep8 errors and warnings - `_ -- ``F***``: PyFlakes codes (see below) -- ``C9**``: McCabe complexity plugin `mccabe - `_ -- ``N8**``: Naming Conventions plugin `pep8-naming - `_ - - -The original PyFlakes does not provide error codes. Flake8 patches the -PyFlakes messages to add the following codes: - -+------+--------------------------------------------------------------------+ -| code | sample message | -+======+====================================================================+ -| F401 | ``module`` imported but unused | -+------+--------------------------------------------------------------------+ -| F402 | import ``module`` from line ``N`` shadowed by loop variable | -+------+--------------------------------------------------------------------+ -| F403 | 'from ``module`` import \*' used; unable to detect undefined names | -+------+--------------------------------------------------------------------+ -| F404 | future import(s) ``name`` after other statements | -+------+--------------------------------------------------------------------+ -+------+--------------------------------------------------------------------+ -| F811 | redefinition of unused ``name`` from line ``N`` | -+------+--------------------------------------------------------------------+ -| F812 | list comprehension redefines ``name`` from line ``N`` | -+------+--------------------------------------------------------------------+ -| F821 | undefined name ``name`` | -+------+--------------------------------------------------------------------+ -| F822 | undefined name ``name`` in __all__ | -+------+--------------------------------------------------------------------+ -| F823 | local variable ``name`` ... referenced before assignment | -+------+--------------------------------------------------------------------+ -| F831 | duplicate argument ``name`` in function definition | -+------+--------------------------------------------------------------------+ -| F841 | local variable ``name`` is assigned to but never used | -+------+--------------------------------------------------------------------+ diff --git a/old/flake8/__init__.py b/old/flake8/__init__.py deleted file mode 100644 index 36dc058..0000000 --- a/old/flake8/__init__.py +++ /dev/null @@ -1 +0,0 @@ -__version__ = '2.5.4' diff --git a/old/flake8/__main__.py b/old/flake8/__main__.py deleted file mode 100644 index aaa497b..0000000 --- a/old/flake8/__main__.py +++ /dev/null @@ -1,4 +0,0 @@ -from flake8.main import main - -# python -m flake8 (with Python >= 2.7) -main() diff --git a/old/flake8/_pyflakes.py b/old/flake8/_pyflakes.py deleted file mode 100644 index 976b2ab..0000000 --- a/old/flake8/_pyflakes.py +++ /dev/null @@ -1,120 +0,0 @@ -# -*- coding: utf-8 -*- -try: - # The 'demandimport' breaks pyflakes and flake8._pyflakes - from mercurial import demandimport -except ImportError: - pass -else: - demandimport.disable() -import os - -import pep8 -import pyflakes -import pyflakes.checker - - -def patch_pyflakes(): - """Add error codes to Pyflakes messages.""" - codes = dict([line.split()[::-1] for line in ( - 'F401 UnusedImport', - 'F402 ImportShadowedByLoopVar', - 'F403 ImportStarUsed', - 'F404 LateFutureImport', - 'F810 Redefined', # XXX Obsolete? - 'F811 RedefinedWhileUnused', - 'F812 RedefinedInListComp', - 'F821 UndefinedName', - 'F822 UndefinedExport', - 'F823 UndefinedLocal', - 'F831 DuplicateArgument', - 'F841 UnusedVariable', - )]) - - for name, obj in vars(pyflakes.messages).items(): - if name[0].isupper() and obj.message: - obj.flake8_msg = '%s %s' % (codes.get(name, 'F999'), obj.message) -patch_pyflakes() - - -class FlakesChecker(pyflakes.checker.Checker): - """Subclass the Pyflakes checker to conform with the flake8 API.""" - name = 'pyflakes' - version = pyflakes.__version__ - - def __init__(self, tree, filename): - filename = pep8.normalize_paths(filename)[0] - withDoctest = self.withDoctest - included_by = [include for include in self.include_in_doctest - if include != '' and filename.startswith(include)] - if included_by: - withDoctest = True - - for exclude in self.exclude_from_doctest: - if exclude != '' and filename.startswith(exclude): - withDoctest = False - overlaped_by = [include for include in included_by - if include.startswith(exclude)] - - if overlaped_by: - withDoctest = True - - super(FlakesChecker, self).__init__(tree, filename, - withDoctest=withDoctest) - - @classmethod - def add_options(cls, parser): - parser.add_option('--builtins', - help="define more built-ins, comma separated") - parser.add_option('--doctests', default=False, action='store_true', - help="check syntax of the doctests") - parser.add_option('--include-in-doctest', default='', - dest='include_in_doctest', - help='Run doctests only on these files', - type='string') - parser.add_option('--exclude-from-doctest', default='', - dest='exclude_from_doctest', - help='Skip these files when running doctests', - type='string') - parser.config_options.extend(['builtins', 'doctests', - 'include-in-doctest', - 'exclude-from-doctest']) - - @classmethod - def parse_options(cls, options): - if options.builtins: - cls.builtIns = cls.builtIns.union(options.builtins.split(',')) - cls.withDoctest = options.doctests - - included_files = [] - for included_file in options.include_in_doctest.split(','): - if included_file == '': - continue - if not included_file.startswith((os.sep, './', '~/')): - included_files.append('./' + included_file) - else: - included_files.append(included_file) - cls.include_in_doctest = pep8.normalize_paths(','.join(included_files)) - - excluded_files = [] - for excluded_file in options.exclude_from_doctest.split(','): - if excluded_file == '': - continue - if not excluded_file.startswith((os.sep, './', '~/')): - excluded_files.append('./' + excluded_file) - else: - excluded_files.append(excluded_file) - cls.exclude_from_doctest = pep8.normalize_paths( - ','.join(excluded_files)) - - inc_exc = set(cls.include_in_doctest).intersection( - set(cls.exclude_from_doctest)) - if inc_exc: - raise ValueError('"%s" was specified in both the ' - 'include-in-doctest and exclude-from-doctest ' - 'options. You are not allowed to specify it in ' - 'both for doctesting.' % inc_exc) - - def run(self): - for m in self.messages: - col = getattr(m, 'col', 0) - yield m.lineno, col, (m.flake8_msg % m.message_args), m.__class__ diff --git a/old/flake8/callbacks.py b/old/flake8/callbacks.py deleted file mode 100644 index 3767f30..0000000 --- a/old/flake8/callbacks.py +++ /dev/null @@ -1,27 +0,0 @@ -import atexit -import sys - - -def install_vcs_hook(option, option_str, value, parser): - # For now, there's no way to affect a change in how pep8 processes - # options. If no args are provided and there's no config file present, - # it will error out because no input was provided. To get around this, - # when we're using --install-hook, we'll say that there were arguments so - # we can actually attempt to install the hook. - # See: https://gitlab.com/pycqa/flake8/issues/2 and - # https://github.com/jcrocholl/pep8/blob/4c5bf00cb613be617c7f48d3b2b82a1c7b895ac1/pep8.py#L1912 - # for more context. - parser.values.install_hook = True - parser.rargs.append('.') - - -def restore_stdout(old_stdout): - sys.stdout.close() - sys.stdout = old_stdout - - -def redirect_stdout(option, option_str, value, parser): - fd = open(value, 'w') - old_stdout, sys.stdout = sys.stdout, fd - - atexit.register(restore_stdout, old_stdout) diff --git a/old/flake8/compat.py b/old/flake8/compat.py deleted file mode 100644 index 9bd00a7..0000000 --- a/old/flake8/compat.py +++ /dev/null @@ -1,12 +0,0 @@ -# -*- coding: utf-8 -*- -"""Compatibility shims for Flake8.""" -import os.path -import sys - - -def relpath(path, start='.'): - """Wallpaper over the differences between 2.6 and newer versions.""" - if sys.version_info < (2, 7) and path.startswith(start): - return path[len(start):] - else: - return os.path.relpath(path, start=start) diff --git a/old/flake8/engine.py b/old/flake8/engine.py deleted file mode 100644 index 816f1ee..0000000 --- a/old/flake8/engine.py +++ /dev/null @@ -1,316 +0,0 @@ -# -*- coding: utf-8 -*- -import errno -import io -import platform -import re -import sys -import warnings - -import pep8 - -from flake8 import __version__ -from flake8 import callbacks -from flake8.reporter import (multiprocessing, BaseQReport, FileQReport, - QueueReport) -from flake8 import util - -_flake8_noqa = re.compile(r'\s*# flake8[:=]\s*noqa', re.I).search - -EXTRA_EXCLUDE = ['.tox', '.eggs', '*.egg'] - -pep8.PROJECT_CONFIG += ('.flake8',) - - -def _load_entry_point(entry_point, verify_requirements): - """Based on the version of setuptools load an entry-point correctly. - - setuptools 11.3 deprecated `require=False` in the call to EntryPoint.load. - To load entry points correctly after that without requiring all - dependencies be present, the proper way is to call EntryPoint.resolve. - - This function will provide backwards compatibility for older versions of - setuptools while also ensuring we do the right thing for the future. - """ - if hasattr(entry_point, 'resolve') and hasattr(entry_point, 'require'): - if verify_requirements: - entry_point.require() - plugin = entry_point.resolve() - else: - plugin = entry_point.load(require=verify_requirements) - - return plugin - - -def _register_extensions(): - """Register all the extensions.""" - extensions = util.OrderedSet() - extensions.add(('pep8', pep8.__version__)) - parser_hooks = [] - options_hooks = [] - ignored_hooks = [] - try: - from pkg_resources import iter_entry_points - except ImportError: - pass - else: - for entry in iter_entry_points('flake8.extension'): - # Do not verify that the requirements versions are valid - checker = _load_entry_point(entry, verify_requirements=False) - pep8.register_check(checker, codes=[entry.name]) - extensions.add((checker.name, checker.version)) - if hasattr(checker, 'add_options'): - parser_hooks.append(checker.add_options) - if hasattr(checker, 'parse_options'): - options_hooks.append(checker.parse_options) - if getattr(checker, 'off_by_default', False) is True: - ignored_hooks.append(entry.name) - return extensions, parser_hooks, options_hooks, ignored_hooks - - -def get_parser(): - """This returns an instance of optparse.OptionParser with all the - extensions registered and options set. This wraps ``pep8.get_parser``. - """ - (extensions, parser_hooks, options_hooks, ignored) = _register_extensions() - details = ', '.join('%s: %s' % ext for ext in extensions) - python_version = get_python_version() - parser = pep8.get_parser('flake8', '%s (%s) %s' % ( - __version__, details, python_version - )) - for opt in ('--repeat', '--testsuite', '--doctest'): - try: - parser.remove_option(opt) - except ValueError: - pass - - if multiprocessing: - parser.config_options.append('jobs') - parser.add_option('-j', '--jobs', type='string', default='auto', - help="number of jobs to run simultaneously, " - "or 'auto'. This is ignored on Windows.") - - parser.add_option('--exit-zero', action='store_true', - help="exit with code 0 even if there are errors") - for parser_hook in parser_hooks: - parser_hook(parser) - # See comment above regarding why this has to be a callback. - parser.add_option('--install-hook', default=False, dest='install_hook', - help='Install the appropriate hook for this ' - 'repository.', action='callback', - callback=callbacks.install_vcs_hook) - parser.add_option('--output-file', default=None, - help='Redirect report to a file.', - type='string', nargs=1, action='callback', - callback=callbacks.redirect_stdout) - parser.add_option('--enable-extensions', default='', - dest='enable_extensions', - help='Enable plugins and extensions that are disabled ' - 'by default', - type='string') - parser.config_options.extend(['output-file', 'enable-extensions']) - parser.ignored_extensions = ignored - return parser, options_hooks - - -class NoQAStyleGuide(pep8.StyleGuide): - - def input_file(self, filename, lines=None, expected=None, line_offset=0): - """Run all checks on a Python source file.""" - if self.options.verbose: - print('checking %s' % filename) - fchecker = self.checker_class( - filename, lines=lines, options=self.options) - # Any "flake8: noqa" comments to ignore the entire file? - if any(_flake8_noqa(line) for line in fchecker.lines): - return 0 - return fchecker.check_all(expected=expected, line_offset=line_offset) - - -class StyleGuide(object): - """A wrapper StyleGuide object for Flake8 usage. - - This allows for OSErrors to be caught in the styleguide and special logic - to be used to handle those errors. - """ - - # Reasoning for error numbers is in-line below - serial_retry_errors = set([ - # ENOSPC: Added by sigmavirus24 - # > On some operating systems (OSX), multiprocessing may cause an - # > ENOSPC error while trying to trying to create a Semaphore. - # > In those cases, we should replace the customized Queue Report - # > class with pep8's StandardReport class to ensure users don't run - # > into this problem. - # > (See also: https://gitlab.com/pycqa/flake8/issues/74) - errno.ENOSPC, - # NOTE(sigmavirus24): When adding to this list, include the reasoning - # on the lines before the error code and always append your error - # code. Further, please always add a trailing `,` to reduce the visual - # noise in diffs. - ]) - - def __init__(self, **kwargs): - # This allows us to inject a mocked StyleGuide in the tests. - self._styleguide = kwargs.pop('styleguide', NoQAStyleGuide(**kwargs)) - - @property - def options(self): - return self._styleguide.options - - @property - def paths(self): - return self._styleguide.paths - - def _retry_serial(self, func, *args, **kwargs): - """This will retry the passed function in serial if necessary. - - In the event that we encounter an OSError with an errno in - :attr:`serial_retry_errors`, this function will retry this function - using pep8's default Report class which operates in serial. - """ - try: - return func(*args, **kwargs) - except OSError as oserr: - if oserr.errno in self.serial_retry_errors: - self.init_report(pep8.StandardReport) - else: - raise - return func(*args, **kwargs) - - def check_files(self, paths=None): - return self._retry_serial(self._styleguide.check_files, paths=paths) - - def excluded(self, filename, parent=None): - return self._styleguide.excluded(filename, parent=parent) - - def init_report(self, reporter=None): - return self._styleguide.init_report(reporter) - - def input_file(self, filename, lines=None, expected=None, line_offset=0): - return self._retry_serial( - self._styleguide.input_file, - filename=filename, - lines=lines, - expected=expected, - line_offset=line_offset, - ) - - -def _parse_multi_options(options, split_token=','): - r"""Split and strip and discard empties. - - Turns the following: - - A, - B, - - into ["A", "B"]. - - Credit: Kristian Glass as contributed to pep8 - """ - if options: - return [o.strip() for o in options.split(split_token) if o.strip()] - else: - return options - - -def _disable_extensions(parser, options): - ignored_extensions = set(getattr(parser, 'ignored_extensions', [])) - enabled = set(_parse_multi_options(options.enable_extensions)) - - # Remove any of the selected extensions from the extensions ignored by - # default. - ignored_extensions -= enabled - - # Whatever is left afterwards should be unioned with options.ignore and - # options.ignore should be updated with that. - options.ignore = tuple(ignored_extensions.union(options.ignore)) - - -def get_style_guide(**kwargs): - """Parse the options and configure the checker. This returns a sub-class - of ``pep8.StyleGuide``.""" - kwargs['parser'], options_hooks = get_parser() - styleguide = StyleGuide(**kwargs) - options = styleguide.options - _disable_extensions(kwargs['parser'], options) - - if options.exclude and not isinstance(options.exclude, list): - options.exclude = pep8.normalize_paths(options.exclude) - elif not options.exclude: - options.exclude = [] - - # Add patterns in EXTRA_EXCLUDE to the list of excluded patterns - options.exclude.extend(pep8.normalize_paths(EXTRA_EXCLUDE)) - - for options_hook in options_hooks: - options_hook(options) - - if util.warn_when_using_jobs(options): - if not multiprocessing: - warnings.warn("The multiprocessing module is not available. " - "Ignoring --jobs arguments.") - if util.is_windows(): - warnings.warn("The --jobs option is not available on Windows. " - "Ignoring --jobs arguments.") - if util.is_using_stdin(styleguide.paths): - warnings.warn("The --jobs option is not compatible with supplying " - "input using - . Ignoring --jobs arguments.") - if options.diff: - warnings.warn("The --diff option was specified with --jobs but " - "they are not compatible. Ignoring --jobs arguments." - ) - - if options.diff: - options.jobs = None - - force_disable_jobs = util.force_disable_jobs(styleguide) - - if multiprocessing and options.jobs and not force_disable_jobs: - if options.jobs.isdigit(): - n_jobs = int(options.jobs) - else: - try: - n_jobs = multiprocessing.cpu_count() - except NotImplementedError: - n_jobs = 1 - if n_jobs > 1: - options.jobs = n_jobs - reporter = QueueReport - if options.quiet: - reporter = BaseQReport - if options.quiet == 1: - reporter = FileQReport - report = styleguide.init_report(reporter) - report.input_file = styleguide.input_file - styleguide.runner = report.task_queue.put - - return styleguide - - -def get_python_version(): - # The implementation isn't all that important. - try: - impl = platform.python_implementation() + " " - except AttributeError: # Python 2.5 - impl = '' - return '%s%s on %s' % (impl, platform.python_version(), platform.system()) - - -def make_stdin_get_value(original): - def stdin_get_value(): - if not hasattr(stdin_get_value, 'cached_stdin'): - value = original() - if sys.version_info < (3, 0): - stdin = io.BytesIO(value) - else: - stdin = io.StringIO(value) - stdin_get_value.cached_stdin = stdin - else: - stdin = stdin_get_value.cached_stdin - return stdin.getvalue() - - return stdin_get_value - - -pep8.stdin_get_value = make_stdin_get_value(pep8.stdin_get_value) diff --git a/old/flake8/hooks.py b/old/flake8/hooks.py deleted file mode 100644 index 14f844e..0000000 --- a/old/flake8/hooks.py +++ /dev/null @@ -1,297 +0,0 @@ -# -*- coding: utf-8 -*- -from __future__ import with_statement -import os -import pep8 -import sys -import stat -from subprocess import Popen, PIPE -import shutil -import tempfile -try: - from configparser import ConfigParser -except ImportError: # Python 2 - from ConfigParser import ConfigParser - -from flake8 import compat -from flake8.engine import get_parser, get_style_guide -from flake8.main import DEFAULT_CONFIG - - -def git_hook(complexity=-1, strict=False, ignore=None, lazy=False): - """This is the function used by the git hook. - - :param int complexity: (optional), any value > 0 enables complexity - checking with mccabe - :param bool strict: (optional), if True, this returns the total number of - errors which will cause the hook to fail - :param str ignore: (optional), a comma-separated list of errors and - warnings to ignore - :param bool lazy: (optional), allows for the instances where you don't add - the files to the index before running a commit, e.g., git commit -a - :returns: total number of errors if strict is True, otherwise 0 - """ - gitcmd = "git diff-index --cached --name-only --diff-filter=ACMRTUXB HEAD" - if lazy: - # Catch all files, including those not added to the index - gitcmd = gitcmd.replace('--cached ', '') - - if hasattr(ignore, 'split'): - ignore = ignore.split(',') - - # Returns the exit code, list of files modified, list of error messages - _, files_modified, _ = run(gitcmd) - - # We only want to pass ignore and max_complexity if they differ from the - # defaults so that we don't override a local configuration file - options = {} - if ignore: - options['ignore'] = ignore - if complexity > -1: - options['max_complexity'] = complexity - - tmpdir = tempfile.mkdtemp() - - flake8_style = get_style_guide(config_file=DEFAULT_CONFIG, paths=['.'], - **options) - filepatterns = flake8_style.options.filename - - # Copy staged versions to temporary directory - files_to_check = [] - try: - for file_ in files_modified: - # get the staged version of the file - gitcmd_getstaged = "git show :%s" % file_ - _, out, _ = run(gitcmd_getstaged, raw_output=True, decode=False) - # write the staged version to temp dir with its full path to - # avoid overwriting files with the same name - dirname, filename = os.path.split(os.path.abspath(file_)) - prefix = os.path.commonprefix([dirname, tmpdir]) - dirname = compat.relpath(dirname, start=prefix) - dirname = os.path.join(tmpdir, dirname) - if not os.path.isdir(dirname): - os.makedirs(dirname) - - # check_files() only does this check if passed a dir; so we do it - if ((pep8.filename_match(file_, filepatterns) and - not flake8_style.excluded(file_))): - - filename = os.path.join(dirname, filename) - files_to_check.append(filename) - # write staged version of file to temporary directory - with open(filename, "wb") as fh: - fh.write(out) - - # Run the checks - report = flake8_style.check_files(files_to_check) - # remove temporary directory - finally: - shutil.rmtree(tmpdir, ignore_errors=True) - - if strict: - return report.total_errors - - return 0 - - -def hg_hook(ui, repo, **kwargs): - """This is the function executed directly by Mercurial as part of the - hook. This is never called directly by the user, so the parameters are - undocumented. If you would like to learn more about them, please feel free - to read the official Mercurial documentation. - """ - complexity = ui.config('flake8', 'complexity', default=-1) - strict = ui.configbool('flake8', 'strict', default=True) - ignore = ui.config('flake8', 'ignore', default=None) - config = ui.config('flake8', 'config', default=DEFAULT_CONFIG) - - paths = _get_files(repo, **kwargs) - - # We only want to pass ignore and max_complexity if they differ from the - # defaults so that we don't override a local configuration file - options = {} - if ignore: - options['ignore'] = ignore - if complexity > -1: - options['max_complexity'] = complexity - - flake8_style = get_style_guide(config_file=config, paths=['.'], - **options) - report = flake8_style.check_files(paths) - - if strict: - return report.total_errors - - return 0 - - -def run(command, raw_output=False, decode=True): - p = Popen(command.split(), stdout=PIPE, stderr=PIPE) - (stdout, stderr) = p.communicate() - # On python 3, subprocess.Popen returns bytes objects which expect - # endswith to be given a bytes object or a tuple of bytes but not native - # string objects. This is simply less mysterious than using b'.py' in the - # endswith method. That should work but might still fail horribly. - if decode: - if hasattr(stdout, 'decode'): - stdout = stdout.decode('utf-8') - if hasattr(stderr, 'decode'): - stderr = stderr.decode('utf-8') - if not raw_output: - stdout = [line.strip() for line in stdout.splitlines()] - stderr = [line.strip() for line in stderr.splitlines()] - return (p.returncode, stdout, stderr) - - -def _get_files(repo, **kwargs): - seen = set() - for rev in range(repo[kwargs['node']], len(repo)): - for file_ in repo[rev].files(): - file_ = os.path.join(repo.root, file_) - if file_ in seen or not os.path.exists(file_): - continue - seen.add(file_) - if file_.endswith('.py'): - yield file_ - - -def find_vcs(): - try: - _, git_dir, _ = run('git rev-parse --git-dir') - except OSError: - pass - else: - if git_dir and os.path.isdir(git_dir[0]): - if not os.path.isdir(os.path.join(git_dir[0], 'hooks')): - os.mkdir(os.path.join(git_dir[0], 'hooks')) - return os.path.join(git_dir[0], 'hooks', 'pre-commit') - try: - _, hg_dir, _ = run('hg root') - except OSError: - pass - else: - if hg_dir and os.path.isdir(hg_dir[0]): - return os.path.join(hg_dir[0], '.hg', 'hgrc') - return '' - - -def get_git_config(option, opt_type='', convert_type=True): - # type can be --bool, --int or an empty string - _, git_cfg_value, _ = run('git config --get %s %s' % (opt_type, option), - raw_output=True) - git_cfg_value = git_cfg_value.strip() - if not convert_type: - return git_cfg_value - if opt_type == '--bool': - git_cfg_value = git_cfg_value.lower() == 'true' - elif git_cfg_value and opt_type == '--int': - git_cfg_value = int(git_cfg_value) - return git_cfg_value - - -_params = { - 'FLAKE8_COMPLEXITY': '--int', - 'FLAKE8_STRICT': '--bool', - 'FLAKE8_IGNORE': '', - 'FLAKE8_LAZY': '--bool', -} - - -def get_git_param(option, default=''): - global _params - opt_type = _params[option] - param_value = get_git_config(option.lower().replace('_', '.'), - opt_type=opt_type, convert_type=False) - if param_value == '': - param_value = os.environ.get(option, default) - if opt_type == '--bool' and not isinstance(param_value, bool): - param_value = param_value.lower() == 'true' - elif param_value and opt_type == '--int': - param_value = int(param_value) - return param_value - - -git_hook_file = """#!/usr/bin/env python -import sys -from flake8.hooks import git_hook, get_git_param - -# `get_git_param` will retrieve configuration from your local git config and -# then fall back to using the environment variables that the hook has always -# supported. -# For example, to set the complexity, you'll need to do: -# git config flake8.complexity 10 -COMPLEXITY = get_git_param('FLAKE8_COMPLEXITY', 10) -STRICT = get_git_param('FLAKE8_STRICT', False) -IGNORE = get_git_param('FLAKE8_IGNORE', None) -LAZY = get_git_param('FLAKE8_LAZY', False) - -if __name__ == '__main__': - sys.exit(git_hook( - complexity=COMPLEXITY, - strict=STRICT, - ignore=IGNORE, - lazy=LAZY, - )) -""" - - -def _install_hg_hook(path): - getenv = os.environ.get - if not os.path.isfile(path): - # Make the file so we can avoid IOError's - open(path, 'w').close() - - c = ConfigParser() - c.readfp(open(path, 'r')) - if not c.has_section('hooks'): - c.add_section('hooks') - - if not c.has_option('hooks', 'commit'): - c.set('hooks', 'commit', 'python:flake8.hooks.hg_hook') - - if not c.has_option('hooks', 'qrefresh'): - c.set('hooks', 'qrefresh', 'python:flake8.hooks.hg_hook') - - if not c.has_section('flake8'): - c.add_section('flake8') - - if not c.has_option('flake8', 'complexity'): - c.set('flake8', 'complexity', str(getenv('FLAKE8_COMPLEXITY', 10))) - - if not c.has_option('flake8', 'strict'): - c.set('flake8', 'strict', getenv('FLAKE8_STRICT', False)) - - if not c.has_option('flake8', 'ignore'): - c.set('flake8', 'ignore', getenv('FLAKE8_IGNORE', '')) - - if not c.has_option('flake8', 'lazy'): - c.set('flake8', 'lazy', getenv('FLAKE8_LAZY', False)) - - with open(path, 'w') as fd: - c.write(fd) - - -def install_hook(): - vcs = find_vcs() - - if not vcs: - p = get_parser()[0] - sys.stderr.write('Error: could not find either a git or mercurial ' - 'directory. Please re-run this in a proper ' - 'repository.\n') - p.print_help() - sys.exit(1) - - status = 0 - if 'git' in vcs: - if os.path.exists(vcs): - sys.exit('Error: hook already exists (%s)' % vcs) - with open(vcs, 'w') as fd: - fd.write(git_hook_file) - # rwxr--r-- - os.chmod(vcs, stat.S_IRWXU | stat.S_IRGRP | stat.S_IROTH) - elif 'hg' in vcs: - _install_hg_hook(vcs) - else: - status = 1 - - sys.exit(status) diff --git a/old/flake8/main.py b/old/flake8/main.py deleted file mode 100644 index 570b318..0000000 --- a/old/flake8/main.py +++ /dev/null @@ -1,142 +0,0 @@ -# -*- coding: utf-8 -*- -import os -import re -import sys - -import setuptools - -from flake8.engine import get_parser, get_style_guide -from flake8.util import option_normalizer - -if sys.platform.startswith('win'): - DEFAULT_CONFIG = os.path.expanduser(r'~\.flake8') -else: - DEFAULT_CONFIG = os.path.join( - os.getenv('XDG_CONFIG_HOME') or os.path.expanduser('~/.config'), - 'flake8' - ) - -EXTRA_IGNORE = [] - - -def main(): - """Parse options and run checks on Python source.""" - # Prepare - flake8_style = get_style_guide(parse_argv=True, config_file=DEFAULT_CONFIG) - options = flake8_style.options - - if options.install_hook: - from flake8.hooks import install_hook - install_hook() - - # Run the checkers - report = flake8_style.check_files() - - exit_code = print_report(report, flake8_style) - if exit_code > 0: - raise SystemExit(exit_code > 0) - - -def print_report(report, flake8_style): - # Print the final report - options = flake8_style.options - if options.statistics: - report.print_statistics() - if options.benchmark: - report.print_benchmark() - if report.total_errors: - if options.count: - sys.stderr.write(str(report.total_errors) + '\n') - if not options.exit_zero: - return 1 - return 0 - - -def check_file(path, ignore=(), complexity=-1): - """Checks a file using pep8 and pyflakes by default and mccabe - optionally. - - :param str path: path to the file to be checked - :param tuple ignore: (optional), error and warning codes to be ignored - :param int complexity: (optional), enables the mccabe check for values > 0 - """ - ignore = set(ignore).union(EXTRA_IGNORE) - flake8_style = get_style_guide( - config_file=DEFAULT_CONFIG, ignore=ignore, max_complexity=complexity) - return flake8_style.input_file(path) - - -def check_code(code, ignore=(), complexity=-1): - """Checks code using pep8 and pyflakes by default and mccabe optionally. - - :param str code: code to be checked - :param tuple ignore: (optional), error and warning codes to be ignored - :param int complexity: (optional), enables the mccabe check for values > 0 - """ - ignore = set(ignore).union(EXTRA_IGNORE) - flake8_style = get_style_guide( - config_file=DEFAULT_CONFIG, ignore=ignore, max_complexity=complexity) - return flake8_style.input_file(None, lines=code.splitlines(True)) - - -class Flake8Command(setuptools.Command): - """The :class:`Flake8Command` class is used by setuptools to perform - checks on registered modules. - """ - - description = "Run flake8 on modules registered in setuptools" - user_options = [] - - def initialize_options(self): - self.option_to_cmds = {} - parser = get_parser()[0] - for opt in parser.option_list: - cmd_name = opt._long_opts[0][2:] - option_name = cmd_name.replace('-', '_') - self.option_to_cmds[option_name] = opt - setattr(self, option_name, None) - - def finalize_options(self): - self.options_dict = {} - for (option_name, opt) in self.option_to_cmds.items(): - if option_name in ['help', 'verbose']: - continue - value = getattr(self, option_name) - if value is None: - continue - value = option_normalizer(value, opt, option_name) - # Check if there's any values that need to be fixed. - if option_name == "include" and isinstance(value, str): - value = re.findall('[^,;\s]+', value) - - self.options_dict[option_name] = value - - def distribution_files(self): - if self.distribution.packages: - package_dirs = self.distribution.package_dir or {} - for package in self.distribution.packages: - pkg_dir = package - if package in package_dirs: - pkg_dir = package_dirs[package] - elif '' in package_dirs: - pkg_dir = package_dirs[''] + os.path.sep + pkg_dir - yield pkg_dir.replace('.', os.path.sep) - - if self.distribution.py_modules: - for filename in self.distribution.py_modules: - yield "%s.py" % filename - # Don't miss the setup.py file itself - yield "setup.py" - - def run(self): - # Prepare - paths = list(self.distribution_files()) - flake8_style = get_style_guide(config_file=DEFAULT_CONFIG, - paths=paths, - **self.options_dict) - - # Run the checkers - report = flake8_style.check_files() - exit_code = print_report(report, flake8_style) - if exit_code > 0: - raise SystemExit(exit_code > 0) diff --git a/old/flake8/reporter.py b/old/flake8/reporter.py deleted file mode 100644 index 1df3d9e..0000000 --- a/old/flake8/reporter.py +++ /dev/null @@ -1,152 +0,0 @@ -# -*- coding: utf-8 -*- -# Adapted from a contribution of Johan Dahlin - -import collections -import errno -import re -import sys -try: - import multiprocessing -except ImportError: # Python 2.5 - multiprocessing = None - -import pep8 - -__all__ = ['multiprocessing', 'BaseQReport', 'QueueReport'] - - -class BaseQReport(pep8.BaseReport): - """Base Queue Report.""" - _loaded = False # Windows support - - # Reasoning for ignored error numbers is in-line below - ignored_errors = set([ - # EPIPE: Added by sigmavirus24 - # > If output during processing is piped to something that may close - # > its own stdin before we've finished printing results, we need to - # > catch a Broken pipe error and continue on. - # > (See also: https://gitlab.com/pycqa/flake8/issues/69) - errno.EPIPE, - # NOTE(sigmavirus24): When adding to this list, include the reasoning - # on the lines before the error code and always append your error - # code. Further, please always add a trailing `,` to reduce the visual - # noise in diffs. - ]) - - def __init__(self, options): - assert options.jobs > 0 - super(BaseQReport, self).__init__(options) - self.counters = collections.defaultdict(int) - self.n_jobs = options.jobs - - # init queues - self.task_queue = multiprocessing.Queue() - self.result_queue = multiprocessing.Queue() - if sys.platform == 'win32': - # Work around http://bugs.python.org/issue10845 - sys.modules['__main__'].__file__ = __file__ - - def _cleanup_queue(self, queue): - while not queue.empty(): - queue.get_nowait() - - def _put_done(self): - # collect queues - for i in range(self.n_jobs): - self.task_queue.put('DONE') - self.update_state(self.result_queue.get()) - - def _process_main(self): - if not self._loaded: - # Windows needs to parse again the configuration - from flake8.main import get_style_guide, DEFAULT_CONFIG - get_style_guide(parse_argv=True, config_file=DEFAULT_CONFIG) - for filename in iter(self.task_queue.get, 'DONE'): - self.input_file(filename) - - def start(self): - super(BaseQReport, self).start() - self.__class__._loaded = True - # spawn processes - for i in range(self.n_jobs): - p = multiprocessing.Process(target=self.process_main) - p.daemon = True - p.start() - - def stop(self): - try: - self._put_done() - except KeyboardInterrupt: - pass - finally: - # cleanup queues to unlock threads - self._cleanup_queue(self.result_queue) - self._cleanup_queue(self.task_queue) - super(BaseQReport, self).stop() - - def process_main(self): - try: - self._process_main() - except KeyboardInterrupt: - pass - except IOError as ioerr: - # If we happen across an IOError that we aren't certain can/should - # be ignored, we should re-raise the exception. - if ioerr.errno not in self.ignored_errors: - raise - finally: - # ensure all output is flushed before main process continues - sys.stdout.flush() - sys.stderr.flush() - self.result_queue.put(self.get_state()) - - def get_state(self): - return {'total_errors': self.total_errors, - 'counters': self.counters, - 'messages': self.messages} - - def update_state(self, state): - self.total_errors += state['total_errors'] - for key, value in state['counters'].items(): - self.counters[key] += value - self.messages.update(state['messages']) - - -class FileQReport(BaseQReport): - """File Queue Report.""" - print_filename = True - - -class QueueReport(pep8.StandardReport, BaseQReport): - """Standard Queue Report.""" - - def get_file_results(self): - """Print the result and return the overall count for this file.""" - self._deferred_print.sort() - - for line_number, offset, code, text, doc in self._deferred_print: - print(self._fmt % { - 'path': self.filename, - 'row': self.line_offset + line_number, 'col': offset + 1, - 'code': code, 'text': text, - }) - # stdout is block buffered when not stdout.isatty(). - # line can be broken where buffer boundary since other processes - # write to same file. - # flush() after print() to avoid buffer boundary. - # Typical buffer size is 8192. line written safely when - # len(line) < 8192. - sys.stdout.flush() - if self._show_source: - if line_number > len(self.lines): - line = '' - else: - line = self.lines[line_number - 1] - print(line.rstrip()) - sys.stdout.flush() - print(re.sub(r'\S', ' ', line[:offset]) + '^') - sys.stdout.flush() - if self._show_pep8 and doc: - print(' ' + doc.strip()) - sys.stdout.flush() - return self.file_errors diff --git a/old/flake8/run.py b/old/flake8/run.py deleted file mode 100644 index aca929e..0000000 --- a/old/flake8/run.py +++ /dev/null @@ -1,11 +0,0 @@ - -""" -Implementation of the command-line I{flake8} tool. -""" -from flake8.hooks import git_hook, hg_hook # noqa -from flake8.main import check_code, check_file, Flake8Command # noqa -from flake8.main import main - - -if __name__ == '__main__': - main() diff --git a/old/flake8/tests/__init__.py b/old/flake8/tests/__init__.py deleted file mode 100644 index 792d600..0000000 --- a/old/flake8/tests/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# diff --git a/old/flake8/tests/_test_warnings.py b/old/flake8/tests/_test_warnings.py deleted file mode 100644 index 004a597..0000000 --- a/old/flake8/tests/_test_warnings.py +++ /dev/null @@ -1,309 +0,0 @@ -""" - _test_warnings.py - - Tests for the warnings that are emitted by flake8. - - This module is named _test_warnings instead of test_warnings so that a - normal nosetests run does not collect it. The tests in this module pass - when they are run alone, but they fail when they are run along with other - tests (nosetests --with-isolation doesn't help). - - In tox.ini, these tests are run separately. - -""" - -from __future__ import with_statement - -import os -import warnings -import unittest -try: - from unittest import mock -except ImportError: - import mock # < PY33 - -from flake8 import engine -from flake8.util import is_windows - -# The Problem -# ------------ -# -# Some of the tests in this module pass when this module is run on its own, but -# they fail when this module is run as part of the whole test suite. These are -# the problematic tests: -# -# test_jobs_verbose -# test_stdin_jobs_warning -# -# On some platforms, the warnings.capture_warnings function doesn't work -# properly when run with the other flake8 tests. It drops some warnings, even -# though the warnings filter is set to 'always'. However, when run separately, -# these tests pass. -# -# This problem only occurs on Windows, with Python 3.3 and older. Maybe it's -# related to PEP 446 - Inheritable file descriptors? -# -# -# -# -# Things that didn't work -# ------------ -# -# Nose --attr -# I tried using the nosetests --attr feature to run the tests separately. I -# put the following in setup.cfg -# -# [nosetests] -# atttr=!run_alone -# -# Then I added a tox section thst did this -# -# nosetests --attr=run_alone -# -# However, the command line --attr would not override the config file --attr, -# so the special tox section wound up runing all the tests, and failing. -# -# -# -# Nose --with-isolation -# The nosetests --with-isolation flag did not help. -# -# -# -# unittest.skipIf -# I tried decorating the problematic tests with the unittest.skipIf -# decorator. -# -# @unittest.skipIf(is_windows() and sys.version_info < (3, 4), -# "Fails on Windows with Python < 3.4 when run with other" -# " tests.") -# -# The idea is, skip the tests in the main test run, on affected platforms. -# Then, only on those platforms, come back in later and run the tests -# separately. -# -# I added a new stanza to tox.ini, to run the tests separately on the -# affected platforms. -# -# nosetests --no-skip -# -# I ran in to a bug in the nosetests skip plugin. It would report the test as -# having been run, but it would not actually run the test. So, when run with -# --no-skip, the following test would be reported as having run and passed! -# -# @unittest.skip("This passes o_o") -# def test_should_fail(self): -# assert 0 -# -# This bug has been reported here: -# "--no-skip broken with Python 2.7" -# https://github.com/nose-devs/nose/issues/512 -# -# -# -# py.test -# -# I tried using py.test, and its @pytest.mark.xfail decorator. I added some -# separate stanzas in tox, and useing the pytest --runxfail option to run the -# tests separately. This allows us to run all the tests together, on -# platforms that allow it. On platforms that don't allow us to run the tests -# all together, this still runs all the tests, but in two separate steps. -# -# This is the same solution as the nosetests --no-skip solution I described -# above, but --runxfail does not have the same bug as --no-skip. -# -# This has the advantage that all tests are discoverable by default, outside -# of tox. However, nose does not recognize the pytest.mark.xfail decorator. -# So, if a user runs nosetests, it still tries to run the problematic tests -# together with the rest of the test suite, causing them to fail. -# -# -# -# -# -# -# Solution -# ------------ -# Move the problematic tests to _test_warnings.py, so nose.collector will not -# find them. Set up a separate section in tox.ini that runs this: -# -# nosetests flake8.tests._test_warnings -# -# This allows all tests to pass on all platforms, when run through tox. -# However, it means that, even on unaffected platforms, the problematic tests -# are not discovered and run outside of tox (if the user just runs nosetests -# manually, for example). - - -class IntegrationTestCaseWarnings(unittest.TestCase): - """Integration style tests to check that warnings are issued properly for - different command line options.""" - - windows_warning_text = ("The --jobs option is not available on Windows." - " Ignoring --jobs arguments.") - stdin_warning_text = ("The --jobs option is not compatible with" - " supplying input using - . Ignoring --jobs" - " arguments.") - - def this_file(self): - """Return the real path of this file.""" - this_file = os.path.realpath(__file__) - if this_file.endswith("pyc"): - this_file = this_file[:-1] - return this_file - - @staticmethod - def get_style_guide_with_warnings(engine, *args, **kwargs): - """ - Return a style guide object (obtained by calling - engine.get_style_guide) and a list of the warnings that were raised in - the process. - - Note: not threadsafe - """ - - # Note - # https://docs.python.org/2/library/warnings.html - # - # The catch_warnings manager works by replacing and then later - # restoring the module's showwarning() function and internal list of - # filter specifications. This means the context manager is modifying - # global state and therefore is not thread-safe - - with warnings.catch_warnings(record=True) as collected_warnings: - # Cause all warnings to always be triggered. - warnings.simplefilter("always") - - # Get the style guide - style_guide = engine.get_style_guide(*args, **kwargs) - - # Now that the warnings have been collected, return the style guide and - # the warnings. - return (style_guide, collected_warnings) - - def verify_warnings(self, collected_warnings, expected_warnings): - """ - Verifies that collected_warnings is a sequence that contains user - warnings that match the sequence of string values passed in as - expected_warnings. - """ - if expected_warnings is None: - expected_warnings = [] - - collected_user_warnings = [w for w in collected_warnings - if issubclass(w.category, UserWarning)] - - self.assertEqual(len(collected_user_warnings), - len(expected_warnings)) - - collected_warnings_set = set(str(warning.message) - for warning - in collected_user_warnings) - expected_warnings_set = set(expected_warnings) - self.assertEqual(collected_warnings_set, expected_warnings_set) - - def check_files_collect_warnings(self, - arglist=[], - explicit_stdin=False, - count=0, - verbose=False): - """Call check_files and collect any warnings that are issued.""" - if verbose: - arglist.append('--verbose') - if explicit_stdin: - target_file = "-" - else: - target_file = self.this_file() - argv = ['flake8'] + arglist + [target_file] - with mock.patch("sys.argv", argv): - (style_guide, - collected_warnings, - ) = self.get_style_guide_with_warnings(engine, - parse_argv=True) - report = style_guide.check_files() - self.assertEqual(report.total_errors, count) - return style_guide, report, collected_warnings - - def check_files_no_warnings_allowed(self, - arglist=[], - explicit_stdin=False, - count=0, - verbose=False): - """Call check_files, and assert that there were no warnings issued.""" - (style_guide, - report, - collected_warnings, - ) = self.check_files_collect_warnings(arglist=arglist, - explicit_stdin=explicit_stdin, - count=count, - verbose=verbose) - self.verify_warnings(collected_warnings, expected_warnings=None) - return style_guide, report - - def _job_tester(self, jobs, verbose=False): - # mock stdout.flush so we can count the number of jobs created - with mock.patch('sys.stdout.flush') as mocked: - (guide, - report, - collected_warnings, - ) = self.check_files_collect_warnings( - arglist=['--jobs=%s' % jobs], - verbose=verbose) - - if is_windows(): - # The code path where guide.options.jobs gets converted to an - # int is not run on windows. So, do the int conversion here. - self.assertEqual(int(guide.options.jobs), jobs) - # On windows, call count is always zero. - self.assertEqual(mocked.call_count, 0) - else: - self.assertEqual(guide.options.jobs, jobs) - self.assertEqual(mocked.call_count, jobs) - - expected_warings = [] - if verbose and is_windows(): - expected_warings.append(self.windows_warning_text) - self.verify_warnings(collected_warnings, expected_warings) - - def test_jobs(self, verbose=False): - self._job_tester(2, verbose=verbose) - self._job_tester(10, verbose=verbose) - - def test_no_args_no_warnings(self, verbose=False): - self.check_files_no_warnings_allowed(verbose=verbose) - - def test_stdin_jobs_warning(self, verbose=False): - self.count = 0 - - def fake_stdin(): - self.count += 1 - with open(self.this_file(), "r") as f: - return f.read() - - with mock.patch("pep8.stdin_get_value", fake_stdin): - (style_guide, - report, - collected_warnings, - ) = self.check_files_collect_warnings(arglist=['--jobs=4'], - explicit_stdin=True, - verbose=verbose) - expected_warings = [] - if verbose: - expected_warings.append(self.stdin_warning_text) - if is_windows(): - expected_warings.append(self.windows_warning_text) - self.verify_warnings(collected_warnings, expected_warings) - self.assertEqual(self.count, 1) - - def test_jobs_verbose(self): - self.test_jobs(verbose=True) - - def test_no_args_no_warnings_verbose(self): - self.test_no_args_no_warnings(verbose=True) - - def test_stdin_jobs_warning_verbose(self): - self.test_stdin_jobs_warning(verbose=True) - - -if __name__ == '__main__': - unittest.main() diff --git a/old/flake8/tests/test_engine.py b/old/flake8/tests/test_engine.py deleted file mode 100644 index 82ba0dd..0000000 --- a/old/flake8/tests/test_engine.py +++ /dev/null @@ -1,236 +0,0 @@ -from __future__ import with_statement - -import errno -import unittest -try: - from unittest import mock -except ImportError: - import mock # < PY33 - -from flake8 import engine, util, __version__, reporter -import pep8 - - -class TestEngine(unittest.TestCase): - def setUp(self): - self.patches = {} - - def tearDown(self): - assert len(self.patches.items()) == 0 - - def start_patch(self, patch): - self.patches[patch] = mock.patch(patch) - return self.patches[patch].start() - - def stop_patches(self): - patches = self.patches.copy() - for k, v in patches.items(): - v.stop() - del(self.patches[k]) - - def test_get_style_guide(self): - with mock.patch('flake8.engine._register_extensions') as reg_ext: - reg_ext.return_value = ([], [], [], []) - g = engine.get_style_guide() - self.assertTrue(isinstance(g, engine.StyleGuide)) - reg_ext.assert_called_once_with() - - def test_get_style_guide_kwargs(self): - m = mock.Mock() - with mock.patch('flake8.engine.StyleGuide') as StyleGuide: - with mock.patch('flake8.engine.get_parser') as get_parser: - m.ignored_extensions = [] - StyleGuide.return_value.options.jobs = '42' - StyleGuide.return_value.options.diff = False - get_parser.return_value = (m, []) - engine.get_style_guide(foo='bar') - get_parser.assert_called_once_with() - StyleGuide.assert_called_once_with(**{'parser': m, 'foo': 'bar'}) - - def test_register_extensions(self): - with mock.patch('pep8.register_check') as register_check: - registered_exts = engine._register_extensions() - self.assertTrue(isinstance(registered_exts[0], util.OrderedSet)) - self.assertTrue(len(registered_exts[0]) > 0) - for i in registered_exts[1:]: - self.assertTrue(isinstance(i, list)) - self.assertTrue(register_check.called) - - def test_disable_extensions(self): - parser = mock.MagicMock() - options = mock.MagicMock() - - parser.ignored_extensions = ['I123', 'I345', 'I678', 'I910'] - - options.enable_extensions = 'I345,\nI678,I910' - options.ignore = ('E121', 'E123') - - engine._disable_extensions(parser, options) - self.assertEqual(set(options.ignore), set(['E121', 'E123', 'I123'])) - - def test_get_parser(self): - # setup - re = self.start_patch('flake8.engine._register_extensions') - gpv = self.start_patch('flake8.engine.get_python_version') - pgp = self.start_patch('pep8.get_parser') - m = mock.Mock() - re.return_value = ([('pyflakes', '0.7'), ('mccabe', '0.2')], [], [], - []) - gpv.return_value = 'Python Version' - pgp.return_value = m - # actual call we're testing - parser, hooks = engine.get_parser() - # assertions - self.assertTrue(re.called) - self.assertTrue(gpv.called) - pgp.assert_called_once_with( - 'flake8', - '%s (pyflakes: 0.7, mccabe: 0.2) Python Version' % __version__) - self.assertTrue(m.remove_option.called) - self.assertTrue(m.add_option.called) - self.assertEqual(parser, m) - self.assertEqual(hooks, []) - # clean-up - self.stop_patches() - - def test_get_python_version(self): - self.assertTrue('on' in engine.get_python_version()) - # Silly test but it will provide 100% test coverage - # Also we can never be sure (without reconstructing the string - # ourselves) what system we may be testing on. - - def test_windows_disables_jobs(self): - with mock.patch('flake8.util.is_windows') as is_windows: - is_windows.return_value = True - guide = engine.get_style_guide() - assert isinstance(guide, reporter.BaseQReport) is False - - def test_stdin_disables_jobs(self): - with mock.patch('flake8.util.is_using_stdin') as is_using_stdin: - is_using_stdin.return_value = True - guide = engine.get_style_guide() - assert isinstance(guide, reporter.BaseQReport) is False - - def test_disables_extensions_that_are_not_selected(self): - with mock.patch('flake8.engine._register_extensions') as re: - re.return_value = ([('fake_ext', '0.1a1')], [], [], ['X']) - sg = engine.get_style_guide() - assert 'X' in sg.options.ignore - - def test_enables_off_by_default_extensions(self): - with mock.patch('flake8.engine._register_extensions') as re: - re.return_value = ([('fake_ext', '0.1a1')], [], [], ['X']) - parser, options = engine.get_parser() - parser.parse_args(['--select=X']) - sg = engine.StyleGuide(parser=parser) - assert 'X' not in sg.options.ignore - - def test_load_entry_point_verifies_requirements(self): - entry_point = mock.Mock(spec=['require', 'resolve', 'load']) - - engine._load_entry_point(entry_point, verify_requirements=True) - entry_point.require.assert_called_once_with() - entry_point.resolve.assert_called_once_with() - - def test_load_entry_point_does_not_verify_requirements(self): - entry_point = mock.Mock(spec=['require', 'resolve', 'load']) - - engine._load_entry_point(entry_point, verify_requirements=False) - self.assertFalse(entry_point.require.called) - entry_point.resolve.assert_called_once_with() - - def test_load_entry_point_passes_require_argument_to_load(self): - entry_point = mock.Mock(spec=['load']) - - engine._load_entry_point(entry_point, verify_requirements=True) - entry_point.load.assert_called_once_with(require=True) - entry_point.reset_mock() - - engine._load_entry_point(entry_point, verify_requirements=False) - entry_point.load.assert_called_once_with(require=False) - - -def oserror_generator(error_number, message='Ominous OSError message'): - def oserror_side_effect(*args, **kwargs): - if hasattr(oserror_side_effect, 'used'): - return - - oserror_side_effect.used = True - raise OSError(error_number, message) - - return oserror_side_effect - - -class TestStyleGuide(unittest.TestCase): - def setUp(self): - mocked_styleguide = mock.Mock(spec=engine.NoQAStyleGuide) - self.styleguide = engine.StyleGuide(styleguide=mocked_styleguide) - self.mocked_sg = mocked_styleguide - - def test_proxies_excluded(self): - self.styleguide.excluded('file.py', parent='.') - - self.mocked_sg.excluded.assert_called_once_with('file.py', parent='.') - - def test_proxies_init_report(self): - reporter = object() - self.styleguide.init_report(reporter) - - self.mocked_sg.init_report.assert_called_once_with(reporter) - - def test_proxies_check_files(self): - self.styleguide.check_files(['foo', 'bar']) - - self.mocked_sg.check_files.assert_called_once_with( - paths=['foo', 'bar'] - ) - - def test_proxies_input_file(self): - self.styleguide.input_file('file.py', - lines=[9, 10], - expected='foo', - line_offset=20) - - self.mocked_sg.input_file.assert_called_once_with(filename='file.py', - lines=[9, 10], - expected='foo', - line_offset=20) - - def test_check_files_retries_on_specific_OSErrors(self): - self.mocked_sg.check_files.side_effect = oserror_generator( - errno.ENOSPC, 'No space left on device' - ) - - self.styleguide.check_files(['foo', 'bar']) - - self.mocked_sg.init_report.assert_called_once_with(pep8.StandardReport) - - def test_input_file_retries_on_specific_OSErrors(self): - self.mocked_sg.input_file.side_effect = oserror_generator( - errno.ENOSPC, 'No space left on device' - ) - - self.styleguide.input_file('file.py') - - self.mocked_sg.init_report.assert_called_once_with(pep8.StandardReport) - - def test_check_files_reraises_unknown_OSErrors(self): - self.mocked_sg.check_files.side_effect = oserror_generator( - errno.EADDRINUSE, - 'lol why are we talking about binding to sockets' - ) - - self.assertRaises(OSError, self.styleguide.check_files, - ['foo', 'bar']) - - def test_input_file_reraises_unknown_OSErrors(self): - self.mocked_sg.input_file.side_effect = oserror_generator( - errno.EADDRINUSE, - 'lol why are we talking about binding to sockets' - ) - - self.assertRaises(OSError, self.styleguide.input_file, - ['foo', 'bar']) - -if __name__ == '__main__': - unittest.main() diff --git a/old/flake8/tests/test_hooks.py b/old/flake8/tests/test_hooks.py deleted file mode 100644 index ba46794..0000000 --- a/old/flake8/tests/test_hooks.py +++ /dev/null @@ -1,59 +0,0 @@ -"""Module containing the tests for flake8.hooks.""" -import os -import unittest - -try: - from unittest import mock -except ImportError: - import mock - -import flake8.hooks -from flake8.util import is_windows - - -def excluded(filename): - return filename.endswith('afile.py') - - -class TestGitHook(unittest.TestCase): - if is_windows: - # On Windows, absolute paths start with a drive letter, for example C: - # Here we build a fake absolute path starting with the current drive - # letter, for example C:\fake\temp - current_drive, ignore_tail = os.path.splitdrive(os.getcwd()) - fake_abs_path = os.path.join(current_drive, os.path.sep, 'fake', 'tmp') - else: - fake_abs_path = os.path.join(os.path.sep, 'fake', 'tmp') - - @mock.patch('os.makedirs') - @mock.patch('flake8.hooks.open', create=True) - @mock.patch('shutil.rmtree') - @mock.patch('tempfile.mkdtemp', return_value=fake_abs_path) - @mock.patch('flake8.hooks.run', - return_value=(None, - [os.path.join('foo', 'afile.py'), - os.path.join('foo', 'bfile.py')], - None)) - @mock.patch('flake8.hooks.get_style_guide') - def test_prepends_tmp_directory_to_exclude(self, get_style_guide, run, - *args): - style_guide = get_style_guide.return_value = mock.Mock() - style_guide.options.exclude = [os.path.join('foo', 'afile.py')] - style_guide.options.filename = [os.path.join('foo', '*')] - style_guide.excluded = excluded - - flake8.hooks.git_hook() - - dirname, filename = os.path.split( - os.path.abspath(os.path.join('foo', 'bfile.py'))) - if is_windows: - # In Windows, the absolute path in dirname will start with a drive - # letter. Here, we discad the drive letter. - ignore_drive, dirname = os.path.splitdrive(dirname) - tmpdir = os.path.join(self.fake_abs_path, dirname[1:]) - tmpfile = os.path.join(tmpdir, 'bfile.py') - style_guide.check_files.assert_called_once_with([tmpfile]) - - -if __name__ == '__main__': - unittest.main() diff --git a/old/flake8/tests/test_integration.py b/old/flake8/tests/test_integration.py deleted file mode 100644 index d1417c6..0000000 --- a/old/flake8/tests/test_integration.py +++ /dev/null @@ -1,79 +0,0 @@ -from __future__ import with_statement - -import os -import unittest -try: - from unittest import mock -except ImportError: - import mock # < PY33 - -from flake8 import engine -from flake8.util import is_windows - - -class IntegrationTestCase(unittest.TestCase): - """Integration style tests to exercise different command line options.""" - - def this_file(self): - """Return the real path of this file.""" - this_file = os.path.realpath(__file__) - if this_file.endswith("pyc"): - this_file = this_file[:-1] - return this_file - - def check_files(self, arglist=[], explicit_stdin=False, count=0): - """Call check_files.""" - if explicit_stdin: - target_file = "-" - else: - target_file = self.this_file() - argv = ['flake8'] + arglist + [target_file] - with mock.patch("sys.argv", argv): - style_guide = engine.get_style_guide(parse_argv=True) - report = style_guide.check_files() - self.assertEqual(report.total_errors, count) - return style_guide, report - - def test_no_args(self): - # assert there are no reported errors - self.check_files() - - def _job_tester(self, jobs): - # mock stdout.flush so we can count the number of jobs created - with mock.patch('sys.stdout.flush') as mocked: - guide, report = self.check_files(arglist=['--jobs=%s' % jobs]) - if is_windows(): - # The code path where guide.options.jobs gets converted to an - # int is not run on windows. So, do the int conversion here. - self.assertEqual(int(guide.options.jobs), jobs) - # On windows, call count is always zero. - self.assertEqual(mocked.call_count, 0) - else: - self.assertEqual(guide.options.jobs, jobs) - self.assertEqual(mocked.call_count, jobs) - - def test_jobs(self): - self._job_tester(2) - self._job_tester(10) - - def test_stdin(self): - self.count = 0 - - def fake_stdin(): - self.count += 1 - with open(self.this_file(), "r") as f: - return f.read() - - with mock.patch("pep8.stdin_get_value", fake_stdin): - guide, report = self.check_files(arglist=['--jobs=4'], - explicit_stdin=True) - self.assertEqual(self.count, 1) - - def test_stdin_fail(self): - def fake_stdin(): - return "notathing\n" - with mock.patch("pep8.stdin_get_value", fake_stdin): - # only assert needed is in check_files - guide, report = self.check_files(arglist=['--jobs=4'], - explicit_stdin=True, - count=1) diff --git a/old/flake8/tests/test_main.py b/old/flake8/tests/test_main.py deleted file mode 100644 index af08093..0000000 --- a/old/flake8/tests/test_main.py +++ /dev/null @@ -1,18 +0,0 @@ -from __future__ import with_statement - -import unittest - -import setuptools -from flake8 import main - - -class TestMain(unittest.TestCase): - def test_issue_39_regression(self): - distribution = setuptools.Distribution() - cmd = main.Flake8Command(distribution) - cmd.options_dict = {} - cmd.run() - - -if __name__ == '__main__': - unittest.main() diff --git a/old/flake8/tests/test_pyflakes.py b/old/flake8/tests/test_pyflakes.py deleted file mode 100644 index fb2f042..0000000 --- a/old/flake8/tests/test_pyflakes.py +++ /dev/null @@ -1,73 +0,0 @@ -from __future__ import with_statement - -import ast -import unittest - -from collections import namedtuple - -from flake8._pyflakes import FlakesChecker - -Options = namedtuple("Options", ['builtins', 'doctests', - 'include_in_doctest', - 'exclude_from_doctest']) - - -class TestFlakesChecker(unittest.TestCase): - - def setUp(self): - self.tree = ast.parse('print("cookies")') - - def test_doctest_flag_enabled(self): - options = Options(builtins=None, doctests=True, - include_in_doctest='', - exclude_from_doctest='') - FlakesChecker.parse_options(options) - flake_checker = FlakesChecker(self.tree, 'cookies.txt') - assert flake_checker.withDoctest is True - - def test_doctest_flag_disabled(self): - options = Options(builtins=None, doctests=False, - include_in_doctest='', - exclude_from_doctest='') - FlakesChecker.parse_options(options) - flake_checker = FlakesChecker(self.tree, 'cookies.txt') - assert flake_checker.withDoctest is False - - def test_doctest_flag_enabled_exclude_file(self): - options = Options(builtins=None, doctests=True, - include_in_doctest='', - exclude_from_doctest='cookies.txt,' - 'hungry/cookies.txt') - FlakesChecker.parse_options(options) - flake_checker = FlakesChecker(self.tree, './cookies.txt') - assert flake_checker.withDoctest is False - - def test_doctest_flag_disabled_include_file(self): - options = Options(builtins=None, doctests=False, - include_in_doctest='./cookies.txt,cake_yuck.txt', - exclude_from_doctest='') - FlakesChecker.parse_options(options) - flake_checker = FlakesChecker(self.tree, './cookies.txt') - assert flake_checker.withDoctest is True - - def test_doctest_flag_disabled_include_file_exclude_dir(self): - options = Options(builtins=None, doctests=False, - include_in_doctest='./cookies.txt', - exclude_from_doctest='./') - FlakesChecker.parse_options(options) - flake_checker = FlakesChecker(self.tree, './cookies.txt') - assert flake_checker.withDoctest is True - - def test_doctest_flag_disabled_include_dir_exclude_file(self): - options = Options(builtins=None, doctests=False, - include_in_doctest='./', - exclude_from_doctest='./cookies.txt') - FlakesChecker.parse_options(options) - flake_checker = FlakesChecker(self.tree, './cookies.txt') - assert flake_checker.withDoctest is False - - def test_doctest_flag_disabled_include_file_exclude_file_error(self): - options = Options(builtins=None, doctests=False, - include_in_doctest='./cookies.txt', - exclude_from_doctest='./cookies.txt,cake_yuck.txt') - self.assertRaises(ValueError, FlakesChecker.parse_options, options) diff --git a/old/flake8/tests/test_reporter.py b/old/flake8/tests/test_reporter.py deleted file mode 100644 index f91bb52..0000000 --- a/old/flake8/tests/test_reporter.py +++ /dev/null @@ -1,36 +0,0 @@ -from __future__ import with_statement - -import errno -import unittest -try: - from unittest import mock -except ImportError: - import mock # < PY33 - -from flake8 import reporter - - -def ioerror_report_factory(errno_code): - class IOErrorBaseQReport(reporter.BaseQReport): - def _process_main(self): - raise IOError(errno_code, 'Fake bad pipe exception') - - options = mock.MagicMock() - options.jobs = 2 - return IOErrorBaseQReport(options) - - -class TestBaseQReport(unittest.TestCase): - def test_does_not_raise_a_bad_pipe_ioerror(self): - """Test that no EPIPE IOError exception is re-raised or leaked.""" - report = ioerror_report_factory(errno.EPIPE) - try: - report.process_main() - except IOError: - self.fail('BaseQReport.process_main raised an IOError for EPIPE' - ' but it should have caught this exception.') - - def test_raises_a_enoent_ioerror(self): - """Test that an ENOENT IOError exception is re-raised.""" - report = ioerror_report_factory(errno.ENOENT) - self.assertRaises(IOError, report.process_main) diff --git a/old/flake8/tests/test_util.py b/old/flake8/tests/test_util.py deleted file mode 100644 index bfbc660..0000000 --- a/old/flake8/tests/test_util.py +++ /dev/null @@ -1,120 +0,0 @@ -import optparse -import unittest - -from flake8.util import option_normalizer - - -class TestOptionSerializerParsesTrue(unittest.TestCase): - - def setUp(self): - self.option = optparse.Option('--foo', action='store_true') - self.option_name = 'fake_option' - - def test_1_is_true(self): - value = option_normalizer('1', self.option, self.option_name) - self.assertTrue(value) - - def test_T_is_true(self): - value = option_normalizer('T', self.option, self.option_name) - self.assertTrue(value) - - def test_TRUE_is_true(self): - value = option_normalizer('TRUE', self.option, self.option_name) - self.assertTrue(value, True) - - def test_ON_is_true(self): - value = option_normalizer('ON', self.option, self.option_name) - self.assertTrue(value) - - def test_t_is_true(self): - value = option_normalizer('t', self.option, self.option_name) - self.assertTrue(value) - - def test_true_is_true(self): - value = option_normalizer('true', self.option, self.option_name) - self.assertTrue(value) - - def test_on_is_true(self): - value = option_normalizer('on', self.option, self.option_name) - self.assertTrue(value) - - -class TestOptionSerializerParsesFalse(unittest.TestCase): - - def setUp(self): - self.option = optparse.Option('--foo', action='store_true') - self.option_name = 'fake_option' - - def test_0_is_false(self): - value = option_normalizer('0', self.option, self.option_name) - self.assertFalse(value) - - def test_F_is_false(self): - value = option_normalizer('F', self.option, self.option_name) - self.assertFalse(value) - - def test_FALSE_is_false(self): - value = option_normalizer('FALSE', self.option, self.option_name) - self.assertFalse(value) - - def test_OFF_is_false(self): - value = option_normalizer('OFF', self.option, self.option_name) - self.assertFalse(value) - - def test_f_is_false(self): - value = option_normalizer('f', self.option, self.option_name) - self.assertFalse(value) - - def test_false_is_false(self): - value = option_normalizer('false', self.option, self.option_name) - self.assertFalse(value) - - def test_off_is_false(self): - value = option_normalizer('off', self.option, self.option_name) - self.assertFalse(value) - - -class TestOptionSerializerParsesLists(unittest.TestCase): - - def setUp(self): - self.option = optparse.Option('--select') - self.option_name = 'select' - self.answer = ['F401', 'F402', 'F403', 'F404'] - - def test_parses_simple_comma_separated_lists(self): - value = option_normalizer('F401,F402,F403,F404', self.option, - self.option_name) - self.assertEqual(value, self.answer) - - def test_parses_less_simple_comma_separated_lists(self): - value = option_normalizer('F401 ,F402 ,F403 ,F404', self.option, - self.option_name) - self.assertEqual(value, self.answer) - - value = option_normalizer('F401, F402, F403, F404', self.option, - self.option_name) - self.assertEqual(value, self.answer) - - def test_parses_comma_separated_lists_with_newlines(self): - value = option_normalizer('''\ - F401, - F402, - F403, - F404, - ''', self.option, self.option_name) - self.assertEqual(value, self.answer) - - -class TestOptionSerializerParsesInts(unittest.TestCase): - - def setUp(self): - self.option = optparse.Option('--max-complexity', type='int') - self.option_name = 'max_complexity' - - def test_parses_an_int(self): - value = option_normalizer('2', self.option, self.option_name) - self.assertEqual(value, 2) - - -if __name__ == '__main__': - unittest.main() diff --git a/old/flake8/util.py b/old/flake8/util.py deleted file mode 100644 index 4e2c0d3..0000000 --- a/old/flake8/util.py +++ /dev/null @@ -1,77 +0,0 @@ -# -*- coding: utf-8 -*- -import os - -try: - import ast - iter_child_nodes = ast.iter_child_nodes -except ImportError: # Python 2.5 - import _ast as ast - - if 'decorator_list' not in ast.ClassDef._fields: - # Patch the missing attribute 'decorator_list' - ast.ClassDef.decorator_list = () - ast.FunctionDef.decorator_list = property(lambda s: s.decorators) - - def iter_child_nodes(node): - """ - Yield all direct child nodes of *node*, that is, all fields that - are nodes and all items of fields that are lists of nodes. - """ - if not node._fields: - return - for name in node._fields: - field = getattr(node, name, None) - if isinstance(field, ast.AST): - yield field - elif isinstance(field, list): - for item in field: - if isinstance(item, ast.AST): - yield item - - -class OrderedSet(list): - """List without duplicates.""" - __slots__ = () - - def add(self, value): - if value not in self: - self.append(value) - - -def is_windows(): - """Determine if the system is Windows.""" - return os.name == 'nt' - - -def is_using_stdin(paths): - """Determine if we're running checks on stdin.""" - return '-' in paths - - -def warn_when_using_jobs(options): - return (options.verbose and options.jobs and options.jobs.isdigit() and - int(options.jobs) > 1) - - -def force_disable_jobs(styleguide): - return is_windows() or is_using_stdin(styleguide.paths) - - -INT_TYPES = ('int', 'count') -BOOL_TYPES = ('store_true', 'store_false') -LIST_OPTIONS = ('select', 'ignore', 'exclude', 'enable_extensions') - - -def option_normalizer(value, option, option_name): - if option.action in BOOL_TYPES: - if str(value).upper() in ('1', 'T', 'TRUE', 'ON'): - value = True - if str(value).upper() in ('0', 'F', 'FALSE', 'OFF'): - value = False - elif option.type in INT_TYPES: - value = int(value) - elif option_name in LIST_OPTIONS: - if isinstance(value, str): - value = [opt.strip() for opt in value.split(',') if opt.strip()] - - return value diff --git a/old/setup.py b/old/setup.py deleted file mode 100644 index fab5e86..0000000 --- a/old/setup.py +++ /dev/null @@ -1,75 +0,0 @@ -# -*- coding: utf-8 -*- -from __future__ import with_statement -from setuptools import setup -try: - # Work around a traceback with Nose on Python 2.6 - # http://bugs.python.org/issue15881#msg170215 - __import__('multiprocessing') -except ImportError: - pass - -try: - # Use https://docs.python.org/3/library/unittest.mock.html - from unittest import mock -except ImportError: - # < Python 3.3 - mock = None - - -tests_require = ['nose'] -if mock is None: - tests_require += ['mock'] - - -def get_version(fname='flake8/__init__.py'): - with open(fname) as f: - for line in f: - if line.startswith('__version__'): - return eval(line.split('=')[-1]) - - -def get_long_description(): - descr = [] - for fname in ('README.rst', 'CHANGES.rst'): - with open(fname) as f: - descr.append(f.read()) - return '\n\n'.join(descr) - - -setup( - name="flake8", - license="MIT", - version=get_version(), - description="the modular source code checker: pep8, pyflakes and co", - long_description=get_long_description(), - author="Tarek Ziade", - author_email="tarek@ziade.org", - maintainer="Ian Cordasco", - maintainer_email="graffatcolmingov@gmail.com", - url="https://gitlab.com/pycqa/flake8", - packages=["flake8", "flake8.tests"], - install_requires=[ - "pyflakes >= 0.8.1, < 1.1", - "pep8 >= 1.5.7, != 1.6.0, != 1.6.1, != 1.6.2", - "mccabe >= 0.2.1, < 0.5", - ], - entry_points={ - 'distutils.commands': ['flake8 = flake8.main:Flake8Command'], - 'console_scripts': ['flake8 = flake8.main:main'], - 'flake8.extension': [ - 'F = flake8._pyflakes:FlakesChecker', - ], - }, - classifiers=[ - "Environment :: Console", - "Intended Audience :: Developers", - "License :: OSI Approved :: MIT License", - "Programming Language :: Python", - "Programming Language :: Python :: 2", - "Programming Language :: Python :: 3", - "Topic :: Software Development :: Libraries :: Python Modules", - "Topic :: Software Development :: Quality Assurance", - ], - tests_require=tests_require, - test_suite='nose.collector', -) diff --git a/old/tox.ini b/old/tox.ini deleted file mode 100644 index 14e2a44..0000000 --- a/old/tox.ini +++ /dev/null @@ -1,40 +0,0 @@ -[tox] -minversion = 1.6 -envlist = - py26,py27,py33,py34,py27-flake8,py34-flake8 - -[testenv] -usedevelop = True -deps = - mock - nose -commands = - python setup.py test -q - python setup.py flake8 - nosetests flake8.tests._test_warnings - -[testenv:py27-flake8] -basepython = python2.7 -deps = - flake8 -commands = flake8 {posargs} flake8/ - -[testenv:py34-flake8] -basepython = python3.4 -deps = - flake8 -commands = flake8 {posargs} flake8/ - -[testenv:release] -basepython = python2.7 -deps = - twine >= 1.5.0 - wheel -commands = - python setup.py sdist bdist_wheel - twine upload --skip-existing {posargs} dist/* - -[flake8] -select = E,F,W -max_line_length = 79 -exclude = .git,.tox,dist,docs,*egg From 601696d5641fea75709f5f8b851c347a9ee0b7cd Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 10:07:16 -0500 Subject: [PATCH 350/364] Trim whitespace from cross-compat plugin docs --- docs/source/dev/cross_compatibility.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/dev/cross_compatibility.rst b/docs/source/dev/cross_compatibility.rst index c546f03..2efadf3 100644 --- a/docs/source/dev/cross_compatibility.rst +++ b/docs/source/dev/cross_compatibility.rst @@ -12,7 +12,7 @@ versions. If your plugin does not register options, it *should* Just Work. -The **only** breaking change in |Flake8| 3.0 is the fact that we no longer +The **only** breaking change in |Flake8| 3.0 is the fact that we no longer check the option parser for a list of strings to parse from a config file. On |Flake8| 2.x, to have an option parsed from the configuration files that |Flake8| finds and parses you would have to do something like: @@ -121,7 +121,7 @@ options with |Flake8| and have it work on |Flake8| 2.x and 3.x. parser.add_option(*option_args, **option_kwargs) if parse_from_config: parser.config_options.append(option_args[-1].lstrip('-')) - + Or, you can write a tiny helper function: From 88d1dd280a7b3d2833d8073a1a9d9246e27d19db Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 14:34:14 -0500 Subject: [PATCH 351/364] Start a glossary of terms --- docs/source/glossary.rst | 29 +++++++++++++++++++++++++++++ docs/source/index.rst | 6 ++++-- 2 files changed, 33 insertions(+), 2 deletions(-) create mode 100644 docs/source/glossary.rst diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst new file mode 100644 index 0000000..422da5e --- /dev/null +++ b/docs/source/glossary.rst @@ -0,0 +1,29 @@ +.. _glossary: + +================================================ + Glossary of Terms Used in Flake8 Documentation +================================================ + +.. glossary:: + :sorted: + + formatter + A :term:`plugin` that augments the output of |Flake8| when passed + to :option:`flake8 --format`. + + plugin + A package that is typically installed from PyPI to augment the + behaviour of |Flake8| either through adding one or more additional + :term:`check`\ s or providing additional :term:`formatter`\ s. + + check + A piece of logic that corresponds to an error code. A check may + be a style check (e.g., check the length of a given line against + the user configured maximum) or a lint check (e.g., checking for + unused imports) or some other check as defined by a plugin. + + error code + The symbol associated with a specific :term:`check`. For example, + pycodestyle implements :term:`check`\ s that look for whitespace + around binary operators and will either return an error code of + W503 or W504. diff --git a/docs/source/index.rst b/docs/source/index.rst index 55dfe3d..9fb02c0 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -69,13 +69,14 @@ Alternatively, if you want to *ignore* only one specific warning or error: Please read our user guide for more information about how to use and configure |Flake8|. -FAQ -=== +FAQ and Glossary +================ .. toctree:: :maxdepth: 2 faq + glossary User Guide ========== @@ -127,3 +128,4 @@ General Indices * :ref:`genindex` * :ref:`Index of Documented Public Modules ` +* :ref:`Glossary of terms ` From 478d5620d8dbfb085b1f8c8ae3631b3bd3c9af35 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 19:13:20 -0500 Subject: [PATCH 352/364] Correct example register_options method --- docs/source/dev/cross_compatibility.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/dev/cross_compatibility.rst b/docs/source/dev/cross_compatibility.rst index 2efadf3..1aa45e3 100644 --- a/docs/source/dev/cross_compatibility.rst +++ b/docs/source/dev/cross_compatibility.rst @@ -140,7 +140,8 @@ Or, you can write a tiny helper function: .. code-block:: python - def register_options(self, parser): + @classmethod + def register_options(cls, parser): register_opt(parser, '-X', '--example-flag', type='string', parse_from_config=True, help='...') From 1de5da8a7fc7c49ba44e0a7273f8d8ffa4d3a529 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 19:27:10 -0500 Subject: [PATCH 353/364] Add more definitions to glossary --- docs/source/glossary.rst | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst index 422da5e..f3231fc 100644 --- a/docs/source/glossary.rst +++ b/docs/source/glossary.rst @@ -22,8 +22,25 @@ the user configured maximum) or a lint check (e.g., checking for unused imports) or some other check as defined by a plugin. + error error code The symbol associated with a specific :term:`check`. For example, pycodestyle implements :term:`check`\ s that look for whitespace around binary operators and will either return an error code of - W503 or W504. + ``W503`` or ``W504``. + + warning + Typically the ``W`` class of :term:`error code`\ s from pycodestyle. + + class + error class + A larger grouping of related :term:`error code`\ s. For example, + ``W503`` and ``W504`` are two codes related to whitespace. ``W50`` + would be the most specific class of codes relating to whitespace. + ``W`` would be the warning class that subsumes all whitespace + errors. + + pyflakes + The project |Flake8| depends upon to lint files (check for unused + imports, variables, etc.). This uses the ``F`` :term:`class` of + :term:`error code`\ s reported by |Flake8|. From cae943bd446953aefe4facfa44d7f1f54da19d43 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Wed, 22 Jun 2016 19:27:23 -0500 Subject: [PATCH 354/364] Add missing - to sub-directories --- docs/source/user/invocation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/user/invocation.rst b/docs/source/user/invocation.rst index 5c92ca3..383e93a 100644 --- a/docs/source/user/invocation.rst +++ b/docs/source/user/invocation.rst @@ -55,7 +55,7 @@ Or It's also possible to narrow what |Flake8| will try to check by specifying exactly the paths and directories you want it to check. Let's assume that -we have a directory with python files and subdirectories which have python +we have a directory with python files and sub-directories which have python files (and may have more sub-directories) called ``my_project``. Then if we only want errors from files found inside ``my_project`` we can do: From c2111656a6776d2f85164d0a8e609ae6271c3054 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Fri, 24 Jun 2016 14:54:48 -0500 Subject: [PATCH 355/364] Add descriptions of pycodestyle and mccable --- docs/source/glossary.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst index f3231fc..dd691b6 100644 --- a/docs/source/glossary.rst +++ b/docs/source/glossary.rst @@ -41,6 +41,16 @@ errors. pyflakes - The project |Flake8| depends upon to lint files (check for unused + The project |Flake8| depends on to lint files (check for unused imports, variables, etc.). This uses the ``F`` :term:`class` of :term:`error code`\ s reported by |Flake8|. + + pycodestyle + The project |Flake8| depends on to provide style enforcement. + pycodestyle implements :term:`check`\ s for :pep:`8`. This uses the + ``E`` and ``W`` :term:`class`\ es of :term:`error code`\ s. + + mccabe + The project |Flake8| depends on to calculate the McCabe complexity + of a unit of code (e.g., a function). This uses the ``C`` + :term:`class` of :term`error code`\ s. From 87b331c552a6d4ff955e79f19546f3d08e8e52d8 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 09:45:08 -0500 Subject: [PATCH 356/364] Add some notes for users about how to use plugins --- docs/source/user/index.rst | 1 + docs/source/user/using-plugins.rst | 66 ++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+) create mode 100644 docs/source/user/using-plugins.rst diff --git a/docs/source/user/index.rst b/docs/source/user/index.rst index 255ec27..0df911e 100644 --- a/docs/source/user/index.rst +++ b/docs/source/user/index.rst @@ -24,6 +24,7 @@ This guide will cover all of these and the nuances for using |Flake8|. configuration options ignoring-errors + using-plugins python-api .. config files diff --git a/docs/source/user/using-plugins.rst b/docs/source/user/using-plugins.rst new file mode 100644 index 0000000..fad1911 --- /dev/null +++ b/docs/source/user/using-plugins.rst @@ -0,0 +1,66 @@ +================================== + Using Plugins For Fun and Profit +================================== + +|Flake8| is useful on its own but a lot of |Flake8|'s popularity is due to +its extensibility. Our community has developed :term:`plugin`\ s that augment +|Flake8|'s behaviour. Most of these plugins are uploaded to PyPI_. The +developers of these plugins often have some style they wish to enforce. + +For example, `flake8-docstrings`_ adds a check for :pep:`257` style +conformance. Others attempt to enforce consistency, like `flake8-future`_. + +.. note:: + + The accuracy or reliability of these plugins may vary wildly from plugin + to plugin and not all plugins are guaranteed to work with |Flake8| 3.0. + +To install a third-party plugin, make sure that you know which version of +Python (or pip) you used to install |Flake8|. You can then use the most +appropriate of: + +.. prompt:: bash + + pip install + pip3 install + python -m pip install + python2.7 -m pip install + python3 -m pip install + python3.4 -m pip install + python3.5 -m pip install + +To install the plugin, where ```` is the package name on PyPI_. +To verify installation use: + +.. prompt:: bash + + flake8 --version + python -m flake8 --version + +To see the plugin's name and version in the output. + +.. seealso:: :ref:`How to Invoke Flake8 ` + +After installation, most plugins immediately start reporting :term:`error`\ s. +Check the plugin's documentation for which error codes it returns and if it +disables any by default. + +.. note:: + + You can use both :option:`flake8 --select` and :option:`flake8 --ignore` + with plugins. + +Some plugins register new options, so be sure to check :option:`flake8 --help` +for new flags and documentation. These plugins may also allow these flags to +be specified in your configuration file. Hopefully, the plugin authors have +documented this for you. + +.. seealso:: :ref:`Configuring Flake8 ` + + +.. _PyPI: + https://pypi.io/ +.. _flake8-docstrings: + https://pypi.io/project/flake8-docstrings/ +.. _flake8-future: + https://pypi.io/project/flake8-future/ From 14ce512b9a62c1f4652eebc0706f7fe674a423b4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 09:49:06 -0500 Subject: [PATCH 357/364] Add extra information to plugin dev index --- docs/source/dev/index.rst | 48 +++++++++++++++++++++++++++++++++++---- 1 file changed, 43 insertions(+), 5 deletions(-) diff --git a/docs/source/dev/index.rst b/docs/source/dev/index.rst index 858ec83..728ae95 100644 --- a/docs/source/dev/index.rst +++ b/docs/source/dev/index.rst @@ -5,14 +5,52 @@ Since |Flake8| 2.0, the |Flake8| tool has allowed for extensions and custom plugins. In |Flake8| 3.0, we're expanding that ability to customize and extend **and** we're attempting to thoroughly document it. Some of the -documentation in this section will reference third-party documentation -to reduce duplication and to point you, the developer, towards -the authoritative documentation for those pieces. +documentation in this section may reference third-party documentation to +reduce duplication and to point you, the developer, towards the authoritative +documentation for those pieces. + +Getting Started +=============== + +To get started writing a |Flake8| :term:`plugin` you first need: + +- An idea for a plugin + +- An available package name on PyPI + +- One or more versions of Python installed + +- A text editor or IDE of some kind + +- An idea of what *kind* of plugin you want to build: + + * Formatter + + * Check + +Once you've gathered these things, you can get started. + +All plugins for |Flake8| must be registered via `entry points`_. In this +section we cover: + +- How to register your plugin so |Flake8| can find it + +- How to make |Flake8| provide your check plugin with information (via + command-line flags, function/class parameters, etc.) + +- How to make a formatter plugin + +- How to write your check plugin so that it works with |Flake8| 2.x and 3.x .. toctree:: + :caption: Plugin Developer Documentation :maxdepth: 2 - cross_compatibility - plugin_parameters registering_plugins + plugin_parameters formatters + cross_compatibility + + +.. _entry points: + https://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points From 5c8d767626a31560494996cd02ec5d654734aab2 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 09:51:15 -0500 Subject: [PATCH 358/364] Rename dev subdirectory to plugin-development This should make the contents clearer --- docs/source/index.rst | 2 +- docs/source/{dev => plugin-development}/.keep | 0 .../cross-compatibility.rst} | 0 docs/source/{dev => plugin-development}/formatters.rst | 0 docs/source/{dev => plugin-development}/index.rst | 6 +++--- .../plugin-parameters.rst} | 0 .../registering-plugins.rst} | 0 7 files changed, 4 insertions(+), 4 deletions(-) rename docs/source/{dev => plugin-development}/.keep (100%) rename docs/source/{dev/cross_compatibility.rst => plugin-development/cross-compatibility.rst} (100%) rename docs/source/{dev => plugin-development}/formatters.rst (100%) rename docs/source/{dev => plugin-development}/index.rst (95%) rename docs/source/{dev/plugin_parameters.rst => plugin-development/plugin-parameters.rst} (100%) rename docs/source/{dev/registering_plugins.rst => plugin-development/registering-plugins.rst} (100%) diff --git a/docs/source/index.rst b/docs/source/index.rst index 9fb02c0..51b0189 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -100,7 +100,7 @@ plugins and distribute them to others. .. toctree:: :maxdepth: 2 - dev/index + plugin-development/index Contributor Guide ================= diff --git a/docs/source/dev/.keep b/docs/source/plugin-development/.keep similarity index 100% rename from docs/source/dev/.keep rename to docs/source/plugin-development/.keep diff --git a/docs/source/dev/cross_compatibility.rst b/docs/source/plugin-development/cross-compatibility.rst similarity index 100% rename from docs/source/dev/cross_compatibility.rst rename to docs/source/plugin-development/cross-compatibility.rst diff --git a/docs/source/dev/formatters.rst b/docs/source/plugin-development/formatters.rst similarity index 100% rename from docs/source/dev/formatters.rst rename to docs/source/plugin-development/formatters.rst diff --git a/docs/source/dev/index.rst b/docs/source/plugin-development/index.rst similarity index 95% rename from docs/source/dev/index.rst rename to docs/source/plugin-development/index.rst index 728ae95..c3efb1d 100644 --- a/docs/source/dev/index.rst +++ b/docs/source/plugin-development/index.rst @@ -46,10 +46,10 @@ section we cover: :caption: Plugin Developer Documentation :maxdepth: 2 - registering_plugins - plugin_parameters + registering-plugins + plugin-parameters formatters - cross_compatibility + cross-compatibility .. _entry points: diff --git a/docs/source/dev/plugin_parameters.rst b/docs/source/plugin-development/plugin-parameters.rst similarity index 100% rename from docs/source/dev/plugin_parameters.rst rename to docs/source/plugin-development/plugin-parameters.rst diff --git a/docs/source/dev/registering_plugins.rst b/docs/source/plugin-development/registering-plugins.rst similarity index 100% rename from docs/source/dev/registering_plugins.rst rename to docs/source/plugin-development/registering-plugins.rst From 1a2c68f5da8ae95b8a156ef6f6a772bf82cf0f88 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 10:12:13 -0500 Subject: [PATCH 359/364] Move flake8 into src This is an emerging best practice and there is little reason to not follow it --- setup.py | 4 ++++ {flake8 => src/flake8}/__init__.py | 0 {flake8 => src/flake8}/__main__.py | 0 {flake8 => src/flake8}/api/__init__.py | 0 {flake8 => src/flake8}/checker.py | 0 {flake8 => src/flake8}/defaults.py | 0 {flake8 => src/flake8}/exceptions.py | 0 {flake8 => src/flake8}/formatting/__init__.py | 0 {flake8 => src/flake8}/formatting/base.py | 0 {flake8 => src/flake8}/formatting/default.py | 0 {flake8 => src/flake8}/main/__init__.py | 0 {flake8 => src/flake8}/main/application.py | 0 {flake8 => src/flake8}/main/cli.py | 0 {flake8 => src/flake8}/main/git.py | 0 {flake8 => src/flake8}/main/mercurial.py | 0 {flake8 => src/flake8}/main/options.py | 0 {flake8 => src/flake8}/main/setuptools_command.py | 0 {flake8 => src/flake8}/main/vcs.py | 0 {flake8 => src/flake8}/options/__init__.py | 0 {flake8 => src/flake8}/options/aggregator.py | 0 {flake8 => src/flake8}/options/config.py | 0 {flake8 => src/flake8}/options/manager.py | 0 {flake8 => src/flake8}/plugins/__init__.py | 0 {flake8 => src/flake8}/plugins/_trie.py | 0 {flake8 => src/flake8}/plugins/manager.py | 0 {flake8 => src/flake8}/plugins/notifier.py | 0 {flake8 => src/flake8}/plugins/pyflakes.py | 0 {flake8 => src/flake8}/processor.py | 0 {flake8 => src/flake8}/style_guide.py | 0 {flake8 => src/flake8}/utils.py | 0 tests/unit/test_utils.py | 6 +++--- tox.ini | 2 +- 32 files changed, 8 insertions(+), 4 deletions(-) rename {flake8 => src/flake8}/__init__.py (100%) rename {flake8 => src/flake8}/__main__.py (100%) rename {flake8 => src/flake8}/api/__init__.py (100%) rename {flake8 => src/flake8}/checker.py (100%) rename {flake8 => src/flake8}/defaults.py (100%) rename {flake8 => src/flake8}/exceptions.py (100%) rename {flake8 => src/flake8}/formatting/__init__.py (100%) rename {flake8 => src/flake8}/formatting/base.py (100%) rename {flake8 => src/flake8}/formatting/default.py (100%) rename {flake8 => src/flake8}/main/__init__.py (100%) rename {flake8 => src/flake8}/main/application.py (100%) rename {flake8 => src/flake8}/main/cli.py (100%) rename {flake8 => src/flake8}/main/git.py (100%) rename {flake8 => src/flake8}/main/mercurial.py (100%) rename {flake8 => src/flake8}/main/options.py (100%) rename {flake8 => src/flake8}/main/setuptools_command.py (100%) rename {flake8 => src/flake8}/main/vcs.py (100%) rename {flake8 => src/flake8}/options/__init__.py (100%) rename {flake8 => src/flake8}/options/aggregator.py (100%) rename {flake8 => src/flake8}/options/config.py (100%) rename {flake8 => src/flake8}/options/manager.py (100%) rename {flake8 => src/flake8}/plugins/__init__.py (100%) rename {flake8 => src/flake8}/plugins/_trie.py (100%) rename {flake8 => src/flake8}/plugins/manager.py (100%) rename {flake8 => src/flake8}/plugins/notifier.py (100%) rename {flake8 => src/flake8}/plugins/pyflakes.py (100%) rename {flake8 => src/flake8}/processor.py (100%) rename {flake8 => src/flake8}/style_guide.py (100%) rename {flake8 => src/flake8}/utils.py (100%) diff --git a/setup.py b/setup.py index 8b9617d..9afe94a 100644 --- a/setup.py +++ b/setup.py @@ -3,10 +3,13 @@ from __future__ import with_statement import functools +import os import sys import setuptools +sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src')) + import flake8 # noqa try: @@ -66,6 +69,7 @@ setuptools.setup( maintainer="Ian Cordasco", maintainer_email="graffatcolmingov@gmail.com", url="https://gitlab.com/pycqa/flake8", + package_dir={"": "src"}, packages=[ "flake8", "flake8.api", diff --git a/flake8/__init__.py b/src/flake8/__init__.py similarity index 100% rename from flake8/__init__.py rename to src/flake8/__init__.py diff --git a/flake8/__main__.py b/src/flake8/__main__.py similarity index 100% rename from flake8/__main__.py rename to src/flake8/__main__.py diff --git a/flake8/api/__init__.py b/src/flake8/api/__init__.py similarity index 100% rename from flake8/api/__init__.py rename to src/flake8/api/__init__.py diff --git a/flake8/checker.py b/src/flake8/checker.py similarity index 100% rename from flake8/checker.py rename to src/flake8/checker.py diff --git a/flake8/defaults.py b/src/flake8/defaults.py similarity index 100% rename from flake8/defaults.py rename to src/flake8/defaults.py diff --git a/flake8/exceptions.py b/src/flake8/exceptions.py similarity index 100% rename from flake8/exceptions.py rename to src/flake8/exceptions.py diff --git a/flake8/formatting/__init__.py b/src/flake8/formatting/__init__.py similarity index 100% rename from flake8/formatting/__init__.py rename to src/flake8/formatting/__init__.py diff --git a/flake8/formatting/base.py b/src/flake8/formatting/base.py similarity index 100% rename from flake8/formatting/base.py rename to src/flake8/formatting/base.py diff --git a/flake8/formatting/default.py b/src/flake8/formatting/default.py similarity index 100% rename from flake8/formatting/default.py rename to src/flake8/formatting/default.py diff --git a/flake8/main/__init__.py b/src/flake8/main/__init__.py similarity index 100% rename from flake8/main/__init__.py rename to src/flake8/main/__init__.py diff --git a/flake8/main/application.py b/src/flake8/main/application.py similarity index 100% rename from flake8/main/application.py rename to src/flake8/main/application.py diff --git a/flake8/main/cli.py b/src/flake8/main/cli.py similarity index 100% rename from flake8/main/cli.py rename to src/flake8/main/cli.py diff --git a/flake8/main/git.py b/src/flake8/main/git.py similarity index 100% rename from flake8/main/git.py rename to src/flake8/main/git.py diff --git a/flake8/main/mercurial.py b/src/flake8/main/mercurial.py similarity index 100% rename from flake8/main/mercurial.py rename to src/flake8/main/mercurial.py diff --git a/flake8/main/options.py b/src/flake8/main/options.py similarity index 100% rename from flake8/main/options.py rename to src/flake8/main/options.py diff --git a/flake8/main/setuptools_command.py b/src/flake8/main/setuptools_command.py similarity index 100% rename from flake8/main/setuptools_command.py rename to src/flake8/main/setuptools_command.py diff --git a/flake8/main/vcs.py b/src/flake8/main/vcs.py similarity index 100% rename from flake8/main/vcs.py rename to src/flake8/main/vcs.py diff --git a/flake8/options/__init__.py b/src/flake8/options/__init__.py similarity index 100% rename from flake8/options/__init__.py rename to src/flake8/options/__init__.py diff --git a/flake8/options/aggregator.py b/src/flake8/options/aggregator.py similarity index 100% rename from flake8/options/aggregator.py rename to src/flake8/options/aggregator.py diff --git a/flake8/options/config.py b/src/flake8/options/config.py similarity index 100% rename from flake8/options/config.py rename to src/flake8/options/config.py diff --git a/flake8/options/manager.py b/src/flake8/options/manager.py similarity index 100% rename from flake8/options/manager.py rename to src/flake8/options/manager.py diff --git a/flake8/plugins/__init__.py b/src/flake8/plugins/__init__.py similarity index 100% rename from flake8/plugins/__init__.py rename to src/flake8/plugins/__init__.py diff --git a/flake8/plugins/_trie.py b/src/flake8/plugins/_trie.py similarity index 100% rename from flake8/plugins/_trie.py rename to src/flake8/plugins/_trie.py diff --git a/flake8/plugins/manager.py b/src/flake8/plugins/manager.py similarity index 100% rename from flake8/plugins/manager.py rename to src/flake8/plugins/manager.py diff --git a/flake8/plugins/notifier.py b/src/flake8/plugins/notifier.py similarity index 100% rename from flake8/plugins/notifier.py rename to src/flake8/plugins/notifier.py diff --git a/flake8/plugins/pyflakes.py b/src/flake8/plugins/pyflakes.py similarity index 100% rename from flake8/plugins/pyflakes.py rename to src/flake8/plugins/pyflakes.py diff --git a/flake8/processor.py b/src/flake8/processor.py similarity index 100% rename from flake8/processor.py rename to src/flake8/processor.py diff --git a/flake8/style_guide.py b/src/flake8/style_guide.py similarity index 100% rename from flake8/style_guide.py rename to src/flake8/style_guide.py diff --git a/flake8/utils.py b/src/flake8/utils.py similarity index 100% rename from flake8/utils.py rename to src/flake8/utils.py diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 062f7f9..5d31e82 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -71,15 +71,15 @@ def test_fnmatch_returns_the_default_with_empty_default(): def test_filenames_from_a_directory(): """Verify that filenames_from walks a directory.""" - filenames = list(utils.filenames_from('flake8/')) + filenames = list(utils.filenames_from('src/flake8/')) assert len(filenames) > 2 - assert 'flake8/__init__.py' in filenames + assert 'src/flake8/__init__.py' in filenames def test_filenames_from_a_directory_with_a_predicate(): """Verify that predicates filter filenames_from.""" filenames = list(utils.filenames_from( - arg='flake8/', + arg='src/flake8/', predicate=lambda filename: filename == 'flake8/__init__.py', )) assert len(filenames) > 2 diff --git a/tox.ini b/tox.ini index c17f025..beeb2e3 100644 --- a/tox.ini +++ b/tox.ini @@ -24,7 +24,7 @@ deps = flake8-docstrings>=0.2.7 flake8-import-order commands = - flake8 + flake8 src/flake8/ tests/ setup.py [testenv:pylint] basepython = python3 From 7b31ecf7d50cf63d8d32fd6b5f6f840cc05c2132 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 10:48:52 -0500 Subject: [PATCH 360/364] Update the MANIFEST.in for sdists Let's include our documentation, tests, and src code appropriately --- MANIFEST.in | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/MANIFEST.in b/MANIFEST.in index ab3d229..b87aff1 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,4 +1,6 @@ include *.rst include CONTRIBUTORS.txt include LICENSE -recursive-include flake8 * +recursive-include docs * +recursive-include tests * +recursive-include src * From 1ec83033e96d43c147041f4f44639ee50152d1f6 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 10:49:32 -0500 Subject: [PATCH 361/364] Update the copyright in the LICENSE --- LICENSE | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/LICENSE b/LICENSE index 32b274a..e5e3d6f 100644 --- a/LICENSE +++ b/LICENSE @@ -1,7 +1,7 @@ == Flake8 License (MIT) == Copyright (C) 2011-2013 Tarek Ziade -Copyright (C) 2012-2013 Ian Cordasco +Copyright (C) 2012-2016 Ian Cordasco Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in From 5d1150cc2709f79510cbc3b3b37d4ddebb97a1f4 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 11:20:28 -0500 Subject: [PATCH 362/364] Update the packaging so wheels work --- setup.cfg | 3 +++ setup.py | 3 ++- tox.ini | 13 +++++++++++++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/setup.cfg b/setup.cfg index 831bd62..218ddcc 100644 --- a/setup.cfg +++ b/setup.cfg @@ -8,6 +8,9 @@ universal=1 requires-dist = enum34; python_version<"3.4" configparser; python_version<"3.2" + pyflakes >= 0.8.1, != 1.2.0, != 1.2.1, != 1.2.2, < 1.3.0 + pycodestyle >= 2.0.0, < 2.1.0 + mccabe >= 0.5.0, < 0.6.0 [pytest] norecursedirs = .git .* *.egg* old docs dist build diff --git a/setup.py b/setup.py index 9afe94a..0df57b7 100644 --- a/setup.py +++ b/setup.py @@ -115,7 +115,8 @@ setuptools.setup( PEP8_PLUGIN('comparison_to_singleton'), PEP8_PLUGIN('comparison_negative'), PEP8_PLUGIN('comparison_type'), - PEP8_PLUGIN('ambiguous_identifier'), + # NOTE(sigmavirus24): Add this back once PyCodestyle 2.1.0 is out + # PEP8_PLUGIN('ambiguous_identifier'), PEP8_PLUGIN('python_3000_has_key'), PEP8_PLUGIN('python_3000_raise_comma'), PEP8_PLUGIN('python_3000_not_equal'), diff --git a/tox.ini b/tox.ini index beeb2e3..58e7072 100644 --- a/tox.ini +++ b/tox.ini @@ -108,6 +108,19 @@ deps = commands = python setup.py check -r -s +[testenv:release] +skipsdist = true +basepython = python3 +skip_install = true +use_develop = false +deps = + wheel + setuptools + twine >= 1.5.0 +commands = + python setup.py -q sdist bdist_wheel + twine upload --skip-existing dist/* + # Flake8 Configuration [flake8] # Ignore some flake8-docstrings errors From 7340b3e05715c73d4f93c177e7f1f978f6ccf525 Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 11:40:41 -0500 Subject: [PATCH 363/364] Add release note for 2.6.1 --- docs/source/release-notes/2.6.1.rst | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 docs/source/release-notes/2.6.1.rst diff --git a/docs/source/release-notes/2.6.1.rst b/docs/source/release-notes/2.6.1.rst new file mode 100644 index 0000000..d05bd18 --- /dev/null +++ b/docs/source/release-notes/2.6.1.rst @@ -0,0 +1,6 @@ +2.6.1 - 2016-06-25 +------------------ + +- **Bug** Update the config files to search for to include ``setup.cfg`` and + ``tox.ini``. This was broken in 2.5.5 when we stopped passing + ``config_file`` to our Style Guide From 6eb2e3a70147baad1cc2ad3d51c269974da2320f Mon Sep 17 00:00:00 2001 From: Ian Cordasco Date: Sat, 25 Jun 2016 11:55:52 -0500 Subject: [PATCH 364/364] Add more release notes for 3.0.0b1 --- docs/source/release-notes/3.0.0.rst | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/source/release-notes/3.0.0.rst b/docs/source/release-notes/3.0.0.rst index 605f8a7..8f13cfe 100644 --- a/docs/source/release-notes/3.0.0.rst +++ b/docs/source/release-notes/3.0.0.rst @@ -1,5 +1,5 @@ -3.0.0 -- Unreleased -------------------- +3.0.0b1 -- 2016-06-25 +--------------------- - Rewrite our documentation from scratch! (http://flake8.pycqa.org) @@ -28,3 +28,15 @@ - Update ``--install-hook`` to require the name of version control system hook you wish to install a Flake8. + +- Stop checking sub-directories more than once via the setuptools command + +- When passing a file on standard-in, allow the caller to specify + ``--stdin-display-name`` so the output is properly formatted + +- The Git hook now uses ``sys.executable`` to format the shebang line. + This allows Flake8 to install a hook script from a virtualenv that points to + that virtualenv's Flake8 as opposed to a global one (without the virtualenv + being sourced). + +- When using ``--count``, the output is no longer written to stderr.