Fix up FileProcessor.file_tokens property

We opted to not copy the file_tokens attribute each time it's accessed
in the merge request discussion but it was never reflected in the code.
Further, the attribute had no documentation or docstring, so we've added
that. Finally, we address a personal style nit that I otherwise wouldn't
have picked at.
This commit is contained in:
Ian Cordasco 2016-10-25 12:02:00 -05:00
parent 1cfc12f366
commit 8dfe38e9e6
No known key found for this signature in database
GPG key ID: 656D3395E4A9791A

View file

@ -103,15 +103,22 @@ class FileProcessor(object):
@property
def file_tokens(self):
"""The complete set of tokens for a file.
Accessing this attribute *may* raise an InvalidSyntax exception.
:raises: flake8.exceptions.InvalidSyntax
"""
if self._file_tokens is None:
line_iter = iter(self.lines)
try:
self._file_tokens = list(tokenize.generate_tokens(
lambda: next(line_iter)))
lambda: next(line_iter)
))
except tokenize.TokenError as exc:
raise exceptions.InvalidSyntax(exc.message, exception=exc)
return self._file_tokens[:]
return self._file_tokens
@contextlib.contextmanager
def inside_multiline(self, line_number):