maximum recursion depth exceeded in 1050 string join

[pyflakes-bot]

Original report by jayvdb (@jayvdb?) on Launchpad:

---

Running current pyflakes on astroid results in maximum recursion depth exceeded on a test module.

https://bitbucket.org/logilab/astroid/src/6d4e198bdc7091f36c2c24d911c5ee92b64847c2/astroid/tests/testdata/python2/data/joined_strings.py

$ pyflakes  astroid/tests/testdata/python2/data/joined_strings.py
Traceback (most recent call last):
  File "/usr/bin/pyflakes", line 9, in <module>
    load_entry_point('pyflakes==1.0.0', 'console_scripts', 'pyflakes')()
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/api.py", line 172, in main
    warnings = checkRecursive(args, reporter)
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/api.py", line 129, in checkRecursive
    warnings += checkPath(sourcePath, reporter)
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/api.py", line 96, in checkPath
    return check(codestr, filename, reporter)
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/api.py", line 57, in check
    w = checker.Checker(tree, filename)
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/checker.py", line 294, in __init__
    self.handleChildren(tree)
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/checker.py", line 567, in handleChildren
    self.handleNode(node, tree)
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/checker.py", line 609, in handleNode
    handler(node)
...
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/checker.py", line 609, in handleNode
    handler(node)
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/checker.py", line 566, in handleChildren
    for node in iter_child_nodes(tree, omit=omit):
  File "/usr/lib/python2.7/site-packages/pyflakes-1.0.0-py2.7.egg/pyflakes/checker.py", line 80, in iter_child_nodes
    if isinstance(field, ast.AST):
RuntimeError: maximum recursion depth exceeded while calling a Python object

Setting sys.setrecursionlimit(2500) fixes the problem

[pyflakes-bot]

Original comment by icordasc (@sigmavirus24?) on Launchpad:

---

So I looked into this a little.

As you can (maybe) tell by the stack trace what happens is we get stuck in a recursion loop between handleChildren and handleNode. Why?

So the first thing we see is the Assign statement, and the value of the assign is a BinOp which consists of a left with another BinOp and a right with a string. You might have an idea where this is going. That BinOp then has a BinOp on the left side and a String on the right. Etc. etc. etc.

I don't know how many things like this we'll see in the real world, so my inclination is to maybe figure out what operations we might want to try to flatten before recursing through them if that makes any sense. We could potentially flatten this like so:

[BinOp, String]
[BinOp, String, String]
[BinOp, String, String, String]
...

But I have a hunch that'll be slow to do and could end up being impractical for anything other than some what special cases like this one. (I'm also operating on low coffee so it could just be I'm missing an obvious solution.)

[pyflakes-bot]

Original comment by jayvdb (@jayvdb?) on Launchpad:

---

WIP patch up at #70
It looks like my solution performs roughly equivalently as the current code, with a definite measurable reduction in sys time, but I havent done any extensive timing yet, since the code/timing will need to change quite a bit to fix cases where it currently passes the tests but reports non-existent problems.

A more simplistic approach, suitable for merging promptly, is #68

[pyflakes-bot]

Original comment by jayvdb (@jayvdb?) on Launchpad:

---

I ran into this bug on https://pypi.python.org/pypi/idna recently released v2.1, file https://github.com/kjd/idna/blob/master/idna/idnadata.py

This is ranked 119th most popular package : http://pypi-ranking.info/module/idna

Both patches fix this bug for idna as well as astroid's test data.

[Congee]

can we have a command line flag or config to set the recursion limit? edition the source code is really annoying.

[AbdealiLoKo]

I am also facing this issue in some of my projects.
Is there any interest in resolving the recursive nature ?
I am happy to help and contribute and work on top of jayvdb's previous PR

[asottile]

I suspect to actually fix this a full rewrite of pyflakes is necessary. the current structure depends on recursion to do a tree traversal. jayvdb's patch improves this but does not eliminate the recursion (so while it may fix some cases, it does not solve all of them)

[AbdealiLoKo]

I see @asottile - thanks for the reply.
Do you think it worthwhile I take a look and see what could potentially be done ?
I assume a full-rewrite is not a great idea ... as it can cause major breakages.

But if there is something to reduce the chance of finding this error - would you be open to getting that merged ? Or are you thinking we should just find a complete fix to the solution ?