RuntimeError: dictionary changed size during iteration

[Kostanos]

Running test on python 3.6 with pyfakefs==3.3 and got this error:

Traceback (most recent call last):
  File "/root/app/lib/settings/__init___test.py", line 80, in setUp
    self.setUpPyfakefs()
  File "/usr/local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py", line 221, in setUpPyfakefs
    self._stubber.setUp()
  File "/usr/local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py", line 400, in setUp
    self._findModules()
  File "/usr/local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py", line 324, in _findModules
    for name, module in set(sys.modules.items()):
RuntimeError: dictionary changed size during iteration

In previous python version it worked well.

[mrbean-bremen]

Thanks for the report!
Can you give an example for a test where that happens? I never saw this, and I mostly work under Python 3.6. What OS are you working under?

[Kostanos]

Here is the public gist for the code of my test file: https://gist.github.com/Kostanos/736dddbc36d226962c3c43a7f95416d3#file-__init___test-py-L81

I'm running it on python:3-alpine3.6 docker image: https://github.com/docker-library/python/blob/a1aa406bfd8c7b129e6e0ee0ba972b863624ac0d/3.6/alpine3.6/Dockerfile

I found the similar issues in other projects with python 3.x: https://stackoverflow.com/a/11941855/2215679

Let me know if you need more info.
Thank you.

[mrbean-bremen]

I still don't get it... The problem usually occurs, if the iterated dictionary is changed during iteration. This seems not to happen here, though there must be some side effect that leads to this in your case...
From your code I see nothing that would trigger that. Can you strip down your example code to contain nothing but the setup and an empty test, to clarify if this caused by your enviroment? E.g.:

import unittest

from pyfakefs import fake_filesystem_unittest


class TestSettings(fake_filesystem_unittest.TestCase):
    def setUp(self):
        self.setUpPyfakefs()

    def testDefaultSettings(self):
        pass


if __name__ == '__main__':
    unittest.main()

[mrbean-bremen]

@Kostanos - any news there?

[mrbean-bremen]

@Kostanos - if you still have the problem, please test with the newly released version 3.4.

[Kostanos]

Hi, sorry, already on completely different project. Thank you.

[mrbean-bremen]

Can we close this then, as we have no real chance to get this figured out?

[Kostanos]

Suspend issue until we can test it again

[mrbean-bremen]

Ok, feel free to reopen it any time you get back to it. Thanks for the report!

[con-f-use]

I'm still experiencing this or a similar problem.

As a temporary workaround in fake_filesystem_uinittest.py I added an extra copy:

            # suppress specific pytest warning - see #466
            with warnings.catch_warnings():
                warnings.filterwarnings(
                    'ignore',
                    message='The compiler package is deprecated',
                    category=DeprecationWarning,
                    module='py'
                )
                modules = {name: mod for name, mod in module.__dict__.copy().items()
                           if is_fs_module(mod, name)}
                # ~~~~         formerly:    ~~~~~~
                #~modules = {name: mod for name, mod in module.__dict__.items()
                #~          if is_fs_module(mod, name)}
            for name, mod in modules.items():
                self._modules.setdefault(name, set()).add((module,
                                                           mod.__name__))
            functions = {name: fct for name, fct in module.__dict__.items()
                         if is_fs_function(fct)}

[mrbean-bremen]

Ok, reopening in this case.
@con-f-use - what Python and pyfakefs versions do you use, and what OS? Do you get this for each test?

[con-f-use]

@mrbean-bremen

┌─08-22:20 [confus@confusion6 ~/devel/pipenv-to-requirements/pipenv_to_requirements]  😱  master|✓ 😱
└$ python3 -m pytest . -vvv                                                                                                                                                                                                                      
============================================================================================================== test session starts ==============================================================================================================
platform linux -- Python 3.6.8, pytest-5.0.1, py-1.7.0, pluggy-0.12.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/confus/devel/pipenv-to-requirements
plugins: pyfakefs-3.6, mock-1.10.4, xonsh-0.9.7
collected 8 items                                                                                                                                                                                                                               

test_cli.py::test_no_opt ERROR                                                                                                                                                                                                            [ 12%]
test_cli.py::test_opt_output PASSED                                                                                                                                                                                                       [ 25%]
test_cli.py::test_opt_dev_output PASSED                                                                                                                                                                                                   [ 37%]
test_cli.py::test_opt_dev_output_freeze PASSED                                                                                                                                                                                            [ 50%]
test_cli.py::test_opt_output_and_dev_output PASSED                                                                                                                                                                                        [ 62%]
test_parsing.py::TestParsing::test_normal PASSED                                                                                                                                                                                          [ 75%]
test_parsing.py::TestParsing::test_editable PASSED                                                                                                                                                                                        [ 87%]
test_parsing.py::TestParsing::test_markers PASSED                                                                                                                                                                                         [100%]

==================================================================================================================== ERRORS =====================================================================================================================
_________________________________________________________________________________________________________ ERROR at setup of test_no_opt _________________________________________________________________________________________________________

    @pytest.fixture
    def fs():
        """ Fake filesystem. """
        patcher = Patcher()
>       patcher.setUp()

../../../.local/lib/python3.6/site-packages/pyfakefs/pytest_plugin.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../.local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py:524: in setUp
    self._find_modules()
../../../.local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py:487: in _find_modules
    modules = {name: mod for name, mod in module.__dict__.items()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <dict_itemiterator object at 0x7ff21f3ef4f8>

>   modules = {name: mod for name, mod in module.__dict__.items()
    #modules = {name: mod for name, mod in module.__dict__.copy().items()
               if is_fs_module(mod, name)}
E   RuntimeError: dictionary changed size during iteration

../../../.local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py:487: RuntimeError
======================================================================================================= 7 passed, 1 error in 1.19 seconds =======================================================================================================

It's probably unrelated but how does xonsh (which is the shell I use) list under plugins?

[mrbean-bremen]

Hm, looks some kind of weird side effect with a specific module - probably your change is the best fix in this case. Out of interest - can you check which module it is choking on (e.g. by adding some print output to is_fs_module)? Interesting to know how that can happen... I think I will incorporate your fix tonight in this case.
As for xonsh - never had heard about this one, sounds interesting! And by the description I saw it is quite possible that it registers a pytest plugin (like pyfakefs does). Anyway, I may try it out and check that :)

[con-f-use]

Adding a print output in is_fs_module() just prints a lot of module names after the test is done, so no idea which one is the culprit.

[mrbean-bremen]

Well - the last printed one should be the one (given that you run only the failing test), but that's not that important anyway - just a matter of interest.

[con-f-use]

Alright, suit yourself then.

            # modules = {name: mod for name, mod in module.__dict__.copy().items()
            #                if is_fs_module(mod, name)}
            def iis_fs_module(m,n):
                print('name: ', n)
                return is_fs_module(m,n)
            modules = {name: mod for name, mod in module.__dict__.items()
                if iis_fs_module(mod, name)}

python3 -m pytest . -vvv -k 'test_no_opt' 2>&1 | xclip -selection clipboard

[mrbean-bremen]

Ok, don't have the time today, will have another look tomorrow.

[mrbean-bremen]

BTW, I checked that log file, and the last entry is RE_HIDDEN_BYTES, which is defined in xonsh.proc - so it has to do with xonsh, after all. I will see if I get to it tonight.

[con-f-use]

Who knew. The xonsh developers are usually very active, friendly and helpful. If one of us tells them of this issue, I suspect they'll be very quick to fix it. Given it's a flaw in their code.

[mrbean-bremen]

Probably it's just unusual code, not a bug - and with your fix (e.g. copying the dict) we can handle this. I don't think the performance penalty is worth considering, but I'll check it.

[mrbean-bremen]

Ok, couldn't reproduce the problem under Windows with xonsh installed, but then I didn't really expect that - just wanted to play around with xonsh ;)
I'm just adding your fix in now - it doesn't impact the performance, as far as I could see. I didn't have an idea how to test this, though.

[hasansalimkanmaz]

Same issue here, It is not always giving error. I couldn't understand where it results from.

I read all the thread and upgraded my pyfakefs==4.1.0 that is the most recent one. OS is linux or mac, it doesn't matter.

___________ ERROR at setup of test_get_local_file_metadata_persistor ___________

    @pytest.fixture
    def fs():
        """ Fake filesystem. """
        patcher = Patcher()
>       patcher.setUp()

/usr/local/lib/python3.6/site-packages/pyfakefs/pytest_plugin.py:37: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py:514: in setUp
    self._find_modules()
/usr/local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py:484: in _find_modules
    modules = {name: mod for name, mod in module.__dict__.items()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

.0 = <dict_itemiterator object at 0x7fd7a1768ea8>

>   modules = {name: mod for name, mod in module.__dict__.items()
               if is_fs_module(mod, name)}
E   RuntimeError: dictionary changed size during iteration

/usr/local/lib/python3.6/site-packages/pyfakefs/fake_filesystem_unittest.py:484: RuntimeError

[mrbean-bremen]

@hasansalimkanmaz - can you please check with current master? I added more robust error handling recently, but this is not yet released.

[hasansalimkanmaz]

@mrbean-bremen - It works with master branch. Thanks for the info.

[mrbean-bremen]

Ok, thanks - will close this again. If needed we can make a patch release soon.