"Run Test with Coverage" runs *all* tests when test execution fails

[bersbersbers]

TLDR:

• do not run all tests when test execution fails
• also, do not run tests at all when you cannot collect the IDs

I am currently debugging a problem in my tests, most likely related to coveragepy/coveragepy#1392. I am trying the omit workaround, but currently, I am getting vscode-pytest errors such as:

test_bug.py .                                                            [100%]Error[vscode-pytest]: unable to read testIds from temp fileNo source for code: 'C:\Git\Bug\shibokensupport\signature\lib\tool.py'.

When this happens, VS Code re-starts all tests, which is not what I want at all.

Compare this snippet:

vscode-python/python_files/vscode_pytest/run_pytest_script.py

Lines 54 to 67 in e8dd8c0

	try:
	# Read the test ids from the file, delete file, and run pytest.
	ids_path = pathlib.Path(run_test_ids_pipe)
	ids = ids_path.read_text(encoding="utf-8").splitlines()
	try:
	ids_path.unlink()
	except Exception as e:
	print("Error[vscode-pytest]: unable to delete temp file" + str(e))
	arg_array = ["-p", "vscode_pytest", *args, *ids]
	print("Running pytest with args: " + str(arg_array))
	pytest.main(arg_array)
	except Exception as e:
	print("Error[vscode-pytest]: unable to read testIds from temp file" + str(e))
	run_pytest(args)

IMHO, this should rather be something like (untested!) - note the return in the first block and the different arg_array in the last block:

try: 
    # Read the test ids from the file, delete file, and run pytest. 
    ids_path = pathlib.Path(run_test_ids_pipe) 
    ids = ids_path.read_text(encoding="utf-8").splitlines() 
except Exception as e: 
    print("Error[vscode-pytest]: unable to read testIds from temp file: " + str(e)) 
    return
finally:
    try: 
        ids_path.unlink() 
    except Exception as e: 
        print("Error[vscode-pytest]: unable to delete temp file: " + str(e)) 

try: 
    arg_array = ["-p", "vscode_pytest", *args, *ids] 
    print("Running pytest with args: " + str(arg_array)) 
    pytest.main(arg_array) 
except Exception as e: 
    print("Error[vscode-pytest]: unable to run tests: " + str(e)) 
    arg_array = [*args, *ids] 
    print("Running pytest with args: " + str(arg_array)) 
    run_pytest(arg_array)

[eleanorjboyd]

hoping to take a look at fixing this soon- sorry for the delay

[RamiAwar]

I'm getting something similar actually, might be relevant given closeness in time of reports:
Error[vscode-pytest]: unable to read testIds from temp file[Errno 2] No such file or directory: '/tmp/test-ids-e2faf2fe418633dcef80.txt'

I'm trying to run one test but it 'fails to detect the test ID'.

Note that I'm using devcontainers, with Python + Pylance + pytest explorer installed inside.

Also, not sure if relevant: I looked at /tmp/ but can't see any files named like this. I only see files named like python-test-discovery-698debb64df061b8b350.sock

LOGS:

2024-10-31 22:11:25.650 [info] Attempting to use temp directory for test ids file, file name: test-ids-750dd803c28e00398642.txt
env vars: "TEST_RUN_PIPE":"/tmp/python-test-results-59956af0fc8ae043f0d0.sock","RUN_TEST_IDS_PIPE":"/tmp/test-ids-750dd803c28e00398642.txt"}

2024-10-31 22:11:25.658 [info] Running pytest with arguments: /root/.vscode-server/extensions/ms-python.python-2024.16.1-linux-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=/code/myproj

2024-10-31 22:11:25.658 [info] > /usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2024.16.1-linux-arm64/python_files/vscode_pytest/run_pytest_script.py --rootdir=.

2024-10-31 22:11:25.658 [info] cwd: .

At this point I see this in my test results:

Error[vscode-pytest]: unable to read testIds from temp file[Errno 2] No such file or directory: '/tmp/test-ids-750dd803c28e00398642.txt'

Python logs don't show anything useful after that and I cancel the test runs. I also watch ls /tmp as I clicked on a test run and only saw the -result files appear. No IDs files.

[eleanorjboyd]

hm interesting- this seems different, could you file a new issue? We use that txt file as a temp to transfer the names of the test ids to the subprocess we spin up so I would guess it might be permissions having to do with the temp directory since we aren't doing anythign specific beyond that

[RamiAwar]

hm interesting- this seems different, could you file a new issue? We use that txt file as a temp to transfer the names of the test ids to the subprocess we spin up so I would guess it might be permissions having to do with the temp directory since we aren't doing anythign specific beyond that

Done, put it in #24406

[eleanorjboyd]

Hi @bersbersbers, could you clarify what you mean here: "do not run all tests when test execution fails". Which test execution fails? The reading the testIds? Also could you give it another try - we have changed to using a different communication method which might help, this is only out on the pre-release of the python extension so you would need to try it there. Thanks!

[bersbersbers]

Which test execution fails?

What I mean is that in the code snippet above, pytest.main(arg_array) throws an exception (in my case, it was a NoSource exception, but it can be anything). See the far end of the console output starting with test_bug.py .. The code then incorrectly prints it is "unable to read testIds from temp file", but that is a misleading error message. IMHO, reading of test IDs and running the tests should be in two independent try blocks, see my proposed code snippet above.

Also could you give it another try

Yes, will do tomorrow. But I am fairly certain the commuication method does not have anything to do with this, because it's not the reading of test IDs that fails, but the test process itself.

[bersbersbers]

Also could you give it another try

I did. Obviously, since #24308 seems to be fixed, my original reproducer does not work any more. But I can still reproduce the behavior by adding

if any("::" in arg for arg in args):
    raise RuntimeError("foo")

to _pytest\config\__init__.py, main() function.

(This is supposed to simulate a pytest crash similar to before. Note that the above main() function does not catch all exceptions.)

Then run test with coverage for a single test. What then happens is:

• pytest starts for one test and crashes.
• pytest_code reports "unable to read testIds from temp file". This is incorrect.
• pytest is re-started for all tests. This is the bug.

(This is with v2024.21.2024111501)