You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After the pytest header and test collection (but before test execution) pytest incorrectly reports how many tests are collected. This is visible with and without --collect-only.
For example:
fill tests/istanbul/eip1344_chainid/ --fork=Cancun --collect-only
outputs (incorrect output is marked as such):
================================== test session starts ===================================platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0Executing tests for: Cancun Only executing tests with stable/deployed forks: Specify an upcoming fork via --until=fork to add forks under development.rootdir: /home/dtopz/code/github/danceratopz/execution-spec-testsconfigfile: pytest.iniplugins: cov-4.1.0, xdist-3.6.1collecting ... startcollected 3 items / 1 deselected / 2 selected # <-- INCORRECT, should be "collected 3 items"<Package eip1344_chainid> <Module test_chainid.py> <Function test_chainid[fork_Cancun-blockchain_test]> <Function test_chainid[fork_Cancun-blockchain_test_hive]> <Function test_chainid[fork_Cancun-state_test]>====================== 2/3 tests collected (1 deselected) in 0.01s ======================= # <-- INCORRECT, should be "3 tests collected in 0.01 s"
For every test module collected, one additional "test" will be incorrectly reported as deselected (above we only collect one module).
The error is due to the fact that we automatically add EIP version tests in the spec_version_checker plugin (src/pytest_plugins/spec_version_checker/spec_version_checker.py) but then also disable them by default in fill's pytest.ini (L9 and L11):
The most reasonable solution would be to simply not use this plugin when executing fill and have yet another pytest command dedicated to version checking that can be ran regularly in CI. The reason to not enable these tests by default is that each of these version checking tests makes a request to the Github API (to retrieve info from ethereum/EIPs); without authentication setup, fill quickly hits the Github API rate limit.
Note, as of 2024-04-29, eip version checking is currently entirely unused in our CI. To run these tests, it's required to explicitly select tests marked as eip_version_check with:
fill tests/istanbul/eip1344_chainid/ --fork=Cancun -m eip_version_check -v
The text was updated successfully, but these errors were encountered:
After the pytest header and test collection (but before test execution) pytest incorrectly reports how many tests are collected. This is visible with and without
--collect-only
.For example:
fill tests/istanbul/eip1344_chainid/ --fork=Cancun --collect-only
outputs (incorrect output is marked as such):
For every test module collected, one additional "test" will be incorrectly reported as deselected (above we only collect one module).
The error is due to the fact that we automatically add EIP version tests in the
spec_version_checker
plugin (src/pytest_plugins/spec_version_checker/spec_version_checker.py
) but then also disable them by default infill
'spytest.ini
(L9 and L11):execution-spec-tests/pytest.ini
Lines 6 to 11 in 4783e6e
The most reasonable solution would be to simply not use this plugin when executing
fill
and have yet another pytest command dedicated to version checking that can be ran regularly in CI. The reason to not enable these tests by default is that each of these version checking tests makes a request to the Github API (to retrieve info from ethereum/EIPs); without authentication setup,fill
quickly hits the Github API rate limit.Note, as of 2024-04-29, eip version checking is currently entirely unused in our CI. To run these tests, it's required to explicitly select tests marked as
eip_version_check
with:The text was updated successfully, but these errors were encountered: