12、pytest -- 缓存:记录执行的状态
luizyao 人气:1目录
- 1.
cacheprovider
插件- 1.1.
--lf, --last-failed
:只执行上一轮失败的用例 - 1.2.
--ff, --failed-first
:先执行上一轮失败的用例,再执行其它的 - 1.3.
--nf, --new-first
:先执行新加的或修改的用例,再执行其它的 - 1.4.
--cache-clear
:先清除所有缓存,再执行用例 - 1.5. 如果上一轮没有失败的用例
- 1.1.
- 2.
config.cache
对象 - 3.
Stepwise
往期索引:https://www.cnblogs.com/luizyao/p/11771740.html
pytest
会将本轮测试的执行状态写入到.pytest_cache
文件夹,这个行为是由自带的cacheprovider
插件来实现的;
注意:
pytest
默认将测试执行的状态写入到根目录中的.pytest_cache
文件夹,我们也可以通过在pytest.ini
中配置cache_dir
选项来自定义缓存的目录,它可以是相对路径,也可以是绝对路径;相对路径指的是相对于
pytest.ini
文件所在的目录;例如,我们把这一章的缓存和源码放在一起:在
src/chapter-12/pytest.ini
中添加如下配置:[pytest] cache_dir = .pytest-cache
这样,即使我们在项目的根目录下执行
src/chapter-12/
中的用例,也只会在pytest-chinese-doc/src/chapter-12/.pytest_cache
中生成缓存,而不再是pytest-chinese-doc/.pytest_cache
中;pytest-chinese-doc (5.1.3) λ pipenv run pytest src/chapter-12
1. cacheprovider
插件
在介绍这个插件之前,我们先看一个简单例子:
# src/chapter-12/test_failed.py
import pytest
@pytest.mark.parametrize('num', [1, 2])
def test_failed(num):
assert num == 1
# src\chapter-12\test_pass.py
def test_pass():
assert 1
我们有两个简单的测试模块,首先我们来执行一下它们:
λ pipenv run pytest -q src/chapter-12/
.F. [100%]
=============================== FAILURES ================================
____________________________ test_failed[2] _____________________________
num = 2
@pytest.mark.parametrize('num', [1, 2])
def test_failed(num):
> assert num == 1
E assert 2 == 1
src\chapter-12\test_failed.py:27: AssertionError
1 failed, 2 passed in 0.08s
可以看到一共收集到三个测试用例,其中有一个失败,另外两个成功的,并且两个执行成功的用例分属不同的测试模块;
同时,pytest
也在src/chapter-12/
的目录下生成缓存文件夹(.pytest_cache
),具体的目录结构如下所示:
src
├───chapter-12
│ │ pytest.ini # 配置了 cache_dir = .pytest-cache
│ │ test_failed.py
│ │ test_pass.py
│ │
│ └───.pytest-cache
│ │ .gitignore
│ │ CACHEDIR.TAG
│ │ README.md
│ │
│ └───v
│ └───cache
│ lastfailed
│ nodeids
│ stepwise
现在,我们就结合上面的组织结构,具体介绍一下cacheprovider
插件的功能;
1.1. --lf, --last-failed
:只执行上一轮失败的用例
缓存中的lastfailed
文件记录了上次失败的用例ID
,我们可以通过一下--cache-show
命令查看它的内容:
--cache-show
命令也是cacheprovider
提供的新功能,它不会导致任何用例的执行;
λ pipenv run pytest src/chapter-12/ -q --cache-show 'lastfailed'
cachedir: D:\Personal Files\Projects\pytest-chinese-doc\src\chapter-12\.pytest-cache
--------------------- cache values for 'lastfailed' ---------------------
cache\lastfailed contains:
{'test_failed.py::test_failed[2]': True}
no tests ran in 0.01s
我们可以看到,它记录了一个用例,为上次失败的测试用例的ID
:test_failed.py::test_failed[2]
;
下次执行时,当我们使用--lf
选项,pytest
在收集阶段只会选择这个失败的用例,而忽略其它的:
λ pipenv run pytest --lf --collect-only src/chapter-12/
========================== test session starts ==========================
platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
cachedir: .pytest-cache
rootdir: D:\Personal Files\Projects\pytest-chinese-doc\src\chapter-12, inifile: pytest.ini
collected 2 items / 1 deselected / 1 selected
<Module test_failed.py>
<Function test_failed[2]>
run-last-failure: rerun previous 1 failure (skipped 2 files)
========================= 1 deselected in 0.02s =========================
我们仔细观察一下上面的回显,有一句话可能会让我们有点困惑:collected 2 items / 1 deselected / 1 selected
,可我们明明有三个用例,怎么会只收集到两个呢?
实际上,--lf
复写了用例收集阶段的两个钩子方法:pytest_ignore_collect(path, config)
和pytest_collection_modifyitems(session, config, items)
;
我们来先看看pytest_ignore_collect(path, config)
,如果它的结果返回True
,就忽略path
路径中的用例;
# _pytest/cacheprovider.py
def last_failed_paths(self):
"""Returns a set with all Paths()s of the previously failed nodeids (cached).
"""
try:
return self._last_failed_paths
except AttributeError:
rootpath = Path(self.config.rootdir)
result = {rootpath / nodeid.split("::")[0] for nodeid in self.lastfailed}
result = {x for x in result if x.exists()}
self._last_failed_paths = result
return result
def pytest_ignore_collect(self, path):
"""
Ignore this file path if we are in --lf mode and it is not in the list of
previously failed files.
"""
if self.active and self.config.getoption("lf") and path.isfile():
last_failed_paths = self.last_failed_paths()
if last_failed_paths:
skip_it = Path(path) not in self.last_failed_paths()
if skip_it:
self._skipped_files += 1
return skip_it
可以看到,如果当前收集的文件,不在上一次失败的路径集合内,就会忽略这个文件,所以这次执行就不会到test_pass.py
中收集用例了,故而只收集到两个用例;并且pytest.ini
也在忽略的名单上,所以实际上是跳过两个文件:(skipped 2 files)
;
至于pytest_collection_modifyitems(session, config, items)
钩子方法,我们在下一节和--ff
命令一起看;
1.2. --ff, --failed-first
:先执行上一轮失败的用例,再执行其它的
我们先通过实践看看这个命令的效果,再去分析它的实现:
λ pipenv run pytest --collect-only -s --ff src/chapter-12/
========================== test session starts ==========================
platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
cachedir: .pytest-cache
rootdir: D:\Personal Files\Projects\pytest-chinese-doc\src\chapter-12, inifile: pytest.ini
collected 3 items
<Module test_failed.py>
<Function test_failed[2]>
<Function test_failed[1]>
<Module test_pass.py>
<Function test_pass>
run-last-failure: rerun previous 1 failure first
========================= no tests ran in 0.02s =========================
我们可以看到一共收集到三个测试用例,和正常的收集顺序相比,上一轮失败的test_failed.py::test_failed[2]
用例在最前面,将优先执行;
实际上,-ff
只复写了钩子方法:pytest_collection_modifyitems(session, config, items)
,它可以过滤或者重新排序收集到的用例:
# _pytest/cacheprovider.py
def pytest_collection_modifyitems(self, session, config, items):
...
if self.config.getoption("lf"):
items[:] = previously_failed
config.hook.pytest_deselected(items=previously_passed)
else: # --failedfirst
items[:] = previously_failed + previously_passed
...
可以看到,如果使用的是lf
,就把之前成功的用例状态置为deselected
,这轮执行就会忽略它们;如果使用的是-ff
,只是将之前失败的用例,顺序调到前面;
另外,我们也可以看到lf
的优先级要高于ff
,所以它们同时使用的话,ff
是不起作用的;
1.3. --nf, --new-first
:先执行新加的或修改的用例,再执行其它的
缓存中的nodeids
文件记录了上一轮执行的所有的用例:
λ pipenv run pytest src/chapter-12 --cache-show 'nodeids'
========================== test session starts ==========================
platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
cachedir: .pytest-cache
rootdir: D:\Personal Files\Projects\pytest-chinese-doc\src\chapter-12, inifile: pytest.ini
cachedir: D:\Personal Files\Projects\pytest-chinese-doc\src\chapter-12\.pytest-cache
---------------------- cache values for 'nodeids' -----------------------
cache\nodeids contains:
['test_failed.py::test_failed[1]',
'test_failed.py::test_failed[2]',
'test_pass.py::test_pass']
========================= no tests ran in 0.01s =========================
我们看到上一轮共执行了三个测试用例;
现在我们在test_pass.py
中新加一个用例,并修改一下test_failed.py
文件中的用例(但是不添加新用例):
# src\chapter-12\test_pass.py
def test_pass():
assert 1
def test_new_pass():
assert 1
现在我们再来执行一下收集命令:
λ pipenv run pytest --collect-only -s --nf src/chapter-12/
========================== test session starts ==========================
platform win32 -- Python 3.7.3, pytest-5.1.3, py-1.8.0, pluggy-0.13.0
cachedir: .pytest-cache
rootdir: D:\Personal Files\Projects\pytest-chinese-doc\src\chapter-12, inifile: pytest.ini
collected 4 items
<Module test_pass.py>
<Function test_new_pass>
<Module test_failed.py>
<Function test_failed[1]>
<Function test_failed[2]>
<Module test_pass.py>
<Function test_pass>
========================= no tests ran in 0.03s =========================
可以看到,新加的用例顺序在最前面,其次修改过的测试用例紧接其后,最后才是旧的用例;这个行为在源码中有所体现:
# _pytest/cacheprovider.py
def pytest_collection_modifyitems(self, session, config, items):
if self.active:
new_items = OrderedDict()
other_items = OrderedDict()
for item in items:
if item.nodeid not in self.cached_nodeids:
new_items[item.nodeid] = item
else:
other_items[item.nodeid] = item
items[:] = self._get_increasing_order(
new_items.values()
) + self._get_increasing_order(other_items.values())
self.cached_nodeids = [x.nodeid for x in items if isinstance(x, pytest.Item)]
def _get_increasing_order(self, items):
return sorted(items, key=lambda item: item.fspath.mtime(), reverse=True)
item.fspath.mtime()
代表用例所在文件的最后修改时间,reverse=True
表明是倒序排列;
items[:] = self._get_increasing_order(new_items.values()) + self._get_increasing_order(other_items.values())
保证新加的用例永远在最前面;
1.4. --cache-clear
:先清除所有缓存,再执行用例
直接看源码:
# _pytest/cacheprovider.py
class Cache:
...
@classmethod
def for_config(cls, config):
cachedir = cls.cache_dir_from_config(config)
if config.getoption("cacheclear") and cachedir.exists():
rm_rf(cachedir)
cachedir.mkdir()
return cls(cachedir, config)
可以看到,它会先把已有的缓存文件夹删除(rm_rf(cachedir)
),再创建一个空的同名文件夹(cachedir.mkdir()
),这样会导致上述的功能失效,所以一般不使用这个命令;
1.5. 如果上一轮没有失败的用例
现在,我们清除缓存,再执行test_pass.py
模块(它的用例都是能测试成功的):
λ pipenv run pytest --cache-clear -q -s src/chapter-12/test_pass.py
.
1 passed in 0.01s
这时候我们再去看一下缓存目录:
.pytest-cache
└───v
└───cache
nodeids
stepwise
是不是少了什么?对!因为没有失败的用例,所以不会生成lastfailed
文件,那么这个时候在使用--lf
和--ff
会发生什么呢?我们来试试:
注意:
如果我们观察的足够仔细,就会发现现在的缓存目录和之前相比不止少了
lastfailed
文件,还少了CACHEDIR.TAG
、.gitignore
和README.md
三个文件;这是一个
bug
,我已经在pytest 5.3.1版本
上提交了issue
,预计会在之后的版本修复,如果你有兴趣深入了解一下它的成因和修复方案,可以参考这个:https://github.com/pytest-dev/pytest/issues/6290
luyao@NJ-LUYAO-T460 https://img.qb5200.com/download-x/d/Personal Files/Projects/pytest-chinese-doc (5.1.3)
λ pipenv run pytest -q -s --lf src/chapter-12/test_pass.py
.
1 passed in 0.01s
luyao@NJ-LUYAO-T460 https://img.qb5200.com/download-x/d/Personal Files/Projects/pytest-chinese-doc (5.1.3)
λ pipenv run pytest -q -s --ff src/chapter-12/test_pass.py
.
1 passed in 0.02s
可以看到,它们没有实施任何影响;为什么会这样?我们去源码里找一下答案吧;
# _pytest/cacheprovider.py
class LFPlugin:
""" Plugin which implements the --lf (run last-failing) option """
def __init__(self, config):
...
self.lastfailed = config.cache.get("cache/lastfailed", {})
...
def pytest_collection_modifyitems(self, session, config, items):
...
if self.lastfailed:
...
else:
self._report_status = "no previously failed tests, "
if self.config.getoption("last_failed_no_failures") == "none":
self._report_status += "deselecting all items."
config.hook.pytest_deselected(items=items)
items[:] = []
else:
self._report_status += "not deselecting items."
可以看到,当self.lastfailed
判断失败时,如果我们指定了last_failed_no_failures
选项为none
,pytest
会忽略所有的用例(items[:] = []
),否则不做任何修改(和没加--lf
或--ff
一样),而判断self.lastfailed
的依据是就是lastfailed
文件;
继续看看,我们会学习到一个新的命令行选项:
# _pytest/cacheprovider.py
group.addoption(
"--lfnf",
"--last-failed-no-failures",
action="store",
dest="last_failed_no_failures",
choices=("all", "none"),
default="all",
help="which tests to run with no previously (known) failures.",
)
来试试吧:
λ pipenv run pytest -q -s --ff --lfnf none src/chapter-12/test_pass.py
1 deselected in 0.01s
λ pipenv run pytest -q -s --ff --lfnf all src/chapter-12/test_pass.py
.
1 passed in 0.01s
注意:
--lfnf
的实参只支持choices=("all", "none")
;
2. config.cache
对象
我们可以通过pytest
的config
对象去访问和设置缓存中的数据;下面是一个简单的例子:
# content of test_caching.py
import pytest
import time
def expensive_computation():
print("running expensive computation...")
@pytest.fixture
def mydata(request):
val = request.config.cache.get("example/value", None)
if val is None:
expensive_computation()
val = 42
request.config.cache.set("example/value", val)
return val
def test_function(mydata):
assert mydata == 23
我们先执行一次这个测试用例:
λ pipenv run pytest -q src/chapter-12/test_caching.py
F [100%]
================================ FAILURES =================================
______________________________ test_function ______________________________
mydata = 42
def test_function(mydata):
> assert mydata == 23
E assert 42 == 23
src/chapter-12/test_caching.py:43: AssertionError
-------------------------- Captured stdout setup --------------------------
running expensive computation...
1 failed in 0.05s
这个时候,缓存中没有example/value
,将val
的值写入缓存,终端打印running expensive computation...
;
查看缓存,其中新加了一个文件:.pytest-cache/v/example/value
;
.pytest-cache/
├── .gitignore
├── CACHEDIR.TAG
├── README.md
└── v
├── cache
│ ├── lastfailed
│ ├── nodeids
│ └── stepwise
└── example
└── value
3 directories, 7 files
通过--cache-show
选项查看,发现其内容正是42
:
λ pipenv run pytest src/chapter-12/ -q --cache-show 'example/value'
cachedir: /Users/yaomeng/Private/Projects/pytest-chinese-doc/src/chapter-12/.pytest-cache
-------------------- cache values for 'example/value' ---------------------
example/value contains:
42
no tests ran in 0.00s
再次执行这个用例,这个时候缓存中已经有我们需要的数据了,终端就不会再打印running expensive computation...
:
λ pipenv run pytest -q src/chapter-12/test_caching.py
F [100%]
================================ FAILURES =================================
______________________________ test_function ______________________________
mydata = 42
def test_function(mydata):
> assert mydata == 23
E assert 42 == 23
src/chapter-12/test_caching.py:43: AssertionError
1 failed in 0.04s
3. Stepwise
试想一下,现在有这么一个场景:我们想要在遇到第一个失败的用例时退出执行,并且下次还是从这个用例开始执行;
以下面这个测试模块为例:
# src/chapter-12/test_sample.py
def test_one():
assert 1
def test_two():
assert 0
def test_three():
assert 1
def test_four():
assert 0
def test_five():
assert 1
我们先执行一下测试:pipenv run pytest --cache-clear --sw src/chapter-12/test_sample.py
;
λ pipenv run pytest --cache-clear --sw -q src/chapter-12/test_sample.py
.F
================================= FAILURES =================================
_________________________________ test_two _________________________________
def test_two():
> assert 0
E assert 0
src/chapter-12/test_sample.py:28: AssertionError
!!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!
1 failed, 1 passed in 0.13s
使用--cache-clear
清除之前的缓存,使用--sw, --stepwise
使其在第一个失败的用例处退出执行;
现在我们的缓存文件中lastfailed
记录了这次执行失败的用例,即为test_two()
;nodeids
记录了所有的测试用例;特殊的是,stepwise
记录了最近一次失败的测试用例,这里也是test_two()
;
接下来,我们用--sw
的方式再次执行:pytest
首先会读取stepwise
中的值,并将其作为第一个用例开始执行;
λ pipenv run pytest --sw -q src/chapter-12/test_sample.py
F
================================= FAILURES =================================
_________________________________ test_two _________________________________
def test_two():
> assert 0
E assert 0
src/chapter-12/test_sample.py:28: AssertionError
!!!!!! Interrupted: Test failed, continuing from this test next run. !!!!!!!
1 failed, 1 deselected in 0.12s
可以看到,test_two()
作为第一个用例开始执行,在第一个失败处退出;
其实,pytest
还提供了一个--stepwise-skip
的命令行选项,它会忽略第一个失败的用例,在第二个失败处退出执行;我们来试一下:
λ pipenv run pytest --sw --stepwise-skip -q src/chapter-12/test_sample.py
F.F
=============================== FAILURES ================================
_______________________________ test_two ________________________________
def test_two():
> assert 0
E assert 0
src\chapter-12\test_sample.py:28: AssertionError
_______________________________ test_four _______________________________
def test_four():
> assert 0
E assert 0
src\chapter-12\test_sample.py:36: AssertionError
!!!!! Interrupted: Test failed, continuing from this test next run. !!!!! 2 failed, 1 passed, 1 deselected in 0.16s
这个时候,在第二个失败的用例test_four()
处退出执行,同时stepwise
文件的值也改成了"test_sample.py::test_four"
;
其实,本章所有的内容都可以在源码的
_pytest/cacheprovider.py
文件中体现,如果能结合源码学习,会有事半功倍的效果;
加载全部内容