About testmon on Testmon

Hidden test dependencies

Wed, 10 Jul 2019 00:00:00 +0000

We all know each one of our tests should be independent, self-contained and therefore able to run by itself or in parallel with other tests with deterministic results. However in real life we many times encounter tests that sometimes pass and sometimes fail. There are many different sources of non-deterministic outcomes of tests. In this post, I would like to specifically focus on tests that depend on a global state which they don’t control for sufficiently.

How to set-up and use py.test in Pycharm

Tue, 10 Dec 2019 00:00:00 +0000

In this article, we’ll take a look and set-up pytest framework in PyCharm using our small example project. We’ll also introduce and try two very useful pytest-plugins to run tests more flexibly and effectively to save our time and CI/CD resources.

MasterMind – example project

For the sake of this article, we’ve prepared a small project called MasterMind. It’s a simple console-based game where the computer generates a random number that you have to guess.

Determining affected tests

Wed, 11 Dec 2019 00:00:00 +0000

Automatically determining affected tests sounds too good to be true. Python developers rightfully have a suspecting attitude towards any tool which tries to be too clever about their source code. Code completion and symbol searching doesn’t need to be 100% reliable but messing with the test suite execution? This page explains what testmon tries and what it does not try to achieve.

There is no heuristics involved. testmon works with these pretty solid assumptions:

New in testmon 1.0.0

Thu, 12 Dec 2019 00:00:00 +0000

Significant portions of testmon have been rewritten for v 1.0.1. Although the UI is mostly the same, there are some significant differences.

End of python 2.7 support

Testmon requires python 3.6 or higher and pytest 5 or higher.

No subprocess measurement

–testmon-singleprocess was removed (because it’s the only option now)

It’s not easy to automate the setup of subprocess measurement and therefore to automate testing it. We’re not sure if it worked outside of our environment and if anybody used it. If you miss subprocess measurement, please let us know.

Version 1.1 is out!

Fri, 19 Mar 2021 00:00:00 +0000

I’m very glad to announce pytest-testmon version 1.1 is out and up on PyPI.

The new feature is lifting a basic limitation of testmon.

There are many things influencing the outcome of a test. Before version 1.1 testmon kept track of code inside tested project and configured environment variables. With version 1.1 comes the ability to detect changes of packages(libraries). With each run testmon quickly checks the version of all installed packages. If any of the versions change, or if there is a new library added, previous outcomes of tests are deemed invalid and therefore they are re-executed :).

Version 1.2 is out!

Wed, 15 Sep 2021 00:00:00 +0000

I’m very glad to announce pytest-testmon version 1.2 is out on PyPI.

In previous versions pytest-testmon stored the failure reports and re-reported them from cache even if tests were not affected. The result was imperfect in many ways and sometimes spooked users. This version always re-executes tests which failed last time.

Database schema was updated to version 8, thus previous versions of pytest-testmon are not compatible with 1.2.0.

New features in 1.3.x

Thu, 03 Mar 2022 00:00:00 +0000

The 1.3.x line of pytest-testmon brought a lot of functional and compatiblity improvements:

much smaller .testmondata saving space and improving performance
doctest and doctest modules support
support for pytest 7
--testmon-noselect now sorts the tests! All the tests are executed but the affected go first. Unaffected tests only follow afterwards. Quickest tests are still prioritized within each subgroup.
compatibility with Python 3.11

v1.4 with xdist support is out!

Sat, 01 Oct 2022 00:00:00 +0000

For many years it hasn’t been possible to use pytest-testmon and run tests in parallel via pytest-xdist. With pytest-testmon=>1.4.0 the support has recently been added.

Test parallelization is probably the most general way to speed up a large test suite so this was a major limitation, and I’m glad I was able to lift it. With this addition pytest-testmon will be more relevant for the intended target audience.

We’ve been also able to fix a bug when executing pytest-testmon on Windows. Testmon should work on Windows now, but we’re building more tests to be sure and officially support Windows. Bug reports much appreciated!

How to make your GitHub Actions builds quicker?

Tue, 04 Apr 2023 00:00:00 +0000

This post is outdated

The recommended GH action is not maintained anymore.

Most of its functionality was added to the standard setup-python action, see https://github.com/actions/setup-python#caching-packages-dependencies.

There is also a new tool to use testmon in CI, see https://www.testmon.net/

Some more thouhgs: here

For many projects creating the Python environment and installing dependencies takes a lot of time. For local development environment this happens occasionally and is usually not a big problem. However, in your CI, this happens for every build and might be a significant part of the build time. GH Actions does have a caching mechanism which is used in various ways.

Version 2.0 is out!

Tue, 04 Apr 2023 00:00:00 +0000

I’m glad to announce the release of version 2.0 of pytest-testmon.

Most changes compared to version 1.4.5 are internal. They will allow better maintenance and the ability to add new features.

Testmon now obtains file checksums from git, if possible. There are a couple of fixes regarding basic stats, and the issue of pytest reporting a negative number of deselected tests has been resolved.

CI environments and testmon

Wed, 15 Jan 2025 00:00:00 +0000

Thoughts on CI systems

The modern CI services (GitHub Actions, GitLab Pipelines, etc.) model is very unsatisfactory. They are:

proprietary and enforce vendor lock-in
configured/programmed via YAML
hard or impossible to run/debug locally
unable to execute without a commit/push
misnamed (CI is neither about integration nor truly continuous)

I see clear overlap between automated QA, build systems, CI systems, and the broader OS/platform build-and-execute layer. The logical slicing and composition of these concerns should look very different from today’s model, which revolves around two major players locking everybody into their proprietary systems, leaving universality, simplicity, and developer experience as afterthoughts. This makes me hesitant to invest much into these platforms.