X-Git-Url: https://git.donarmstrong.com/?a=blobdiff_plain;f=sandbox%2Fproposal_regressiontestframwork.moin;h=983a624c5d91550e3f7c12deaec304a560a8f373;hb=HEAD;hp=d1a8d6743d1f663a03cfb2af417bbc9ae0d431f6;hpb=5cfa0c672d486e875e54b622137594b5d2295467;p=neurodebian.git diff --git a/sandbox/proposal_regressiontestframwork.moin b/sandbox/proposal_regressiontestframwork.moin index d1a8d67..983a624 100644 --- a/sandbox/proposal_regressiontestframwork.moin +++ b/sandbox/proposal_regressiontestframwork.moin @@ -9,39 +9,40 @@ This specification describes DebTest -- a framework with conventions and tools that allow Debian to distribute test batteries developed by upstream or Debian -developers. DebTest will enable an extensive testing of a deployed Debian -system or a particular software of interest in a uniform fashion. +developers. DebTest aims to enable developers and users to perform extensive +testing of a deployed Debian system or a particular software of interest in a +uniform fashion. == Rationale == Ideally software packaged for Debian comes with an exhaustive test suite that -can be used to determine whether this software works as expected on the Debian -platform. However, especially for complex software, these test suites are often -resource hungry (CPU time, memory, diskspace, network bandwidth) and cannot be -ran at package build time by buildds. Consequently, test suites are typically -utilized manually only by the respective packager on a particular machine, before -uploading a new version to the archive. - -However, Debian is an integrated system and packaged software typically -relies on functionality provided by other Debian packages (e.g. shared -libraries) instead of shipping duplicates with different versions in every -package -- for many good reasons. Unfortunately, there is also a downside to -this: Debian packages often use versions of 3rd-party tools different from -those tested by upstream, and moreover, the actual versions of dependencies -might change frequently between subsequent uploads of a dependent package. Currently +can be used to determine whether this particular software works as expected on +the Debian platform. However, especially for complex software, these test +suites are often resource hungry (CPU time, memory, disk space, network +bandwidth) and cannot be ran at package build time by buildds. Consequently, +test suites are typically utilized manually and only by the respective packager +on a particular machine, before uploading a new version to the archive. + +However, Debian is an integrated system and packaged software typically relies +on functionality provided by other Debian packages (e.g. shared libraries) +instead of shipping duplicates with different versions in every package -- for +many good reasons. Unfortunately, there is also a downside to this: Debian +packages often use versions of 3rd-party tools that are different from those +tested by upstream, and moreover, the actual versions of dependencies might +change frequently between subsequent uploads of a dependent package. Currently a change in a dependency that introduces an incompatibility cannot be detected -reliably even if upstream provides a test suite that would have caught -the breakage. Therefore integration testing heavily relies on users to detect -incorrect functioning and file bug reports. Although there are archive-wide -QA efforts (e.g. constantly rebuilding all packages) these tests can only -detect API/ABI breakage or functionality tested during build-time checks -- -they are not exhaustive for the aforementioned reasons. +reliably even if upstream provides a test suite that would have caught the +breakage. Therefore integration testing heavily relies on users to detect +incorrect functioning and file bug reports. Although there are archive-wide QA +efforts (e.g. constantly rebuilding all packages) these tests can only detect +API/ABI breakage or functionality tested during build-time checks -- they are +not exhaustive for the aforementioned reasons. This is a proposal to, first of all, package upstream test suites in a way that they can be used to run expensive archive-wide QA tests. However, this is also -a proposal to establish means to test interactions between software from multiple -Debian packages to provide more thorough continued integration and regression testing -for the Debian systems. +a proposal to establish means to test interactions between software from +multiple Debian packages to provide more thorough continued integration and +regression testing for the Debian systems. == Use Cases == @@ -94,6 +95,12 @@ for the Debian systems. This includes the test suite of the authors of his favorite software, but also all distribution test suites provided by Debian developers (see above). + * Sylvestre maintains a core computational library in Debian. + A new version (or other modification) of this library promises performance + advantages. Using DebTest he could not only verify the absence of + regressions but also to obtain direct performance comparison + against the previous version across a range of applications. + * Joerg maintains a repository of backports of Debian packages to be installed in a stable environment. He wants to assure that backporting of the packages has not caused a deviation in their @@ -102,13 +109,19 @@ for the Debian systems. the stability and function as promised within the stable environment. - * Mark creates a Debian-derived distribution and wants to acquire a - large userbase by promising a correctly functioning user-friendly - environment. Unfortunately he has no resources to provide adequate - testing of all the packages through the lifetime of his - derivative. With DebTest he acquires automated assurance that the - "cut" of Debian distribution he relies upon is functioning - correctly alongside with his additions to the distribution. + * Mark wants to create a Debian-derived distribution and needs to + modify a number of essential packages in order to achieve the desired + improvements. He hopes that these changes do not break other Debian + packages, but he is not really sure. A comprehensive test battery for the + whole Debian system would offer him a way to verify proper functioning + of his modified snapshot of Debian -- without having to manually replicate + the testing efforts done by thousands of Debian contributors. + + * Linus is an upstream developer. He just loves the fact that he can tell any + of his Debian-based users to just 'apt-get install' something and send him + the output of a debtest command, whenever they claim that his software + doesn't work properly. It pleases him to see his carefully developed test + suite to be conveniently accessible for users. * Finally, Lucas has access to a powerful computing facility and likes to run all kinds of tests on all packages in the Debian archive. @@ -116,7 +129,11 @@ for the Debian systems. complex test collections (suites for individual packages, interoperability tests, or comparative) in an automated fashion, and file bug reports against the respective packages whenever a - malfunction is detected. + malfunction is detected. Some of Lucas friends are not brave enough to file + bugs, but still want to contribute. They simply run (selected) tests + on their local machines that in turn report results/logs to a Debian + dashboard server, where interested parties can get a weather report of + Debian's status. == Scope == @@ -142,6 +159,63 @@ A specification should be built with the following considerations: Specific issues related to particular sections are described further below. + +=== Core components === + + * Organization of the framework + - packages might register ways to run basic tests against installed + versions + register: + - executable? + + +==== Packaged tests ==== + + * Metainformation: + * duration: .... + * resources: + * suites: + + * Debug symbols: .... + * do not strip symbols from test binary + + * Packages that register tests might provide a virtual package + 'test-' to allow easy test discovery and retrival via + debtest tools. + + +==== debtest tools ==== + + * Invocation:: + * single package tests + * all (with -f to force even if resources are not sufficient) + * tests of dependent packages (discovered via rdepends, + "rrecommends" and "rsuggests") + * given specific resources demands, just run + the ones matching those + * Customization/Output:: + plugins:: + * job resources requirement adjustments + . manual customization + . request from dashboard for the system (or alike) + * executioners + . local execution (monitor resources) + . submit to cluster/cloud + * output/reports + . some structured output + . interfaces to dashboards + + +==== Maintainer helpers ==== + + Helpers: + - assess resources/performance: + + +=== Supplementary infrastructure === + +==== Dashboard server ==== + === Implementation Plan === This section is usually broken down into subsections, such as the packages @@ -160,6 +234,10 @@ as a reference. The implementation is very dependent on the type of feature to be implemented. Refer to the team leader for further suggestions and guidance on this topic. + * Implementation language: + - Python unless someone takes the burden to develop + and maintain for upcoming years. + == Outstanding Issues == The specification process requires experienced people to drive it. More