-## This will eventually do to wiki.debian.org/RegressionTestFramework
+## This will eventually do to wiki.debian.org/DebTestFramework
* '''Created''': <<Date(2010-10-07)>>
- * '''Contributors''': MichaelHanke
+ * '''Contributors''': MichaelHanke, YaroslavHalchenko
* '''Packages affected''':
* '''See also''':
== Summary ==
-This specification describes conventions and tools that allow Debian to
-distribute and run regression test batteries developed by upstream or
-Debian developers in a uniform fashion.
+This specification describes DebTest framework consisting of
+conventions and tools that would allow Debian to distribute test
+batteries developed by upstream or Debian developers. DebTest will
+enable an extensive testing of a deployed Debian system or a
+particular software of interest in a uniform fashion.
== Rationale ==
platform. However, especially for complex software, these test suites are often
resource hungry (CPU time, memory, diskspace, network bandwidth) and cannot be
ran at package build time by buildds. Consequently, test suites are typically
-only utilized by the respective packager on a particular machine, before
+utilized manually only by the respective packager on a particular machine, before
uploading a new version to the archive.
-However, Debian is an integrated system and packaged software is typically made
-to rely on functionality provided by other Debian packages (e.g. shared
+However, Debian is an integrated system and packaged software typically
+relies on functionality provided by other Debian packages (e.g. shared
libraries) instead of shipping duplicates with different versions in every
package -- for many good reasons. Unfortunately, there is also a downside to
-this: Debian packages often use 3rd-party tools with different versions than
+this: Debian packages often use versions of 3rd-party tools different from
those tested by upstream, and moreover, the actual versions of dependencies
-might change frequently between to subsequent uploads of a package. Currently
+might change frequently between subsequent uploads of a dependent package. Currently
a change in a dependency that introduces an incompatibility cannot be detected
-reliably (before users have filed a bug report) -- even if upstream provides a
-test suite that would have caught the breakage. Although there are archive-wide
+reliably even if upstream provides a test suite that would have caught
+the breakage. Therefore integration testing heavily relies on users to detect
+incorrect functioning and file bug reports. Although there are archive-wide
QA efforts (e.g. constantly rebuilding all packages) these tests can only
detect API/ABI breakage or functionality tested during build-time checks --
they are not exhaustive for the aforementioned reasons.
This is a proposal to, first of all, package upstream test suites in a way that
they can be used to run expensive archive-wide QA tests. However, this is also
-a proposal to establish means to test interactions of software from multiple
-Debian packages and test proper, continued, integration into the Debian system.
+a proposal to establish means to test interactions between software from multiple
+Debian packages to provide more thorough continued integration and regression testing
+for the Debian systems.
== Use Cases ==
* Moritz is a member of the security team. Whenever he applies a patch to fix
- a security issue he wants to make sure that the behavior of the software
+ a security issue he wants to make sure that the generic behavior of the software
remains unchanged. However, in general he only has access to test cases that
are included in the source package (if any). In the absence of proper tests
he can only either assume that is would work (bad by design), or rely on the
perform comprehensive testing on his own and release the fixed package as
soon as the tests pass.
- * Michael is a Debian package maintainer that takes care of three packages
- each providing a data format conversion utility. While all three tools have
- their merits there is also lots of overlap. For example, given a particular
- data file they should all generate identical output. With a Debian
- regression test framework, Michael can write and package cross-package test
- suites to ensure that this promise is fulfilled at any time. Moreover,
- Michael can also develop/package "pipeline" tests that ensure proper
- functioning of multi-stage/package processing pipelines. For example,
- testing a whole processing stream from a raw data format conversion, feeding
- into an analysis to into a visualization package.
+ * Michael is a Debian package maintainer that takes care of three
+ packages each providing a data format conversion utility. While
+ all three tools have their merits there is also lots of
+ overlap. For example, given a particular data file they should all
+ generate identical output. With a DebTest framework, Michael can
+ write and package cross-package test suites to ensure that this
+ promise is fulfilled at any time. Moreover, Michael can also
+ develop/package "pipeline" tests that ensure proper functioning of
+ multi-stage/package processing pipelines (from raw data format
+ conversion to visualization), where some stages could be
+ (re)processed using alternative tools from different software
+ packages promising to provide the same functionality. By testing
+ a whole processing stream while changing the alternative
+ implementations, breakage of the compatibility compliance could be
+ detected.
+
+ * Yarik is a Debian maintainer of a package where upstream provides
+ a complete analysis pipeline which was used for an article
+ publication. Such analysis requires relatively large array of
+ data and a range of tools from other packages to acquire
+ publication-ready summary of the results. Therefor such analysis
+ cannot be carried out at package build time. Upstream aims to
+ assure the reproducibility of the published results and encourages
+ Yarik to promise correct functioning of the research product on
+ Debian systems. Within the DebTest framework, Yarik can package
+ upstream analysis pipeline along with the target results to assure
+ reproducibility of the scientific findings.
* Albert is a scientist using Debian for his research activities. The
developers of his favorite software tell him to rather use the GreenPants
distribution, because they cannot guarantee that their software works
- properly on Debian. The reason their giving is that Debian has a different
+ properly on Debian. They reason that Debian has a different
version of a numerical library that hasn't been "tested" by the authors.
- With packaged regression test suites Albert can run, at any given point,
+ With packaged regression test suites Albert can install and run, at any given point,
a complete test of his Debian system to ensure that everything is working
properly given the exact set of base libraries installed at this very moment.
This includes the test suite of the authors of his favorite software, but
- also all distribution test suites written by Debian developers (see above).
-
- * Finally, Lucas likes to run all kinds of tests on all packages in the Debian
- archive. However, they are mostly concerned with individual packages. A
- Debian-wide regression test framework would allow Lucas to execute complex
- tests (suites for individual packages, interoperability tests, or
- comparative) in an automated fashion, and file bug reports against the
- respective packages whenever something fails.
+ also all distribution test suites provided by Debian developers (see above).
+
+ * Joerg maintains a repository of backports of Debian packages to be
+ installed in a stable environment. He wants to assure that
+ backporting of the packages has not caused a deviation in their
+ intended functioning. By using existing DebTest tests suites he
+ could verify that backported versions of the packages do not break
+ the stability and function as promised within the stable
+ environment.
+
+ * Mark creates a Debian-derived distribution and wants to acquire a
+ large userbase by promising a correctly functioning user-friendly
+ environment. Unfortunately he has no resources to provide adequate
+ testing of all the packages through the lifetime of his
+ derivative. With DebTest he acquires automated assurance that the
+ "cut" of Debian distribution he relies upon is functioning
+ correctly alongside with his additions to the distribution.
+
+ * Finally, Lucas has access to a powerful computing facility and
+ likes to run all kinds of tests on all packages in the Debian archive.
+ A Debian-wide regression test framework would allow Lucas to execute
+ complex test collections (suites for individual packages,
+ interoperability tests, or comparative) in an automated fashion,
+ and file bug reports against the respective packages whenever a
+ malfunction is detected.
== Scope ==
A specification should be built with the following considerations:
- * The person implementing it may not be the person writing it. It should be
+ * The person implementing it may not be the person writing it. Specification should be
* clear enough for someone to be able to read it and have a clear path
- * towards implementing it. If it doesn't, it needs more detail.
+ * towards implementing it. If it is not straightforward, it needs more detail.
- * That the use cases covered in the specification should be practical
+ * Use cases covered in the specification should be practical
* situations, not contrived issues.
* Limitations and issues discovered during the creation of a specification
projects / neurodebian.git / commitdiff