1 ## This will eventually do to wiki.debian.org/DebTestFramework
3 * '''Created''': <<Date(2010-10-07)>>
4 * '''Contributors''': MichaelHanke, YaroslavHalchenko
5 * '''Packages affected''':
10 This specification describes DebTest framework consisting of
11 conventions and tools that would allow Debian to distribute test
12 batteries developed by upstream or Debian developers. DebTest will
13 enable an extensive testing of a deployed Debian system or a
14 particular software of interest in a uniform fashion.
18 Ideally software packaged for Debian comes with an exhaustive test suite that
19 can be used to determine whether this software works as expected on the Debian
20 platform. However, especially for complex software, these test suites are often
21 resource hungry (CPU time, memory, diskspace, network bandwidth) and cannot be
22 ran at package build time by buildds. Consequently, test suites are typically
23 utilized manually only by the respective packager on a particular machine, before
24 uploading a new version to the archive.
26 However, Debian is an integrated system and packaged software typically
27 relies on functionality provided by other Debian packages (e.g. shared
28 libraries) instead of shipping duplicates with different versions in every
29 package -- for many good reasons. Unfortunately, there is also a downside to
30 this: Debian packages often use versions of 3rd-party tools different from
31 those tested by upstream, and moreover, the actual versions of dependencies
32 might change frequently between subsequent uploads of a dependent package. Currently
33 a change in a dependency that introduces an incompatibility cannot be detected
34 reliably even if upstream provides a test suite that would have caught
35 the breakage. Therefore integration testing heavily relies on users to detect
36 incorrect functioning and file bug reports. Although there are archive-wide
37 QA efforts (e.g. constantly rebuilding all packages) these tests can only
38 detect API/ABI breakage or functionality tested during build-time checks --
39 they are not exhaustive for the aforementioned reasons.
41 This is a proposal to, first of all, package upstream test suites in a way that
42 they can be used to run expensive archive-wide QA tests. However, this is also
43 a proposal to establish means to test interactions between software from multiple
44 Debian packages to provide more thorough continued integration and regression testing
45 for the Debian systems.
49 * Moritz is a member of the security team. Whenever he applies a patch to fix
50 a security issue he wants to make sure that the generic behavior of the software
51 remains unchanged. However, in general he only has access to test cases that
52 are included in the source package (if any). In the absence of proper tests
53 he can only either assume that is would work (bad by design), or rely on the
54 respective package maintainer to run the appropriate tests (introduces
55 delays). A packaged exhaustive regression test suite would allow Moritz to
56 perform comprehensive testing on his own and release the fixed package as
57 soon as the tests pass.
59 * Michael is a Debian package maintainer that takes care of three
60 packages each providing a data format conversion utility. While
61 all three tools have their merits there is also lots of
62 overlap. For example, given a particular data file they should all
63 generate identical output. With a DebTest framework, Michael can
64 write and package cross-package test suites to ensure that this
65 promise is fulfilled at any time. Moreover, Michael can also
66 develop/package "pipeline" tests that ensure proper functioning of
67 multi-stage/package processing pipelines (from raw data format
68 conversion to visualization), where some stages could be
69 (re)processed using alternative tools from different software
70 packages promising to provide the same functionality. By testing
71 a whole processing stream while changing the alternative
72 implementations, breakage of the compatibility compliance could be
75 * Yarik is a Debian maintainer of a package where upstream provides
76 a complete analysis pipeline which was used for an article
77 publication. Such analysis requires relatively large array of
78 data and a range of tools from other packages to acquire
79 publication-ready summary of the results. Therefor such analysis
80 cannot be carried out at package build time. Upstream aims to
81 assure the reproducibility of the published results and encourages
82 Yarik to promise correct functioning of the research product on
83 Debian systems. Within the DebTest framework, Yarik can package
84 upstream analysis pipeline along with the target results to assure
85 reproducibility of the scientific findings.
87 * Albert is a scientist using Debian for his research activities. The
88 developers of his favorite software tell him to rather use the GreenPants
89 distribution, because they cannot guarantee that their software works
90 properly on Debian. They reason that Debian has a different
91 version of a numerical library that hasn't been "tested" by the authors.
92 With packaged regression test suites Albert can install and run, at any given point,
93 a complete test of his Debian system to ensure that everything is working
94 properly given the exact set of base libraries installed at this very moment.
95 This includes the test suite of the authors of his favorite software, but
96 also all distribution test suites provided by Debian developers (see above).
98 * Joerg maintains a repository of backports of Debian packages to be
99 installed in a stable environment. He wants to assure that
100 backporting of the packages has not caused a deviation in their
101 intended functioning. By using existing DebTest tests suites he
102 could verify that backported versions of the packages do not break
103 the stability and function as promised within the stable
106 * Mark creates a Debian-derived distribution and wants to acquire a
107 large userbase by promising a correctly functioning user-friendly
108 environment. Unfortunately he has no resources to provide adequate
109 testing of all the packages through the lifetime of his
110 derivative. With DebTest he acquires automated assurance that the
111 "cut" of Debian distribution he relies upon is functioning
112 correctly alongside with his additions to the distribution.
114 * Finally, Lucas has access to a powerful computing facility and
115 likes to run all kinds of tests on all packages in the Debian archive.
116 A Debian-wide regression test framework would allow Lucas to execute
117 complex test collections (suites for individual packages,
118 interoperability tests, or comparative) in an automated fashion,
119 and file bug reports against the respective packages whenever a
120 malfunction is detected.
124 This specification is applicable to all Debian packages, and Debian as a whole.
128 A specification should be built with the following considerations:
130 * The person implementing it may not be the person writing it. Specification should be
131 * clear enough for someone to be able to read it and have a clear path
132 * towards implementing it. If it is not straightforward, it needs more detail.
134 * Use cases covered in the specification should be practical
135 * situations, not contrived issues.
137 * Limitations and issues discovered during the creation of a specification
138 * should be clearly pointed out so that they can be dealt with explicitly.
140 * If you don't know enough to be able to competently write a spec, you should
141 * either get help or research the problem further. Avoid spending time making
142 * up a solution: base yourself on your peers' opinions and prior work.
144 Specific issues related to particular sections are described further below.
146 === Implementation Plan ===
148 This section is usually broken down into subsections, such as the packages
149 being affected, data and system migration where necessary, user interface
150 requirements and pictures (photographs of drawings on paper work well).
154 To implement a specification, the developer should observe the use cases
155 carefully, and follow the design specified. He should make note of places in
156 which he has strayed from the design section, adding rationale describing why
157 this happened. This is important so that next iterations of this specification
158 (and new specifications that touch upon this subject) can use the specification
161 The implementation is very dependent on the type of feature to be implemented.
162 Refer to the team leader for further suggestions and guidance on this topic.
164 * Organization of the framework
165 - packages might register ways to run basic tests against installed
176 - assess resources/performance:
179 - single package tests
180 - all (with -f to force even if resources are not sufficient)
181 - given specific resources demands, just run
182 the ones matching those
184 * Customization/Output:
186 + output: some structured output
187 + interface to some dashboard
189 * Debug symbols: ....
190 - do not strip symbols from test binary
193 * Implementation language:
194 - Python unless someone takes the burden to develop
195 and maintain for upcoming years.
197 == Outstanding Issues ==
199 The specification process requires experienced people to drive it. More
200 documentation on the process should be produced.
202 The drafting of a specification requires english skills and a very good
203 understanding of the problem. It must also describe things to an extent that
204 someone else could implement. This is a difficult set of conditions to ensure
205 throughout all the specifications added.
207 There is a lot of difficulty in gardening obsolete, unwanted and abandoned
208 specifications in the Wiki.
210 == BoF agenda and discussion ==
212 Possible meetings where this specification will be discussed.