## This will eventually do to wiki.debian.org/RegressionTestFramework * '''Created''': <> * '''Contributors''': MichaelHanke * '''Packages affected''': * '''See also''': == Summary == This specification describes conventions and tools that allow Debian to distribute and run regression test batteries developed by upstream or Debian developers in a uniform fashion. == Rationale == Ideally software packaged for Debian comes with an exhaustive test suite that can be used to determine whether this software works as expected on the Debian platform. However, especially for complex software, these test suites are often resource hungry (CPU time, memory, diskspace, network bandwidth) and cannot be ran at package build time by buildds. Consequently, test suites are only utilized by the packager on a particular machine, before uploading a new version to the archive. However, Debian is an integrated system and packaged software is typically made to rely on functionality provided by other Debian packages (e.g. shared libraries) instead of shipping duplicates with different versions in every package -- for many good reasons. Unfortunately, there is also a downside to this: Debian packages often use 3rd-party tools with different versions than those tested by upstream, and moreover, the actual versions might change frequently between to subsequent uploads of a package. Currently a change in a dependency that introduces an incompatibility cannot be detected reliably (before users have filed a bug report) -- even if upstream provides a testsuite that would have caught the breakage. Although there are archive-wide QA efforts (e.g. constantly rebuilding all packages) these tests can only detect API/ABI breakage or functionality tested during build-time checks -- they are not exhaustive for the aforementioned reasons. This is a proposal to, first of all, package upstream test suites in a way that they can be used to run expensive archive-wide QA tests. However, this is also a proposal to establish means to test interactions of software from multiple Debian packages and test proper, continued, integration into the Debian system. == Use Cases == * Bob is the maintainer for the boot process for Debian. In the etch cycle, he would like to work on getting the boot time down to two seconds from boot manager to GDM screen. He creates an entry for the specification in the wiki, discusses it at debconf, and starts writing out a braindump of it. * Arnaud is a student participating in Google's Summer of code and wants to make sure that his project fits the short description that has been given on the ideas page. He writes a detailed spec in the wiki. His mentor can then confirm that he's on good track. The specification is published on a mailing list and people's comments help improve it even further. == Scope == This specification covers feature specifications for Debian. It is not meant as a more general specification format. == Design == A specification should be built with the following considerations: * The person implementing it may not be the person writing it. It should be * clear enough for someone to be able to read it and have a clear path * towards implementing it. If it doesn't, it needs more detail. * That the use cases covered in the specification should be practical * situations, not contrived issues. * Limitations and issues discovered during the creation of a specification * should be clearly pointed out so that they can be dealt with explicitly. * If you don't know enough to be able to competently write a spec, you should * either get help or research the problem further. Avoid spending time making * up a solution: base yourself on your peers' opinions and prior work. Specific issues related to particular sections are described further below. === Summary === The summary should not attempt to say '''why''' the spec is being defined, just '''what''' is being specified. === Rationale === This should be the description of '''why''' this spec is being defined. === Scope and Use Cases === While not always required, but in many cases they bring much better clarity to the scope and scale of the specification than could be obtained by talking in abstract terms. === Implementation Plan === This section is usually broken down into subsections, such as the packages being affected, data and system migration where necessary, user interface requirements and pictures (photographs of drawings on paper work well). == Implementation == To implement a specification, the developer should observe the use cases carefully, and follow the design specified. He should make note of places in which he has strayed from the design section, adding rationale describing why this happened. This is important so that next iterations of this specification (and new specifications that touch upon this subject) can use the specification as a reference. The implementation is very dependent on the type of feature to be implemented. Refer to the team leader for further suggestions and guidance on this topic. == Outstanding Issues == The specification process requires experienced people to drive it. More documentation on the process should be produced. The drafting of a specification requires english skills and a very good understanding of the problem. It must also describe things to an extent that someone else could implement. This is a difficult set of conditions to ensure throughout all the specifications added. There is a lot of difficulty in gardening obsolete, unwanted and abandoned specifications in the Wiki. == BoF agenda and discussion == Possible meetings where this specification will be discussed. ---- CategorySpec