I've made a number of edits, based on private feedback and the
on-alias responses. Clarifications:

1. My operational concern about access to the entire database was
specifically about not having a multi-gigabyte download. Nothing
else should be read into that point.

2. The relationship between the development branch and the
maintenance branches of any particular consolidation does not seem
well understood. It seems premature to conclude that a multiple
release relationship belongs in the community DTS, versus a
distribution's annotations. We can come back to this, perhaps as
some of the candidate evaluations proceed.

3. It's pretty much reality that it's going to take Sun time to
transition from one DTS to another; I would expect Sun is
obligated to develop tools of the kind Alan C meant, but I don't
believe that a Phase 2 scenario can be avoided, even with such
tools instantaneously materializing.

4. I would like to come back to Jim's point about "what DTSs do"
versus "what Bugster does" after we actually try one or two of
them. That is, unlike other evaluations, I think there's a
sandbox phase to the evaluation, where we agree that, say, a
notification alias is equivalent to both the IE and RM--or that we
don't agree, and need to formalize that ownership relation in our
requirements. (I suppose this point generalizes #2 above.)

- Stephen

----

OpenSolaris
DEFECT TRACKING SYSTEM REQUIREMENTS [DRAFT]
| Stephen Hahn


1. Summary

This document identifies and explains the requirements for a defect
tracking system (DTS) solution to be used for open development.
Software development at SMI, given that SMI is already offering
value-added products and services atop externally developed
software, has been utilizing a historical DTS that has not been
designed with an adaptable structure. The requirements are grouped
into three sets of decreasing importance. It outlines a number of
specific evaluations that will be used to determine whether there is
an existing candidate DTS capable of meeting the requirements.

2. Discussion

2.1. Constraints

As with most of the problems in transitioning from proprietary to
open development, initial inspection might lead one to conclude that
the defect tracking problem is overconstrained. In the course of
reaching a coherent and ordered set of requirements for defect
tracking, we explicitly identify some of the legacy constraints, and
eliminate their influence on a potential solution.

2.1.1. Anecdotal problems

The present DTS solution at SMI, "Bugster", is the latest in a
sequence of custom and semi-custom DTS installations. Ignoring the
specifics of its technical implementation, Bugster presents three
major problems for open development scenarios:

* Per-account charge. The current licensing of the software
requires that each active account requires a paid user license to
an external software vendor. Given that we expect on the order of
5000 active contributing developers the OpenSolaris and open JDK
communities, an ongoing variable cost of this kind is problematic.

* Performance and scalability. Similarly, the current system is
implemented as an HA single clustered installation, capable of
supporting 1200 concurrent active users.

* Scrubbing and proprietary information boundaries. The information
classification of the existing bug corpus is vague and
inconsistent across the numerous development groups within SMI.
Revisiting the current defects for possible publication is
uneconomic, other than on an as-needed basis. The lack of a clear
boundary has made some employees apprehensive about any defect
publication.

2.1.2. Integration aspects

| * Upstream/downstream. The OpenSolaris program is an additional
| step in a progression from an operating environment composed
| entirely of components created and/or maintained by a single
| organization to one constructed from a wide variety of components
| originating from multiple organizations and individuals. Where
| those components have defect state tracked in their own DTS, it
| would be economically beneficial to have that state represented in
| the community DTS used for the operating environment effort, both
| to minimize duplicate efforts and to maintain accurate status on
| the upstream component.

* Other SMI systems. In contrast to the source code management
replacement, the Bugster DTS provides data to other information
| systems within Sun Microsystems. Although the burden of
confidential data management rests with Sun (for that data it
considers confidential), the DTS API must meet or exceed the
current API provided by Bugster.

2.2. Ideal outcome sequence

Unfortunately, migrations from one DTS to another are expensive, at
least in the experience of the Solaris/OpenSolaris bug corpus. A
realistic idealized outcome would be three-phased, after the
candidate DTS is selected and implemented:

Phase 1. Two systems. Bugs could be filed in either system, and
integrations could refer to bugs in either system. Duplicates
would be encouraged to be closed in favour of the community DTS.
Work on a toolset to minimize the cost of such closures and
| migrations could begin. In principle, a Phase 1 deployment is
| possible--for recording purposes--in one to three months after
| selection. Some tools integration work is required to use the
| selected candidate for commits and defect resolution.

Phase 2. Deprecated second system. Integrations must refer only to
bugs in the community DTS, but bugs could be filed in either system.
New bug categories and classifications are created only in the
community DTS. Client toolsets and distribution-specific systems
[1] are expected to have adjusted to the community DTS capabilities.
| After a successful exit from Phase 1, Phase 2 would run until the
| legacy DTS is not deemed significant by the OpenSolaris Community.

Phase 3. Single system. All relevant public defect data is present
in the community DTS.

Another option is to have the public bug system just be basically a beta until you know that all of your tools work with both systems, and then switch. This is a lot more convenient from the viewpoint of the new DTS, because it's a lot easier to import all the bugs from an old system all at once (as a single, consistent whole) than it is to add them into an already-existing database. -Mkanat 7/30/07 11:16 PM

3. Requirements

As with previous requirements documents [2], the requirements described
below arise from a number of distinct classes: some are social, in
that the requirement is believed necessary for successful use in the
community; some are technical, in that the requirement is believed
necessary to successfully produce software in a multi-project,
multi-committer, multi-site development organization; and some are
economic, in that the requirement is attempting to describe
attributes that would limit the costs of the ongoing use of the
tool.

In an attempt to use neutral terms, we use the phrase "candidate
DTS" to describe the DTS solution we are evaluating and "legacy DTS"
to refer to the DTS solution in use (inside Sun, with partial
visibility via bugs.opensolaris.org) at present.

The requirements are ranked by necessity, using the terminology
proposed in IEEE Std 830-1998 [3]. We incorporate many of the
requirements from Wesolowski [4]. Many of the requirements are
derived from the earlier evaluation of distributed source code
management solutions [2].

3.1. "Essential" requirements

E0. Open source Bugzilla is licensed under MPL1.1 -Timeless 7/18/07 11:31 PM 


To be considered for use by the OpenSolaris community, the
candidate DTS is expected to be available in source form under
an OSI-approved license. At this point in time, the community
expects to be able to operate directly one or more distinct
instances of the DTS.

E1. Universal access Bugs in Bugzilla default to being public unless products are otherwise specially configured -Timeless 7/18/07 11:32 PM 


With the exceptions identified below, all defect data pertaining
to OpenSolaris's open components and imported open components
must be available for search, view, and modify operations to
participants. Data must become available to all users at the
same time, unless restricted by an information management
agreement (such as those used in industry collaborations around
security vulnerabilities). Groups allow for restricting access for such collaboration -Timeless 7/18/07 11:32 PM 
All Participants must be potentially
eligible for any of the generally recognised defect workflow
roles, such as responsible engineer (RE) and initial evaluator
(IE), without reference to their affiliations outside the
| Community. Anybody can hold any role in Bugzilla. -Mkanat 7/30/07 11:18 PM 


|
| Operations will use the Participant's OpenSolaris
| credentials as a representation of the Participant's identity.
| The candidate DTS should provide an API http://www.bugzilla.org/docs/tip/html/api/Bugzilla/Auth.html is the API for adding new authentication systems. -Mkanat 7/30/07 10:32 PM 
that permits the addition of other identity representations, such as OpenID [5] https://bugzilla.mozilla.org/show_bug.cgi?id=300410 -Timeless 7/18/07 11:46 PM 
.

E2. Selective differentiated access Bugzilla's group system fully fulfills this requirement. -Mkanat 7/30/07 11:20 PM 


A mechanism must exist to indicate that a defect is private to a
set of Participants, and no part of it may be made accessible to
Participants not in that set, even if the defect's subcategory
would otherwise cause it to be accessible.

This requirement is to allow for security coordination, and
similar efforts. (Requests for differentiated access are
expected to be managed by the Tools Group, the Board, or a
designee. Distributions are expected to keep customer
confidential data external to the Community DTS.)

E3. Differentiated, configurable messaging

All mail or other messages sent by the DTS to user-supplied
addresses (such as but not limited to a defect's interest list or a
subcategory's initial evaluator) containing defect data must enforce
the confidentiality of defects described in E2. Messaging on
defect transitions is expected to be configurable; one approach
would be a message template system.
Mail is sent according to group restrictions and user preferences, a user can ask to get mail but unless the groups allow for the mail, it won't be sent -Timeless 7/18/07 11:32 PM 

Messaging and notification is expected to be delivered as
| electronic mail. Support for additional protocols is, in
| general, optional; support for some form of syndication is
| desirable. Bug lists can be represented as RSS, which allows a sort of syndication. There are also IRC bots that can announce changes to bugs in IRC channels. Finally, you could also set up a mailing list that got emailed about certain changes on bugs, if you wanted. -Mkanat 7/30/07 10:33 PM 

E4. Access control enforcement

The DTS must enforce the access restrictions described in E2 and
E3. It is not the intent of this specification to require any
particular method of enforcement.

E5. Interface stability and completeness

The storage representation, command line interfaces, network
protocols, and hooks interfaces should be documented and have
some level of declared commitment.

The XML-RPC interface is considered a stable interface. -Mkanat 7/30/07 11:23 PM

There is documentation for the database schema here: http://www.ravenbrook.com/project/p4dti/tool/cgi/bugzilla-schema/  -- It also allows you to show the changes in the schema between any two versions of Bugzilla. -Mkanat 7/30/07 11:25 PM

The state of the storage
representation and the operations that modify it should be well
defined, so that use with advanced file system capabilities can
be assessed for hazards. (For example, consistent use with file
system snapshot capabilities.)
Storage is maintained in MySQL or PostgreSQL. Support for Oracle is likely to arrive soon https://bugzilla.mozilla.org/show_bug.cgi?id=189947

Generally MySQL snapshotting is done via mysqlhotcopy or mysqldump. I know sun loves to emphasize ZFS and filesystem features and I'd love to see these work well,

but I think they're slightly beyond my domain. -Timeless 7/18/07 11:57 PM 

Use of the candidate DTS with advanced file system capabilities
should be defined. (For example: Can ZFS clones be used to
| back up the defect corpus? Can filesystem ACLs be used to control
access to portions of the repository?)
At the present time the only files that might be stored as distinct elements in the filesystem are overly large attachments.

Otherwise the definition is "file system ACLs are not used",

controls are provided by Groups which have basic inheritance and can restrict products or individual bugs.

E6. Standard operations and transactions

| The candidate DTS is expected to support a variety of standard
| operations on defect reports. The following list of
| transactions are to be assessed and documented by the evaluating
| engineer:

- create defect enter_bug.cgi -Timeless 7/19/07 12:04 AM 
- modify defect content show_bug.cgi -Timeless 7/19/07 12:04 AM 
- modify defect metadata show_bug.cgi -Timeless 7/19/07 12:04 AM 
- place defect in resolved state (*) Resolve bug as ... -Timeless 7/19/07 12:04 AM 
- restore defect from resolved to unresolved state (*) Reopen bug http://www.bugzilla.org/docs/3.0/html/lifecycle.html -Timeless 7/19/07 12:07 AM 
- associate defect with external resource URL field, also you can create "attachments" that are URLs http://www.bugzilla.org/docs/3.0/html/attachments.html -Timeless 7/19/07 12:07 AM 
- associate defect with another DTS-tracked defect Easy--Bugzilla supports the idea of bugs that depend on other bugs. -Mkanat 7/30/07 11:29 PM 

| - associate defect with tag/text label keywords are dedicated labels, tags are available for private or shared use, and status whiteboard is freeform text -Timeless 7/19/07 12:13 AM 

- change defect category Products/Components correspond to category/subcategory

as they appear in bugs.opensolaris.org,

a meta beyond Products called Classification also exists -Timeless 7/19/07 12:15 AM 

Each of these transactions should correctly modify the defect
state and send appropriate notifications.
All direct changes to bugs are logged and mail is sent to elligible accounts configured to receive mail.

Changes to resolutions of directly dependent bugs also trigger mail according to the same

general rules. - Timeless 7/19/07 12:16 AM 

Renaming a Product/Component does not generate mail,

if this is considered a problem, it could easily be fixed -Timeless 7/19/07 12:16 AM 

Equivalency (another operation or set of operations that might
be used in place of an operation not specifically supported),
| and the reasons for the omission of a transaction, are to be
assessed and documented by the evaluating engineer.
|
| Additional operations supported by the candidate DTS on defect
| reports should be documented by the evaluating engineer.
Flags can be used to indicate additional state or nominations for states.

Flags can be attached to bugs and attachments -Timeless 7/19/07 12:19 AM 

E7. Scalability

The candidate DTS must have one or more deployment
methods that allow the DTS to carry large query loads. The
latency between a defect report update and its accessibility to
be queried must be documented for such scenarios.
Shadow database used for queries http://www.bugzilla.org/docs/3.0/html/parameters.html#shadowdb -Timeless 7/19/07 12:20 AM 

| E8. Interface extensibility

Beyond the requirements of E3, an extensible interface, so that
OpenSolaris-specific tools might be integrated with DTS
operations is desired. Such an interface might be composed of a
documented "hooks" interface, hooks http://www.bugzilla.org/docs/3.0/html/cust-hooks.html 

are placed around most items and have a documented versioned interface -Timeless 7/19/07 12:25 AM

Also see the list of current code hooks at http://www.bugzilla.org/docs/tip/html/api/Bugzilla/Hook.html- Mkanat 7/30/07 10:36 PM 
a documented library interface, or
some other modular approach. An extensive hooks interface, with
hook evaluations able to terminate operations, is a strongly
desired attribute in a candidate DTS; a candidate DTS with such
an interface will be considered to meet fully this requirement.

| Such interfaces are expected to allow the automation of specific
| operations, such as the mass state updates performed by a
| consolidation's C-Team.
Confused. In bugzilla mass changes can be made by normal users using the UI.

What is the purpose of this automation?

Are you trying to supress bugmail or avoid using the standard user interface or improve speed? -Timeless 7/19/07 12:27 AM 
|
| It should be possible, via the interfaces or extension
| facilities, to perform administrative operations on the DTS:
| creation, modification, and closure of defect categories (or
| other defect classifications);
| creation, modification, and deletion of DTS user accounts (or
| other user or group relationships); and similar operations.
|Deleting anything is not recommended (this is a personal view).

Bugzilla can delete things. However, generally in an open system,

you should never delete anything. -Timeless 7/19/07 12:30 AM 

| E9. Stable defect representation
|
| Specifically extending E3 and E8, the candidate DTS will have a
| stable, formatted representation of a defect report. A
| candidate DTS that emitted defect reports in a documented plain
| text or XML format would satisfy this requirement.
|a dtd https://bugzilla.mozilla.org/bugzilla.dtd is included with bugzilla.

Any accessible bug can be retrieved as an xml file in that format. -Timeless 7/19/07 12:33 AM 

| E10. Hierarchical defect classification
|
| It appears that a candidate DTS, to represent the current defect
| corpus, must provide some means of classification that can
| represent at least a product-category-subcategory hierarchy.

Classification http://www.bugzilla.org/docs/3.0/html/classifications.html /

Product http://www.bugzilla.org/docs/3.0/html/products.html /

Component http://www.bugzilla.org/docs/3.0/html/components.html -Timeless 7/19/07 12:46 AM 

| (Although a candidate DTS could meet this requirement with only
| generic defect metadata ("tagging"), it is expected that queries
| will regularly use the defect classifications to narrow the
| query results, leading to an effective performance criterion.)
|
| 2.2. "Conditional" requirements
|
| C11. Ease of use
|
| The candidate DTS should be easy to install in a reasonably
| self-contained fashion. In principle, shipment in an
| OpenSolaris consolidation should be possible with a finite
| investment of resources, meaning that the candidate DTS does
| not have a complicated makefile system, has dependencies that
| can be easily managed, etc.

The majority of installation is done by a single script, which also prints out all the dependencies (which are mostly just Perl modules) -Mkanat 7/30/07 10:39 PM 
|
| The primary interfaces should be understandable based on the
| interfaces and documentation to a user familiar with
| DTS concepts.
|Bugzilla is the de-facto standard OSS bug-tracker, so most people are familiar with it, particularly in the Open Source world. -Mkanat 7/30/07 10:39 PM 

| C12. Stable per-defect URIs
|
| The candidate DTS should provide a unique URI for each defect
| represented in the system, or at very minimum, a unique ID that
| can be used as the basis for such a URI. The unique component
| is expected to have some association with the product or
| products tracked by the candidate DTS.
|urlbase/show_bug.cgi?id=ID. bugs.opensolaris.org doesn't do more.

If a bug changes product and you encode the product as part of the

unique ID, then you have problems. I don't understand why this

is listed as a requirement. -Timeless 7/19/07 12:48 AM 

| A candidate DTS that provides a monotonically increasing URI
| component, such as a "counter", would satisify this requirement.
urlbase/show_bug.cgi?id=ID -Timeless 7/19/07 12:51 AM 

|
| C13. Tool community health
|
| The community or author of the candidate DTS needs to be active
| and engaged with their user population. The ability of the
| candidate DTS's community to absorb, directly or through a
| liaison, the defects and feature requests of the OpenSolaris
| community should be estimated, preferably by a direct inquiry to
| the candidate DTS community

Bugzilla is generally very active and able to absorb defects and feature requests, particularly if Sun is interested in contributing code. -Mkanat 7/30/07 10:40 PM 

| C14. OpenSolaris community implementation expertise
|
| One or more contributors within the OpenSolaris community need
| to be able to assess potential defects in the implementation of
| the candidate DTS and potentially participate in the development
| of new features or supporting tools for the candidate DTS.
|This is left as an exercise for someone else. It seems that a number

of people in the OpenSolaris community have some familiarity with Bugzilla. -Timeless 7/19/07 12:53 AM 

| C15. Transactional operations and corruption recovery

The operations on the candidate's defect datastore should have
defined semantics, in particular identifying non-atomic
transactions and mechanisms for recovery from a corrupted
datastore. This requirement may be met by the underlying
datastore technology.

This is mostly met by MySQL. Thanks to table locking, there should be no non-atomic operations, and table corruption can be repaired with standard MySQL tools.-Mkanat 7/30/07 10:42 PM

There's also sanitycheck.cgi, which checks the database for consistency from the viewpoint of Bugzilla and offers automatic repairs. -Mkanat 7/30/07 10:43 PM 


| C16. Search capability
|
| An integrated search capability of the text representation of
| the defect corpus is a desired feature in the candidate DTS. It
| is possible to construct a search capability, given E3 and E9,
| but an facility integrated with the various client components is
| preferred.
 http://www.bugzilla.org/docs/3.0/html/query.html -Timeless 7/19/07 5:10 AM 
|
2.3 "Optional" requirements

| O17. Attachment handling

The legacy DTS manages large attachments as an integrated
feature, to manage system crash dumps, process corefiles,
application output, etc. related to the failures captured by a
defect report. Since a simple file archival service can meet
these requirements--once restricted access due to
confidentiality is removed--the candidate DTS need not provide
its own attachment handling feature.

Bugzilla does, however, have excellent attachment features. It capable of storing attachements either in the DB, on the filesystem, or both. -Mkanat 7/30/07 10:44 PM 


3. Evaluations

We anticipate a number of qualitative and quantitative tests to
evaluate the satisfaction of the various requirements, where a
"meets" or "does not meet" result is not applicable.

3.1. Representational and performance criteria

These criteria focus on the ability of the candidate DTS to
represent a large set of distinct, interrelated defects. The
| current Sun Microsystems defect corpus contains in the vicinity of
three million defect reports. Evaluations will be performed on a
| defect corpus derived from that published on bugs.opensolaris.org; this
corpus will have at least 100 000 defect reports.
Bugzilla can easily manage 400 000 reports,

https://bugzilla.mozilla.org/report.cgi?x_axis_field=resolution&y_axis_field=product&z_axis_field=&query_format=report-table&format=table&action=wrap  -Timeless 7/19/07 5:26 AM 
Bugzilla should easily be able to handle a virtually infinite number of bug reports. Performance, in almost every area, is not at all dependent upon the number of bug reports in the DB.  It is already designed with scalability in mind. We're also very willing to work with any user in improving any significant performance issues discovered. -Mkanat 7/30/07 10:45 PM 

The expected set of meaningful operations for performance evaluation
are:

- unresolved defects query against a product or category,

Open bugs in the Bugzilla product (3000 bugs out of 400,000): https://bugzilla.mozilla.org/buglist.cgi?query_format=advanced&order=relevance+desc&bug_status=__open__&product=Bugzilla    -Mkanat 7/30/07 10:50 PM

- defect query by keyword or other text label,

All bugs with the word "bug" in the title (out of 400,000): http://tinyurl.com/34hva5 -Mkanat 7/30/07 10:55 PM

- defects resolved by a single responsible engineer,
All bugs in the Bugzilla product resolved by me: http://tinyurl.com/3agto -Mkanat 7/30/07 10:56 PM 

| Various candidate DTSs may provide notions of defect classification,
| state, importance, or other attributes. Queries against these
| attributes should be evaluated, equivalent to
|
| - defects in either a resolved or an unresolved state, for a given
| classification,

All open bugs in the Testopia product: http://tinyurl.com/2r8ore -Mkanat 7/30/07 11:00 PM

| - defects addressed in a given build/release

All bugs addressed in Bugzilla 3.0: http://tinyurl.com/3e58kk -Mkanat 7/30/07 11:00 PM

| - unresolved defects, ordered by importance

Unresolved bugs targeted to the Bugzilla 3.0 branch, ordered by importance: http://tinyurl.com/2pafg2  (note that many of those bugs have not yet been assigned a priority, and so show up at the top of the list) -Mkanat 7/30/07 11:05 PM 
|
The candidate DTS will be evaluated for data integrity by
interruption of the set of operations by signal and by machine
failure.

The safety of the candidate DTS with respect to file system
capabilities will be evaluated using ZFS snapshot/clone technology
for safe repository copies.

3.2. Implementation criteria

The candidate DTS implementation will be assessed by a design and
code review by an OpenSolaris contributor with expertise in the
implementation language of the candidate DTS.

4. References

[1] S. Hahn, "OpenSolaris: Defect tracking relationships", 2007.
http://blogs.sun.com/sch/entry/opensolaris_defect_tracking_relationships

[2] OpenSolaris Distributed Source Code Management Requirements, 2005.

[3] IEEE Std 830-1998, "IEEE Recommended Practice for Software
Requirements Specifications", 1998.

[4] K. Wesolowski, "Defect Management Requirements in an Open
Development Paradigm", 2006.
|
| [5] OpenID, http://openid.net

--
sch at sun dot com http://blogs.sun.com/sch/
_______________________________________________
tools-discuss mailing list
tools-discuss at opensolaris dot org