Peer Review
Peer review is most often cited asset of Open Source development model, especially in relation to quality. So it's only natural that Peer review has it's place in Firebird Project.
How it works in Firebird project:
Changes in CVS are reviewed as they appear (see
Firebird-checkins mailing list and newsgroup) by other developers. All problems are immediately reported and discussed in
Firebird-devel.
Firebird Foundation issued a grant to secure a dedicated person (currently Claudio Valderrama C.) to that task. Currently, only changes in core engine are reviewed, and although changes to other modules are often reviewed by other developers, due to time contraints, it may happen that some changes slip in unnoticed. So if you're interested to follow closely the development of particular subproject, we'd like encourage you to watch changes in firebird-chekins and immediately report anything you'll find suspicious in
appropriate mailing list or newsgroup.
Code audit is just a form of Peer Review. But while peer review can address only code changes and small parts of codebase that's being work on, Code audit extends the peer review to the whole codebase.
This QA method is not used regularly (except for critical systems) because it's very expensive in terms of human labour and required knowledge.
How it works in Firebird project:
We haven't had resources to do a full "stop everything" code review in the past, but during v3.0 development the codebase was entirely examined. Source code after this version should be also more clean, and well-structured to document it. That mean both,
doxygen documentation from in-line comments and separate FB internals document, where major data structures and subsystems would be documented, including their mutual relations.
This task is huge and will take time, so we would appreciate any help, especially with documentation part. All source code already contains in-line comments and descriptions for all public functions (although is a little bit short in documentation for data structures), so we only need to format them for
doxygen. If you'd like volunteer for this task, don't hesitate to
contact us!
Each source file that has been reviewed in full should be marked as such in header, with auditor's nickname and date of full review.
Development Test Cases
Developers often use home-made test applications to test basic functionality of parts they're working on. They are not always developed nor complete, but if they are developed, they are not deleted afterwards and could be reused in test system or by QA group. They don't need to be so extreme like in eXtreme programming, and developed *before* any real development take place (although XP approach would be nice :), but *any* such application has its value.
How it works in Firebird project:
Right now, development test cases are used only in "JDBC type 4 driver", ".NET Data Provider" and "FDB Python driver" development groups. These tests are based on UnitTest frameworks for Java, .NET and Python.
Due to internal structure of Firebird, it's not practical to start development of Unit tests for Firebird engine codebase.
Automated Black-box Testing
The purpose is to verify correctness and limits of final product. That mean:
- Correct input produce correct result.
- Incorrect input produce correct error result.
- Product reacts correctly to limit conditions and values (memory, disk space, file size, values at behaviour-switching boundary etc.)
- Product reacts correctly to unexpected conditions (power failure, low-memory, I/O error)
- Consistency. Test results from 1)-4) are consistent in time and with different order of execution. This also means concurrency tests.
- Usability (performance, ergonomy, documentation)
If possible, automated test systems are used to do 1),2),3) and 5) tests. 4) and 6) usually depends completely on manual labour. All bugs and issues detected by QA department are tracked in some sort of Problem resolution system. This system is a crucial part of whole QA/development cycle, and any implemented QA procedure can't work successfully for long time without it.
How it works in Firebird project:
We inherited an automated test system called TCS that was released by Borland together with around 390 tests when InterBase code was released. Because TCS is hard to operate and extend, we decided to use another system. For some time we used QMTest, but our specific requirements forced us to develop our own solution (
fbtest). You can learn more about
fbtest and our new test suite in this
document.
Our current QA efforts in this area are focused on development of new test cases, especially on full SQL reference tests for new Firebird features and regression test for resolved issues. It's a lot of work, and we'll appreciate any help, especially with correct test design which is the most time consuming part of this task. To help us, you don't need any special skills or tools, just the good will, time and a text editor. You can learn more about our requirements for new test cases in our
Test design How-To.
We extensively use Atlassian
JIRA tracker. There are several tracking groups to separate different kinds of issues. Please, investigate the Tracker site thoroughly if you are not used to it. Life is too short to 'clean up' all invalid/redundant entries.
If you're planning to work with JIRA extensively, you may appreciate the
JIRA Client that integrates seamlessly with it and brings more searching power and interactivity to frequent JIRA users. It's a commercial product with 30-day evaluation, but you may use our
unlimited open-source licence that's "locked" to work only with Firebird JIRA tracker.
