How Firebird QA works and why it needs to be stronger

October 9, 2026 · Alexey Kovyazin, Firebird Foundation

In short

Firebird is raising money to reform its QA. The flow of code changes grows faster than the project can check it. Without new resources, the quality of releases will suffer.

Firebird is an open-source relational database. Its community develops it and uses it. The project estimates that about one million software developers use Firebird in their applications. Four branches are supported at the same time: 3.0, 4.0, 5.0 and 6.0 (in development). Every bug fix and every improvement must pass manual and automated checks before release.

In 2025, the Firebird repository received 190 pull requests (PRs). This is 3.1 times more than in 2020, and 2026 is already ahead of 2025. AI tools are a big reason: writing code became cheaper, but checking it did not. Complexity also grows. Firebird 6.0 is in development, and in the first half of 2026 there were almost twice as many PRs with more than 200 lines as one year earlier: 40 against 21.

The main idea of the reform is to run QA more often. Today the test suite runs once a day and only after merge. For PRs it does not run at all. After the reform, every approved PR will pass a mandatory QA subset before merge, on every update. Large PRs will also pass the full suite and a performance test.

For this we need QA engineers, cloud machines, an AI agent for the first analysis of PRs, and new tests. We ask Firebird users, and companies whose business depends on Firebird, to support the reform. The first step is done: on October 8, 2026, the Firebird Foundation started a crowdfunding campaign to raise US$20,000 for QA.

Firebird in numbers

Firebird has been developed as an open-source project for 26 years. Today it has four branches, 12 platforms in release 5.0.4 and about 763 thousand lines of code. A suite of 3,138 automated tests checks this code.

We counted the code with the cloc tool in the src directory of release 5.0.4, without blank lines and comments. About 288 thousand of the 763 thousand lines are national character set tables. The C/C++ code of the engine, the SQL layer, the network layer and the utilities is about 453 thousand lines. The platforms are taken from the files of release 5.0.4 from April 17, 2026: Windows (x86, x64), Linux (x86, x64, ARM32, ARM64), macOS (x64, ARM64) and Android (x86, x64, ARM32, ARM64).

The flow of changes is growing

Since 2020, the number of pull requests in Firebird has grown 3.1 times. In the first nine months of 2026 there were almost as many PRs as in all of 2025: 184 against 190.

Besides PRs, the tracker receives bug reports and feature requests. On October 9, 2026, about 1,800 issues and 92 PRs were open. The number of authors also grows: 19 people opened PRs in 2020, 30 in 2025, and already 32 in 2026. Every bug report must be reproduced. Every PR must be checked and tested on its own branch and on the branches where the fix is ported.

Bug reports arrive at a stable rate: from 235 to 368 per year, with no clear growth. What grows is the flow of code changes: in 2026, PRs passed issues for the first time, 184 against 173 by October 9. Together this is about 440 work items per year, and each one must be checked.

How QA works today

Today QA runs only after merge. CI on GitHub builds the code but does not test it. The test suite runs once a day on servers provided by IBSurgeon, and only for branches that had commits. People do the review, the checks and the analysis of results, and these steps do not scale with the flow of changes.

• CI on GitHub: build only. Every push and every PR is built in GitHub Actions: 22–23 build jobs for Linux (x64, x86, ARM, Alpine), Windows (x64, x86, ARM64), macOS and Android. The test suite is not part of CI. For every push it would take hours of machine time, and GitHub gives a free project at most 20 parallel jobs for the whole organization. The project already hits this limit: on October 7, 2026, 7 PRs were opened at the same time, and builds waited in the queue for up to 4 hours. Tests can be started in Actions only by hand and only on Linux x64.
• Review. Core developers read the change, discuss it in the PR and ask for updates when needed. A fix often must be ported to several branches, and each branch must be checked separately.
• Manual verification. A QA engineer first reproduces the bug on a snapshot built before the fix. Then the engineer confirms that the bug is gone on a snapshot with the fix. The test is published only after this. The section “QA day to day” below describes this in detail. More than 1,500 test files have a “Checked on” note with the numbers of the checked builds.
• The firebird-qa test suite. It is a pytest plugin and 3,138 tests. 1,962 of them are regression tests, and each one is linked to a specific bug (1,481 files for bugs from the old CORE tracker and 481 for bugs from GitHub). Another 1,176 are functional tests: SQL, triggers, domains, replication, transactions. The repository has 3,901 commits.
• Nightly runs: only for changed branches. QA runs separately from CI, on servers provided by IBSurgeon: once a day, on Windows and Linux, and only for branches that had commits during the day. For each version, the suite runs twice: in two server architectures, SuperServer and Classic. HQbird builds are checked in the same way. On firebirdtest.com each result is compared with the previous 35 runs: new failures, crashes with dumps and stack traces, and tests that became slower.
• Performance. OLTP-EMUL emulates a real OLTP workload (orders, invoices, part reservations) in dozens of parallel sessions. It measures the number of successful business operations per minute. One measurement takes about an hour. The latest results published on firebirdtest.com are from April 20, 2026, and they do not include branch 6.0 yet.

The rule is the same for all branches: if there was a change, there is a QA run. Older branches change less often, so they have fewer runs. The run time grows faster than the number of tests:

Run time is the median of the last 35 runs on firebirdtest.com as of October 9, 2026. It is the sum of two runs, in SuperServer and Classic modes. “Tests” is the number of tests that run for this branch. All branches use the same machines: the Linux server has 2 CPUs and 7 GB, the Windows server has 8 CPUs and 127 GB. Branch 6.0 has 7% more tests than 5.0, but its Linux run takes 86% longer. The time depends not only on the number of tests, but also on what these tests do.

Today QA does not run for PRs at all. A regression becomes visible only the next morning after merge, together with all commits of that day. Then someone still has to find the exact change that caused it.

QA day to day: how one fix is verified

Verifying one fix is a small investigation. A test is published only if the QA engineer caught the bug on a build before the fix and confirmed that the bug is gone on a build with the fix.

Step by step:

1. Draft test. We copy the .sql scenario from the tracker or write a draft test in pytest.
2. Snapshot before the fix. We find, or build ourselves, a snapshot from the moment closest to the fix push, but before it.
3. Reproduction. We run the tracker scenario with the draft test and confirm that the bug existed at that time. If the bug does not reproduce, there are usually three reasons:
   • the bug disappeared earlier, because of an earlier commit on this topic or even on another topic;
   • reproduction depends on something that the ticket does not describe: a configuration parameter, the OS, or, less often, the build type (debug or release);
   • the check script in the tracker is too “weak”; this is rare, but it happens;
   • in any of these cases we do not continue until we reproduce the bug ourselves. We try a snapshot from an even earlier date, the day the ticket was created. At a dead end, we ask a core developer for advice.
4. Snapshot with the fix. We find or build a snapshot from the sources that match the SHA of the fix commit. If this push was the only one that day, the ready snapshot built automatically with the message “Increment build number” is enough.
5. Fix verification. We confirm that the bug is gone on this snapshot. If it is gone, we clean up the draft test and publish it: git add, git commit, git push. If the bug is still there, we recheck our test several times. If the problem is confirmed, we alert the core developers.

Rules for tests:

• One push, one test. This is the accepted practice. The exception is a group change of something minor in many tests at once.
• One .py file, one test function like def test_N(...). Other functions in the file, except internal ones, must not be named test_*. This is a strict requirement: the processing of pytest logs and the database schema that stores the results of all runs depend on it.

Each step takes engineer time, and often a separate snapshot build. So the bottleneck is not writing code, but checking it. This is exactly where the reform adds people and cloud machines.

What changed: more code than people to check it

AI tools made writing code and bug reports much cheaper, but they did not make checking cheaper. For a database engine this is especially dangerous. A patch that looks correct can break concurrency, crash recovery or performance, and this will show only under load or on another platform.

The trend is common for all open source:

• According to GitHub Octoverse 2025, the number of created pull requests on GitHub grew by 20% in one year, and the number of merged ones by 23%. GitHub links this growth to the launch of the Copilot coding agent and automatic code review.
• curl closed its bug bounty program on February 1, 2026, because of a flood of low-quality reports, often generated by AI.
• In February 2026, the Godot maintainers publicly asked for funding to hire more people to process AI-generated PRs. In the same days, GitHub started to discuss tools to filter such PRs.

The flow in Firebird grew in the same way: 134 PRs in 2024, 190 in 2025 (+42%) and 184 by October 9, 2026, which is about 238 at the yearly rate. We welcome new contributors and do not want to limit contributions. The goal is to scale checking, not to close the door.

But complexity matters more than numbers. Firebird 6.0 is in development, and more and more PRs bring large features. In the first half of 2025, 21 of 104 PRs had more than 200 lines; in the first half of 2026, 40 of 106. The total number of PRs almost did not change, but the share of large PRs grew from 20% to 38%.

Such changes need checks on several levels: functional tests on Linux and Windows in both server modes, performance under load, and porting to other branches. And not only once: a large PR is updated after review comments, and PRs with more than 1,000 lines have about eight such iterations on average. Each iteration must be checked again.

QA reform: check more often and before merge

The main idea of the reform is to run QA more often: not once a day after merge, but for every approved PR before merge, on every update. The second part is more tests, including a fast mandatory QA subset for PRs.

All four directions build on what already works: the test suite, nightly runs on IBSurgeon servers, and CI. Nightly runs stay: direct commits of core developers also go through them, and in master these are about three quarters of all changes.

Some details for each direction:

• QA engineers. Today checking depends on a very small group of people; the reform pays a grant for a part-time QA engineer. PR runs will add about 140 results per month to the 60 nightly ones, and someone must analyze every failure.
• Cloud. Nightly servers are already busy: if all branches have commits, the full suite takes 11 hours on Linux and 6.6 hours on Windows. PRs need separate machines that start in parallel and are deleted after the run, and we pay only for the time they work.
• AI agent. It does not replace the reviewer and does not approve PRs itself. Its task is to prepare the decision: a person sees a ready summary and spends minutes, not an hour.
• Mandatory subset and coverage. The mandatory subset is selected by time, not by the number of tests: a few heavy tests can take longer than a hundred light ones. The goal is at most one hour per OS, and the selection itself is separate work for QA engineers. Also, some areas have weak coverage: for example, there are 6 tests for BLOB, 4 for monitoring and 2 for services.

How the path of a PR changes after the reform: analysis and approval come before QA, and tests move to the time before merge.

The Firebird Foundation crowdfunding campaign for US$20,000 is announced for QA engineers' time and virtual machines. The collected money also pays for the first-analysis agent and the setup of QA for PRs.

How a PR gets into QA

A QA run starts only after a person approves it. The run executes the PR code on our machines, so the PR must be reviewed first. It can be rejected right away as a duplicate, nonsense, an unconfirmed problem or an intentionally destructive change.

1. PR opened. CI builds it, as now. A PR from a new contributor does not even start CI until a maintainer approves it. This is the default GitHub setting, and it also protects against mass generation of junk PRs.
2. First analysis. The AI agent reads the PR and the linked issue and writes a recommendation in the PR: let it into QA or reject it, which complexity class the PR has, and which tests are needed.
3. Approval. A core developer or a QA engineer approves or rejects the PR. A person always makes the decision.
4. QA on every push. After approval, every push automatically passes the mandatory QA subset on Linux and Windows. Each run uses a short-lived VM without access to project secrets.
5. Before merge. Large PRs pass the full suite, and the largest ones also pass OLTP-EMUL compared with the base build.
6. After merge. The nightly run, as now.

How parallel runs work

In the cloud, four VMs in parallel cost the same as one machine in sequence. But the result is ready when the longest branch finishes: after 4 h 39 min instead of 11 hours.

Take the nightly run of the full suite on Linux for all four branches: 3.0 takes 1 h 31 min, 4.0 takes 2 h 19 min, 5.0 takes 2 h 30 min, and 6.0 takes 4 h 39 min. On one server this is 11 hours in a row; on four VMs it is 4 h 39 min. The cloud charges for actual working time, so in both cases we pay for the same 11 machine-hours.

PRs work the same way: the full suite on Linux and Windows and two OLTP-EMUL measurements run at the same time, and a large PR gets an answer in about 5 hours, not 11.

Linux: DigitalOcean, 16 GB droplet

Since January 1, 2026, DigitalOcean bills droplets per second, with a minimum of 60 seconds, so we pay exactly for the run time. Prices are from the DigitalOcean pricing page as of October 9, 2026.

For performance tests we use General Purpose: dedicated cores are not shared with neighbors on the server, so the measurements are repeatable. Full 6.0 suite on Linux: $0.1875/h × 4.65 h ≈ $0.87. The IBSurgeon Linux server is weaker (2 CPUs and 7 GB), so in the cloud the run will probably be faster; in our calculations we use the measured time as is.

Windows: Azure, D4s v5 VM (4 vCPU, 16 GB)

DigitalOcean does not offer Windows droplets, so Windows runs go to Azure. Prices are for the West Europe region, pay-as-you-go, from the Azure Retail Prices API as of October 9, 2026. The Windows license is included in the price; disks and traffic are not included.

Full 6.0 suite on Windows: $0.414/h × 2.47 h ≈ $1.02. The IBSurgeon Windows server is more powerful than the cloud VM (8 CPUs and 127 GB), so in the cloud the run may take longer; the reserve for repeated runs covers this. Spot VMs are more than five times cheaper, but Azure can take them back at any moment. They are fine for functional tests, but not for performance measurements.

What QA for PRs costs for six months

QA for every approved PR will cost about $477 for six months. This uses 2.8 times less machine time than the full suite with OLTP-EMUL on every push. The savings come from the mandatory subset and the complexity factor.

The forecast is 127 PRs in the next six months: 114 PRs in the last six months with 25% growth per year, the average since 2020: 114 × 1.25^0.5 ≈ 127. Issues are not part of cloud runs: a QA engineer reproduces a bug report on snapshots, and the fix comes as a PR or a commit.

There are three types of runs. All are calculated for branch 6.0, where most PRs go and where the suite is the longest. We add 15 minutes of preparation to each VM start:

• Mandatory subset: 1 h on Linux and 1 h on Windows: 2.5 machine-hours, $0.75.
• Full suite: two runs, SuperServer and Classic, together 4 h 39 min on Linux and 2 h 28 min on Windows: 7.6 machine-hours, $2.04.
• OLTP-EMUL: the PR build and the base build in parallel, 1.5 h each on Linux: 3.5 machine-hours, $0.66.

What to run and how many times depends on the PR. A 5-line PR and a 5,000-line PR differ both in the number of updates and in the depth of checks. So we use a complexity factor: the machine time per PR compared with the simplest class.

Class shares and push counts come from 214 PRs of the last 12 months. We estimated pushes by the number of days with new commits in a PR: on average 1.2, 1.6, 2.1 and 8.0 by class. This is a lower bound, because force-push hides some iterations, so the values in the table are rounded up.

Of the 127 forecast PRs, 53 are in class S, 30 in M, 28 in L and 16 in XL. One PR costs $1.13, $1.50, $4.30 and $10.76 by class. We add another 20% for repeated runs because of unstable tests and VM failures.

Compare this with today and with the option “everything on every push”:

Today's machine time is calculated from the last 30 days: 34 runs on Linux and 26 on Windows. In both scenarios the cost is small. The waiting time matters more: with the mandatory subset, the author gets an answer for every update in about 1 hour 15 minutes; with the full suite, only after 5 hours.

If PRs grow faster than the forecast, at the 2025 rate of +42% per year, there will be about 136 of them, and the runs will cost about $511. The reserve covers the difference.

Test expansion for 6.0 and higher test intensity

Preparing the 6.0 release and running tests more intensively will add about $150 to cloud runs for six months, about 30% on top of QA for PRs. The main cost of the expansion is not machines, but the time of QA engineers for new tests.

According to the Firebird 6 roadmap, the alpha version comes in Q4 2026, the beta in Q2 2027, and the final release in Q4 2027. The suite grows with them: tests that run only on 6.0 grew from 93 to 225 in one year; 91 of them were added in the last six months and 49 in the last quarter. New features (schemas, tablespaces, JSON functions, the ROW type, the shared metadata cache) need new and often heavy tests.

Machine-hours are shown without repeats; the cost includes the 20% reserve for repeated runs, as above.

With the expansion, machine time grows about three times compared with today: 153 machine-hours of nightly runs, 232 for QA of PRs and another 82 for the expansion, about 467 per month in total. Nightly servers do not have this capacity. And after the beta, a separate 6.0 branch will appear, so there will be five nightly runs instead of four.

The main expansion is people's work. If the pace of the last quarter continues, about 100 tests for 6.0 only will be added in six months, and more with the alpha and bug reports from testers. Each such test goes through the path described in “QA day to day”. The QA engineer grant pays for this work. If there are more tests, the rest of the collected money and the reserve can pay for more QA hours.

Budget for six months

Six months of the reform cost about $18,402, which is $1,598 less than the crowdfunding goal of US$20,000. The rest can stay in reserve for PR growth or pay for more QA hours. About 83% of the money goes to people's work; cloud and the AI agent together cost about 7%.

How we calculated the runs. Cloud runs for PRs: 127 forecast PRs, by complexity class, with the mandatory subset on every push and the full suite before merge of large PRs; see “What QA for PRs costs for six months” for details. The Linux line also includes nightly OLTP-EMUL for 5.0 and 6.0 on days with commits: about 24 measurements per month, $47.25 for six months. The 6.0 expansion is a separate line: $78.18 on Linux and $71.95 on Windows; see “Test expansion for 6.0 and higher test intensity”. Nightly runs of the test suite still run on IBSurgeon servers and serve as the baseline for comparison.

The AI agent fits into $100 per month. The analysis of one PR (about 200 thousand input tokens and 10 thousand output tokens) costs about $1, even on the most powerful model. This is about $80 per month: an analysis of every new PR and a repeated one after updates. With more PRs after the alpha, it is about $90.

The grant size, $2,133 per month, follows the project plan. The cloud lines are calculated with current prices and real run times. For the mandatory subset, which still has to be selected, we use the target time of 1 hour per OS.

Crowdfunding to improve QA

The first step of the reform has already started: on October 8, 2026, the Firebird Foundation announced a crowdfunding campaign to improve QA. The goal is to raise an additional US$20,000 in six months.

The money will go to two areas:

• the time of QA engineers;
• virtual machines for QA runs, first of all for performance tests.

At the start, on October 8, 2026, 93 open pull requests were waiting for QA checks and testing. The project rule does not change: no change is merged without review and tests. Donations of any size are accepted on the campaign page.

What you get and how to help

Investing in QA is insurance for everyone who keeps data in Firebird. A bug caught before release costs hours of engineer work. A missed bug costs downtime and lost data for thousands of users.

What users and sponsors get:

• regressions are visible in the PR itself, before merge, not the next morning after it;
• stable releases of branches 3.0–5.0 and the future 6.0, without regressions and slowdowns;
• a faster path from a bug report to a fix, because checking stops being the bottleneck;
• the ability to safely accept more external changes, including changes prepared with AI;
• transparency: the results of all runs stay public on firebirdtest.com, and there will be a report on expenses.

How to help:

1. Support the QA crowdfunding campaign with any amount. This is the most direct way to help the QA reform.
2. Support the project through the Firebird Foundation, once or with a subscription (Associate: €100 per year, Partner: €400 per year).
3. Become a corporate sponsor: the terms are discussed with the Firebird Foundation individually.
4. Provide cloud resources or credits in DigitalOcean, Azure or AWS for test machines.
5. Help with people: assign an engineer to check PRs or to write tests in firebird-qa.

Sources

Issue and pull request data are calculated for the FirebirdSQL/firebird repository: PR and issue counts via the GitHub Search API, by creation date; authors, PR sizes and iteration counts from the history of refs/pull branches; PR classes from 214 PRs over 12 months; the share of direct commits from the master history. Test counts and their growth come from the history of the firebird-qa repository; run times and frequency come from firebirdtest.com reports. Data as of October 9, 2026.

• Firebird Foundation: Crowdfunding to improve Firebird QA
• FirebirdSQL/firebird: release 5.0.4
• FirebirdSQL/firebird-qa
• Firebird QA Results: firebirdtest.com
• firebirdtest.com: all tests for 35 runs, branch 6.0, Linux
• firebirdtest.com: OLTP-EMUL, Firebird
• Firebird QA: firebirdsql.org
• Firebird: Roadmap v6
• Firebird OLTP-EMULator test: IBSurgeon
• Firebird (database server): Wikipedia
• Support Firebird: Firebird Foundation
• GitHub Docs: Managing GitHub Actions settings for a repository
• GitHub Docs: Actions limits
• FirebirdSQL/firebird: CI runs
• DigitalOcean: Droplet pricing
• DigitalOcean: Spaces Object Storage pricing
• Azure Retail Prices API: D4s v5, West Europe
• GitHub Octoverse 2025
• BleepingComputer: curl ending bug bounty program
• DevClass: GitHub itself to blame for AI slop PRs, say devs