The Bitmap Test Suite

The Bitmap Test Suite provides a set of Windows bitmap files that yield near-complete code coverage of bitmap processing software. It includes a sampling of various well-formed bitmap encodings, as well as an extensive set of malformed bitmap files.

Motivation

The software development community does an excellent job of sharing code, but it could do a better job sharing its test infrastructure. We often praise high-quality products like GIMP or Apache and proudly offer to their source code for inspection. However, we rarely emphasize the testing-side of software development. In a sense, this help perpetuate the myth that programming is more important than testing, when, in fact, both are critical to producing high-quality software.

Let's face it, bitmaps are boring. And when it comes time to supporting an image format, many project write their own implementation because the format is so simple. However, there are enough gotchas that can lead to improper rendering and possibly memory leaks, crashes, or even vulnerabilities. What's worse, many programmers have a difficult time obtaining test data and may only verify their implementation with the bitmaps that they have on hand. Few programmers take the time to build a complete set of test cases, especially when the primary purpose of their software is NOT manipulating images. Furthermore, the official bitmap specification glosses over many details, leaving room for misinterpretation.

All of this creates a problem. Instead of everyone using the single, well-tested bitmap implementation, everyone has their own. And not everyone has the time or inclination to thoroughly test their particular implementation. The Bitmap Test Suite makes it easier to do so.

Using the Test Suite

The Bitmap Test Suite is just a collection of bitmap files, so it's up to you to figure out how to integrate it into your testing framework. However, because it is just a collection of bitmap files, even a casual end-user with no test framework can use it to test their favorite bitmap processing software.

The bitmaps are organized into three directories: valid, questionable, and corrupt.

The "valid" directory contains well-formed bitmap files that all confoming bitmap processors should be able to process.

The "questionable" directory contains bitmap files that malformed in some non-critical way. A conforming bitmap processor may reject any of these bitmaps but most will ignore the error and process them anyway.

The "corrupt" directory contains bitmap files that are seriously malformed, possibly even malicious. A conforming bitmap processor may reject any of these files, but should not crash or leak memory when asked to process any of them. A superior bitmap processor will display an informative, accurate diagnostic for each file that it cannot process.

All three directories contain an "index.html" file, which is an HTML document that describes the purpose of each file. Most of the valid and questionable bitmaps should look like one of the images in the table, below. Otherwise, the English description in the "index.html" file should suffice.

Description	Image
Most bitmaps looks like the canonical image to the right. This image has several characteristics that make it good for testing. The image contains, black, white, red, green, and blue to make it easy to verify that the color vectors are processed correctly. It uses vertical lines to make it easy to verify that scanline padding is processed correctly. The words "TOP LEFT" appear in the upper left-hand corner to make it easy to verify that the orientation (top-down vs. bottom-up, right-left vs. left-right) is correct. The image is surrounded by a one-pixel white border inside a one-pixel black border to make it easy to see any off-by-one errors that would clip the image. This image does NOT provide a way to verify high color fidelity.
When the bit depth is 1 bit per pixel, the bitmap does not have enough colors to encode the canonical image. In this case, the image looks like the one to the right. This image has still shares many characteristics of the canonical image. The image uses both colors in the palette. The image uses vertical lines. The words "TOP LEFT" appears in the upper left-hand corner. The image is surrounded by a one-pixel white border inside a one-pixel black border.
Files whose filename contains "rle4-alternate" also cannot use the canonical image. This tests a special type of run-length encoding for runs of two alternating colors, such as what happens in dithered images. In this case, image looks like the one to the right. The image contains five colors: black, white, red, green, and blue. The three color bands are composed of vertical lines of alternating colors. The right-most line of the right band is red, followed by green. The right-most line of the center band is green, followed by blue. The right-most line of the left band is blue, followed by red. The word "TOP LEFT" is in the upper left-hand corner. The image is surrounded by a one-pixel white border inside a one-pixel black border.
Bitmaps that are one pixel wide and one pixel high are just a single blue dot.

A Word on Copyrights

I have dedicated the bitmaps and the Python script that generates them to the public domain. I chose not to use a license like BSD or GPL because I didn't want legal ambiguity to scare someone into not using this test suite. In short, I want to make it easier to test all software, even non-free software.

How You Can Help

A project like this has an important, but limited audience. As a result, many people who would want to use it may never even know that it exists. If you file a bug using the Bitmap Test Suite, please include a reference to the test suite in your bug report, so that the QA engineer will know where they can easily obtain more test cases.

Likewise, if you find a bug (or even just a code branch) that the Bitmap Test Suite doesn't cover (but should cover), please let me know so that I can add the corresponding test case.

I encourage you to share any contribution, suggestion, or criticism that you may have of this test suite. You can do so by using one of the many facilities provided on the project's page on SourceForge. However, please understand that all contributions must be accompanied by the following dedication:

As the author of this work (the "Work"), I hereby disclaim all copyrights to the Work.

I make this dedication for the benefit of the public at large and to the detriment of the my heirs and successors. I intend this dedication to be an overt act of relinquishment in perpetuity of all present and future rights under copyright law, whether vested or contingent, in the Work. I understand that such relinquishment of all rights includes the relinquishment of all rights to enforce (by lawsuit or otherwise) those copyrights in the Work.

I recognize that, once placed in the public domain, the Work may be freely reproduced, distributed, transmitted, used, modified, built upon, or otherwise exploited by anyone for any purpose, commercial or non-commercial, and in any way, including by methods that have not yet been invented or conceived.