Test::More memory issues

You gotta love testing. And, writing Perl code/tests, you gotta love Test::More. It has a great API, it's cool, it's fast, it's stable, it's leaking memory... "Say what?!"

Yeah. I mean, it's not leaking leaking, just eating more memory at every test you run, be it is(), ok(), or whatever. See for yourself:

-------------8<-------------
use Test::More qw(no_plan);
while (1) {
 is(1,1);
}
------------->8-------------

If you run this, you'll see memory consumption for the process going up at a very fast pace (use top or any other viewer), this is what it looks like on my system after 40K tests (a couple of seconds using the code above):

After a quick stop by #perl-qa, rjbs++ said it was a feature, not a bug. Apparently, Test::Builder (the backend for Test::More, Test::Simple and their siblings/derivatives) stores each test result, so by the time you're at 50_000 results, you'll have 50_000 hashes in memory, even if they are all just PASS tests.

This does explain the ever-increasing memory issue, but it's still a no go for me. Although 95% of Test::Builder users will never see this as a problem, I'm doing a lot of combinatorics tests, so a single .t of mine has to go through well over 500K tests, hitting a memory wall real hard.

Now, if you stumbled in that same problem, don't panic: there are at least two possible workarounds for it.

1. Split you combinatorics tests into smaller test files, picking the variable with the most possible...erm... variations... and turn it into a different constant for each test file. If you picked a good one, the number of tests on each file will be exponetially decreased and the collected test data won't be such a memory burden.
   
   
2. If you have huge/nested loops in your tests, you can take a reverse approach and instead of using "ok or not ok" functions, use "normal" (code) compares, triggering fail() if something bad happens. This way the testing framework will only store the history of failed tests, eating as little memory as possible.

Just a couple of hours before my "potential bug" report, Schwern++ replied confirming it was "working as designed", and was kind enough to patch those workaround tips in the "CAVEATS and NOTES" part of the documentation for Test::Builder (as it affects everything, not just Test::More). I really should thank Apocalypse++ for talking me into writing it :)

Of course, I still hope to see this problem go away in future releases, and (fortunately for me) so does Schwern. According to him, this was one of the design issues which brought Test::AtRuntime to a halt. As a result, his upcoming Test::Builder2 will have the ability to turn off history and the history sub-system will be a separate object. There also might be a happy medium where history contains just a count of each type of result - just as I hoped - which will allow most of Test::More's features based on knowing the results to continue to work while not eating up too much memory in a long-running test file.

Yay for testing, and for the Perl QA community!