The Onion Stand: testing

Showing posts with label testing. Show all posts

Monday, 18 January 2010

Better output testing HTML and other long strings

If you ever wrote or ran a website test in Perl, you probably used Rafaël Garcia-Suarez's wonderful Test::LongString, even if you never heard of it before. Not only is it used by lots of CPAN modules, it is the foundation of Test::WWW::Mechanize test functions, with flavors for Catalyst, PSGI, CGIApp and several others. But it doesn't have to be HTML content testing - any string output can be tested with its awesome functions.

Maybe Test::LongString's most common function is contains_string(), which you may know as Test::WWW::Mechanize's $mech->content_contains() method. When it fails, the failing test output shows the original string and the text it was trying to find:

#     searched: "foo bar"
#   can't find: "baz"

Problem is, when the original string is too long, you get only the beginning of it. This happens a lot when you're trying to find content inside HTML, like:

#     searched: "<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Trans"...
#   can't find: "some random content"

Such output is only barely useful, and I got fed up of having to edit a test file manually, add diag() calls to see the complete output, then ack for the wanted string to see what went wrong. So I made a small patch to Test::LongString, which Rafaël promptly accepted (yup, he rocks, but we all know that ;-).

Now, whenever such test fails, you'll get two extra lines:

#     searched: "<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Trans"...
#   can't find: "some random content"
#         LCSS: "ome random content"
# LCSS context: "d="content" class="foo">Some random content!</div>"

"LCSS" stands for Longest Common SubString, meaning you get whatever matched the most inside the original content (note that LCSS is not the same as LCS, or Longest Common Subsequence, which would go for a non-sequential match as seen in diff-like outputs). And, of course, "LCSS context" means the surroundings of the LCSS string just found.

Now, just by looking at our example, we know exactly why the test failed, and we are free to fix either the test suite or the application.

Know what's best? You get all that for free, no need to change a single line in your tests. Just update Test::LongString to 0.12 or later and enjoy! :-)

Friday, 15 May 2009

Test::More memory issues

You gotta love testing. And, writing Perl code/tests, you gotta love Test::More. It has a great API, it's cool, it's fast, it's stable, it's leaking memory... "Say what?!"

Yeah. I mean, it's not leaking leaking, just eating more memory at every test you run, be it is(), ok(), or whatever. See for yourself:

-------------8<-------------
use Test::More qw(no_plan);
while (1) {
 is(1,1);
}
------------->8-------------

If you run this, you'll see memory consumption for the process going up at a very fast pace (use top or any other viewer), this is what it looks like on my system after 40K tests (a couple of seconds using the code above):

After a quick stop by #perl-qa, rjbs++ said it was a feature, not a bug. Apparently, Test::Builder (the backend for Test::More, Test::Simple and their siblings/derivatives) stores each test result, so by the time you're at 50_000 results, you'll have 50_000 hashes in memory, even if they are all just PASS tests.

This does explain the ever-increasing memory issue, but it's still a no go for me. Although 95% of Test::Builder users will never see this as a problem, I'm doing a lot of combinatorics tests, so a single .t of mine has to go through well over 500K tests, hitting a memory wall real hard.

Now, if you stumbled in that same problem, don't panic: there are at least two possible workarounds for it.

Split you combinatorics tests into smaller test files, picking the variable with the most possible...erm... variations... and turn it into a different constant for each test file. If you picked a good one, the number of tests on each file will be exponetially decreased and the collected test data won't be such a memory burden.
If you have huge/nested loops in your tests, you can take a reverse approach and instead of using "ok or not ok" functions, use "normal" (code) compares, triggering fail() if something bad happens. This way the testing framework will only store the history of failed tests, eating as little memory as possible.

Just a couple of hours before my "potential bug" report, Schwern++ replied confirming it was "working as designed", and was kind enough to patch those workaround tips in the "CAVEATS and NOTES" part of the documentation for Test::Builder (as it affects everything, not just Test::More). I really should thank Apocalypse++ for talking me into writing it :)

Of course, I still hope to see this problem go away in future releases, and (fortunately for me) so does Schwern. According to him, this was one of the design issues which brought Test::AtRuntime to a halt. As a result, his upcoming Test::Builder2 will have the ability to turn off history and the history sub-system will be a separate object. There also might be a happy medium where history contains just a count of each type of result - just as I hoped - which will allow most of Test::More's features based on knowing the results to continue to work while not eating up too much memory in a long-running test file.

Yay for testing, and for the Perl QA community!

Saturday, 9 May 2009

Testing Differences in Templates

I've been doing some template refactoring now, and since I want to make sure the new templates render exactly the same as the old ones, I used Schwern++'s Test::More to compare their outputs for all combinations of input variables I had. Things were great for the first small-and-obvious changes, but at some point both outputs looked the same to me, and yet were still failing the test, meaning something was off. But what?

A quick CPAN search pointed me to Barrie Slaymaker++'s Test::Differences (now maintained by Ovid++). After that, all I had to do was replace my is() call with eq_or_diff() and voilá:

#   Failed test 'templates should produce the exact same output'
#   at t/refactoring.t line 128.
# +---+------+---+----------+
# | Ln|  Got | Ln| Expected |
# +---+------+---+----------+
# | 3 |      | 3 |          |
# | 4 |      | 4 |          |
# | 5 |      | 5 |          |
# * 6 |\t\n  * 6 |\n        |
# | 7 |      | 7 |          |
# | 8 |      | 8 |          |
# | 9 |      | 9 |          |
# +---+------+---+----------+

Ha! Much easier, much direct. After a simple extra tab removal, all tests were successful and I could move on. Yay!