regression test suite for security components. by Michael Brouwer GOALS ===== The goals of this test setup are to have something that required 0 configuration and setup and allows developers to quickly write new standalone test cases. USAGE ===== The tests are runnable from the top level Makefile by typing: make test or individually from the command line or with gdb. Tests will be built into a directory called build by default or into LOCAL_BUILD_DIR if you set that in your environment. DIRECTORY LAYOUT ================ Currently there are subdirectories for a number of different parts of the security stack. Each directory contains some of the unit tests I've managed to find from radar and other places. The test programs output their results in a format called TAP. This is described here: http://search.cpan.org/~petdance/Test-Harness-2.46/lib/Test/Harness/TAP.pod Because of this we can use Perl's Test::Harness to run the tests and produce some nice looking output without the need to write an entire test harness. Tests can be written in C, C++ or Objective-C or perl (using Test::More in perl). WRITING TESTS ============= To add a new test simply copy one of the existing ones and hack away. Any file with a main() function in it will be built into a test automatically by the top level Makefile (no configuration required). To use the testmore C "library" all you need to do is #include "testmore.h" in your test program. Then in your main function you must call: plan_tests(NUMTESTS) where NUMTESTS is the number of test cases you test program will run. After that you can start writing tests. There are a couple of macros to help you get started: The following are ways to run an actual test case (as in they count towards the NUMTESTS number above): ok(EXPR, NAME) Evaluate EXPR if it's true the test passes if false it fails. The second argument is a descriptive name of the test for debugging purposes. is(EXPR, VALUE, NAME) Evaluate EXPR if it's equal to VALUE the test passes otherwise it fails. This is equivalent to ok(EXPR == VALUE, NAME) except this produces nicer output in a failure case. CAVEAT: Currently EXPR and VALUE must both be type int. isnt(EXPR, VALUE, NAME) Opposite of is() above. CAVEAT: Currently EXPR and VALUE must both be type int. cmp_ok(EXPR, OP, VALUE, NAME) Succeeds if EXPR OP VALUE is true. Produces a diagnostic if not. CAVEAT: Currently EXPR and VALUE must both be type int. ok_status(EXPR, NAME) Evaluate EXPR, if it's 0 the test passes otherwise print a diagnostic with the name and number of the error returned. is_status(EXPR, VALUE, NAME) Evaluate EXPR, if the error returned equals VALUE the test passes, otherwise print a diagnostic with the expected and actual error returned. ok_unix(EXPR, NAME) Evaluate EXPR, if it's >= 0 the test passes otherwise print a diagnostic with the name and number of the errno. is_unix(EXPR, VALUE, NAME) Evaluate EXPR, if the errno set by it equals VALUE the test passes, otherwise print a diagnostic with the expected and actual errno. Finally if you somehow can't express the success or failure of a test using the macros above you can use pass(NAME) or fail(NAME) explicitly. These are equivalent to ok(1, NAME) and ok(0, NAME) respectively. LEAKS ===== If you want to check for leaks in your test you can #include "testleaks.h" in your program and call: ok_leaks(NAME) Passes if there are no leaks in your program. is_leaks(VALUE, NAME) Passes if there are exactly VALUE leaks in your program. Useful if you are calling code that is known to leak and you can't fix it. But you still want to make sure there are no new leaks in your code. C++ === For C++ programs you can #include "testcpp.h" which defines these additional macros: no_throw(EXPR, NAME) Success if EXPR doesn't throw. does_throw(EXPR, NAME) Success if EXPR does throw. is_throw(EXPR, CLASS, FUNC, VALUE, NAME) Success if EXPR throws an exception of type CLASS and CLASS.FUNC == VALUE. Example usage: is_throw(CssmError::throwMe(42), CssmError, osStatus(), 42, "throwMe(42)"); TODO and SKIP ============= Sometimes you write a test case that is known to fail (because you found a bug). Rather than commenting out that test case you should put it inside a TODO block. This will cause the test to run but the failure will not be reported as an error. When the test starts passing (presumably because someone fixed the bug) you can comment out the TODO block and leave the test in place. The syntax for doing this looks like so: TODO: { todo(" ER: AAPL target: $4,000,000/share"); cmp_ok(apple_stock_price(), >=, 4000000, "stock over 4M"); } Sometimes you don't want to run a particular test case or set of test cases because something in the environment is missing or you are running on a different version of the OS than the test was designed for. To achieve this you can use a SKIP block. The syntax for a SKIP block looks like so: SKIP: { skip("only runs on Tiger and later", 4, os_version() >= os_tiger); ok(tiger_test1(), "test1"); ok(tiger_test2(), "test2"); ok(tiger_test3(), "test3"); ok(tiger_test4(), "test4"); } How it works is like so: If the third argument to skip evaluates to false it breaks out of the SKIP block and reports N tests as being skipped (where N is the second argument to skip) The reason for the test being skipped is given as the first argument to skip. Utility Functions ================= Anyone writing tests can add new utility functions. Currently there is a pair called tests_begin and tests_end. To get them #include "testenv.h". Calling them doesn't count as running a test case, unless you wrap them in an ok() macro. tests_begin creates a unique dir in /tmp and sets HOME in the environment to that dir. tests_end cleans up the mess that tests_begin made. When writing your own utility functions you will probably want to use the setup("task") macro so that any uses of ok() and other macros don't count as actual test cases run, but do report errors when they fail. Here is an example of how tests_end() does this: int tests_end(int result) { setup("tests_end"); /* Restore previous cwd and remove scratch dir. */ return (ok_unix(fchdir(current_dir), "fchdir") && ok_unix(close(current_dir), "close") && ok_unix(rmdir_recursive(scratch_dir), "rmdir_recursive")); } Setup cases all tests unil the end of the current funtion to not count against your test cases test count and they output nothing if they succeed. There is also a simple utility header called "testcssm.h" which currently defines cssm_attach and cssm_detach functions for loading and initializing cssm and loading a module.