xnu-2782.1.97.tar.gz

[apple/xnu.git] / tools / tests / xnu_quick_test / README

xnu_quick_test - this tool will do a quick test of every (well, to be
honest most) system calls we support in xnu.

WARNING - this is not meant to be a full regression test of all the
system calls.  The intent is to have a quick test of each system call that
can be run very easily and quickly when a new kernel is built.

This tool is meant to grow as we find xnu problems that could have be
caught before we submit to a build train.  So please add more tests and
make the existing ones better.  Some of the original tests are nothing
more than place holders and quite lame.  Just keep in mind that the tool
should run as fast as possible.  If it gets too slow then most people
will stop running it.

LP64 testing tip - when adding or modifying tests, keep in mind the
variants in the LP64 world.  If xnu gets passed a structure the varies in
size between 32 and 64-bit processes, try to test that a field in the 
structure contains valid data.  For example, if we know foo structure
looks like:
struct foo {
             int             an_int;
             long    a_long;
             int             another_int;
}
And if we know what another_int should contain then test for the known
value since it's offset will vary depending on whether the calling process
is 32 or 64 bits.

NOTE - we have several workarounds and test exceptions for some
outstanding bugs in xnu.  All the workarounds are marked with "todo" and
some comments noting the radar number of the offending bug.  Do a search
for "todo" in the source files for this project to locate which tests have
known failures.   And please tag any new exceptions you find with "todo"
in the comment and the radar number of the bug.

Building:
xnu_quick_test is built automatically by BNI for both Mac (10.9 and later), and
iOS (7 and later) trains, and is delivered on AppleInternal builds in
/AppleInternal/CoreOS/xnu_quick_test. It is built as part of the xnu_quick_test
build alias, so you can also find a copy on ~rc at:
~rc/Software/$RELEASE/Updates/$RELEASEVERSION/Roots/xnu_quick_test/AppleInternal/CoreOS/xnu_quick_test.

Alternatively you can build it yourself using make like so:
SDKROOT=/path/to/sdk make

For example:
# build for Mac, current OS
SDKROOT=/ make 
# build for iOS
SDKROOT=`xcodebuild -sdk iphoneos.internal -version Path` make

By default xnu builds all-way fat, but you can restrict this by explicitly
specifying architectures like so:
# build for only armv7 and armv7s
SDKROOT=`xcodebuild -sdk iphoneos.internal -version Path` make ARCH="armv7 armv7s"

There are four defines which you can use at the compile line to build variants.
DEBUG
       turn on additional printfs
CONFORMANCE_TESTS_IN_XNU 
       when conformance tests are in xnu, set this to 1
TEST_SYSTEM_CALLS
       test system calls (doesn't compile; a different bug)
RUN_UNDER_TESTBOTS
       when running under testbots, set this to 1		
by default, all four are set to 0, i.e. disabled.  To build, export
MORECFLAGS with the values you want set, e.g. 
       export MORECFLAGS="-D DEBUG=1 -D CONFORMANCE_TESTS_IN_XNU=1"

todo:
-- have folks with area expertise fix lame tests
   (most of the networking related tests are pretty lame)
-- mach system calls support


USAGE:  xnu_quick_test -target TARGET_PATH 

         -f[ailures] MAX_FAILS_ALLOWED   # number of test cases that may fail before we give up.  defaults to 0  
         -l[ist]                         # list all the tests this tool performs   
         -r[un] 1, 3, 10 - 19            # run specific tests.  enter individual test numbers and/or range of numbers.  use -list to list tests.   
         -s[kip]                         # skip setuid tests   
         -t[arget] TARGET_PATH           # path to directory where tool will create test files.  defaults to "/tmp/"  
         -testbot                        # output results in CoreOS TestBot compatible format

examples:  
--- Place all test files and directories at the root of volume "test_vol" --- 
xnu_quick_test -t /Volumes/test_vol/ 
 
--- Run the tool for tests 10 thru 15, test 18 and test 20 --- 
xnu_quick_test -r 10-15, 18, 20 


--- example of running the tool to list all the tests it currently supports ---
xnu_quick_test -l
List of all tests this tool performs... 
 1         syscall 
 2         fork, wait4, exit 
 3         fsync, ftruncate, lseek, pread, pwrite, read, readv, truncate, write, writev 
 4         close, fpathconf, fstat, open, pathconf 
 5         link, stat, unlink 
 6         chdir, fchdir 
 7         access, chmod, fchmod 
 8         chown, fchown, lchown, lstat, readlink, symlink 
 9         fstatfs, getattrlist, getfsstat, statfs 
 10        getpid, getppid, pipe 
 11        getauid, gettid, getuid, geteuid, issetugid, setaudit_addr, seteuid, settid, settid_with_pid, setuid 
 12        mkdir, rmdir, umask 
 13        mknod, sync 
 14        fsync, getsockopt, poll, select, setsockopt, socketpair 
 15        accept, bind, connect, getpeername, getsockname, listen, socket, recvmsg, sendmsg, sendto, sendfile
 16        chflags, fchflags 
 17        kill, vfork, execve 
 18        getegid, getgid, getgroups, setegid, setgid, setgroups 
 19        dup, dup2, getdtablesize 
 20        getrusage, profil 
 21        getitimer, setitimer, sigaction, sigpending, sigprocmask, sigsuspend, sigwait 
 22        acct 
 23        ioctl 
 24        chroot 
 25        madvise, mincore, minherit, mlock, mlock, mmap, mprotect, msync, munmap 
 26        getpgrp, getpgid, getsid, setpgid, setpgrp, setsid 
 27        fcntl 
 28        getlogin, setlogin 
 29        getpriority, setpriority 
 30        futimes, gettimeofday, settimeofday, utimes 
 31        rename, stat 
 32        flock 
 33        mkfifo, read, write 
 34        quotactl 
 35        getrlimit, setrlimit 
 36        getattrlist, getdirentries, getdirentriesattr, setattrlist 
 37        exchangedata 
 38        searchfs 
 39        sem_close, sem_open, sem_post, sem_trywait, sem_unlink, sem_wait 
 40        semctl, semget, semop 
 41        shm_open, shm_unlink 
 42        shmat, shmctl, shmdt, shmget 
 43        fgetxattr, flistxattr, fremovexattr, fsetxattr, getxattr, listxattr, removexattr, setxattr 
 44        aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, fcntl, lio_listio 
 45        kevent, kqueue 
 46        msgctl, msgget, msgrcv, msgsnd 


--- example of a full test run ---
cottje% ./BUILD/dst/xnu_quick_test 
created test directory at "/tmp/xnu_quick_test-1660251855" 
Will allow 0 failures before testing is aborted 

Begin testing - Tue Dec 13 15:56:50 2005
 
test #1 - syscall 
test #2 - fork, wait4, exit 
test #3 - fsync, ftruncate, lseek, pread, pwrite, read, readv, truncate, write, writev 
test #4 - close, fpathconf, fstat, open, pathconf 
test #5 - link, stat, unlink 
test #6 - chdir, fchdir 
test #7 - access, chmod, fchmod 
test #8 - chown, fchown, lchown, lstat, readlink, symlink 
test #9 - fstatfs, getattrlist, getfsstat, statfs 
test #10 - getpid, getppid, pipe 
test #11 - getauid, gettid, getuid, geteuid, issetugid, setauid, seteuid, settid, settid_with_pid, setuid 
test #12 - mkdir, rmdir, umask 
test #13 - mknod, sync 
test #14 - fsync, getsockopt, poll, select, setsockopt, socketpair 
test #15 - accept, bind, connect, getpeername, getsockname, listen, socket, recvmsg, sendmsg, sendto 
test #16 - chflags, fchflags 
test #17 - kill, vfork, execve 
test #18 - getegid, getgid, getgroups, setegid, setgid, setgroups 
test #19 - dup, dup2, getdtablesize 
test #20 - getrusage, profil 
test #21 - ktrace 
test #22 - getitimer, setitimer, sigaction, sigpending, sigprocmask, sigsuspend, sigwait 
test #23 - acct 
test #24 - ioctl 
test #25 - chroot 
test #26 - madvise, mincore, minherit, mlock, mlock, mmap, mprotect, msync, munmap 
test #27 - getpgrp, getpgid, getsid, setpgid, setpgrp, setsid 
test #28 - fcntl 
test #29 - getlogin, setlogin 
test #30 - getpriority, setpriority 
test #31 - futimes, gettimeofday, settimeofday, utimes 
test #32 - rename, stat 
test #33 - flock 
test #34 - mkfifo, read, write 
test #35 - quotactl 
test #36 - getrlimit, setrlimit 
test #37 - getattrlist, getdirentries, getdirentriesattr, setattrlist 
test #38 - exchangedata 
test #39 - searchfs 
test #40 - sem_close, sem_open, sem_post, sem_trywait, sem_unlink, sem_wait 
test #41 - semctl, semget, semop 
test #42 - shm_open, shm_unlink 
test #43 - shmat, shmctl, shmdt, shmget 
test #44 - fgetxattr, flistxattr, fremovexattr, fsetxattr, getxattr, listxattr, removexattr, setxattr 
test #45 - aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, fcntl, lio_listio 
test #46 - kevent, kqueue 
test #47 - msgctl, msgget, msgrcv, msgsnd 

End testing - Tue Dec 13 15:57:08 2005