tools/tests/xnu_quick_test/README

   1 xnu_quick_test - this tool will do a quick test of every (well, to be
   2 honest most) system calls we support in xnu.
   3
   4 WARNING - this is not meant to be a full regression test of all the
   5 system calls.  The intent is to have a quick test of each system call that
   6 can be run very easily and quickly when a new kernel is built.
   7
   8 This tool is meant to grow as we find xnu problems that could have be
   9 caught before we submit to a build train.  So please add more tests and
  10 make the existing ones better.  Some of the original tests are nothing
  11 more than place holders and quite lame.  Just keep in mind that the tool
  12 should run as fast as possible.  If it gets too slow then most people
  13 will stop running it.
  14
  15 LP64 testing tip - when adding or modifying tests, keep in mind the
  16 variants in the LP64 world.  If xnu gets passed a structure the varies in
  17 size between 32 and 64-bit processes, try to test that a field in the
  18 structure contains valid data.  For example, if we know foo structure
  19 looks like:
  20 struct foo {
  21              int             an_int;
  22              long    a_long;
  23              int             another_int;
  24 }
  25 And if we know what another_int should contain then test for the known
  26 value since it's offset will vary depending on whether the calling process
  27 is 32 or 64 bits.
  28
  29 NOTE - we have several workarounds and test exceptions for some
  30 outstanding bugs in xnu.  All the workarounds are marked with "todo" and
  31 some comments noting the radar number of the offending bug.  Do a search
  32 for "todo" in the source files for this project to locate which tests have
  33 known failures.   And please tag any new exceptions you find with "todo"
  34 in the comment and the radar number of the bug.
  35
  36 To build a fat binary, export ARCH="i386 x86_64". This will work
  37 for any architectures that Apple gcc recognizes.
  38
  39 Added four defines which you can use at the compile line to build variants.
  40 DEBUG
  41        turn on additional printfs
  42 CONFORMANCE_TESTS_IN_XNU
  43        when conformance tests are in xnu, set this to 1
  44 TEST_SYSTEM_CALLS
  45        test system calls (doesn't compile; a different bug)
  46 RUN_UNDER_TESTBOTS
  47        when running under testbots, set this to 1
  48 by default, all four are set to 0, i.e. disabled.  To build, export
  49 MORECFLAGS with the values you want set, e.g.
  50        export MORECFLAGS="-D DEBUG=1 -D CONFORMANCE_TESTS_IN_XNU=1"
  51
  52 todo:
  53 -- have folks with area expertise fix lame tests
  54    (most of the networking related tests are pretty lame)
  55 -- mach system calls support
  56
  57
  58 USAGE:  xnu_quick_test -target TARGET_PATH
  59
  60          -f[ailures] MAX_FAILS_ALLOWED   # number of test cases that may fail before we give up.  defaults to 0
  61          -l[ist]                         # list all the tests this tool performs
  62          -r[un] 1, 3, 10 - 19            # run specific tests.  enter individual test numbers and/or range of numbers.  use -list to list tests.
  63          -s[kip]                         # skip setuid tests
  64          -t[arget] TARGET_PATH           # path to directory where tool will create test files.  defaults to "/tmp/"
  65          -testbot                        # output results in CoreOS TestBot compatible format
  66
  67 examples:
  68 --- Place all test files and directories at the root of volume "test_vol" ---
  69 xnu_quick_test -t /Volumes/test_vol/
  70
  71 --- Run the tool for tests 10 thru 15, test 18 and test 20 ---
  72 xnu_quick_test -r 10-15, 18, 20
  73
  74
  75 --- example of running the tool to list all the tests it currently supports ---
  76 xnu_quick_test -l
  77 List of all tests this tool performs...
  78  1         syscall
  79  2         fork, wait4, exit
  80  3         fsync, ftruncate, lseek, pread, pwrite, read, readv, truncate, write, writev
  81  4         close, fpathconf, fstat, open, pathconf
  82  5         link, stat, unlink
  83  6         chdir, fchdir
  84  7         access, chmod, fchmod
  85  8         chown, fchown, lchown, lstat, readlink, symlink
  86  9         fstatfs, getattrlist, getfsstat, statfs
  87  10        getpid, getppid, pipe
  88  11        getauid, gettid, getuid, geteuid, issetugid, setaudit_addr, seteuid, settid, settid_with_pid, setuid
  89  12        mkdir, rmdir, umask
  90  13        mknod, sync
  91  14        fsync, getsockopt, poll, select, setsockopt, socketpair
  92  15        accept, bind, connect, getpeername, getsockname, listen, socket, recvmsg, sendmsg, sendto, sendfile
  93  16        chflags, fchflags
  94  17        kill, vfork, execve
  95  18        getegid, getgid, getgroups, setegid, setgid, setgroups
  96  19        dup, dup2, getdtablesize
  97  20        getrusage, profil
  98  21        getitimer, setitimer, sigaction, sigpending, sigprocmask, sigsuspend, sigwait
  99  22        acct
 100  23        ioctl
 101  24        chroot
 102  25        madvise, mincore, minherit, mlock, mlock, mmap, mprotect, msync, munmap
 103  26        getpgrp, getpgid, getsid, setpgid, setpgrp, setsid
 104  27        fcntl
 105  28        getlogin, setlogin
 106  29        getpriority, setpriority
 107  30        futimes, gettimeofday, settimeofday, utimes
 108  31        rename, stat
 109  32        flock
 110  33        mkfifo, read, write
 111  34        quotactl
 112  35        getrlimit, setrlimit
 113  36        getattrlist, getdirentries, getdirentriesattr, setattrlist
 114  37        exchangedata
 115  38        searchfs
 116  39        sem_close, sem_open, sem_post, sem_trywait, sem_unlink, sem_wait
 117  40        semctl, semget, semop
 118  41        shm_open, shm_unlink
 119  42        shmat, shmctl, shmdt, shmget
 120  43        fgetxattr, flistxattr, fremovexattr, fsetxattr, getxattr, listxattr, removexattr, setxattr
 121  44        aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, fcntl, lio_listio
 122  45        kevent, kqueue
 123  46        msgctl, msgget, msgrcv, msgsnd
 124
 125
 126 --- example of a full test run ---
 127 cottje% ./BUILD/dst/xnu_quick_test
 128 created test directory at "/tmp/xnu_quick_test-1660251855"
 129 Will allow 0 failures before testing is aborted
 130
 131 Begin testing - Tue Dec 13 15:56:50 2005
 132
 133 test #1 - syscall
 134 test #2 - fork, wait4, exit
 135 test #3 - fsync, ftruncate, lseek, pread, pwrite, read, readv, truncate, write, writev
 136 test #4 - close, fpathconf, fstat, open, pathconf
 137 test #5 - link, stat, unlink
 138 test #6 - chdir, fchdir
 139 test #7 - access, chmod, fchmod
 140 test #8 - chown, fchown, lchown, lstat, readlink, symlink
 141 test #9 - fstatfs, getattrlist, getfsstat, statfs
 142 test #10 - getpid, getppid, pipe
 143 test #11 - getauid, gettid, getuid, geteuid, issetugid, setauid, seteuid, settid, settid_with_pid, setuid
 144 test #12 - mkdir, rmdir, umask
 145 test #13 - mknod, sync
 146 test #14 - fsync, getsockopt, poll, select, setsockopt, socketpair
 147 test #15 - accept, bind, connect, getpeername, getsockname, listen, socket, recvmsg, sendmsg, sendto
 148 test #16 - chflags, fchflags
 149 test #17 - kill, vfork, execve
 150 test #18 - getegid, getgid, getgroups, setegid, setgid, setgroups
 151 test #19 - dup, dup2, getdtablesize
 152 test #20 - getrusage, profil
 153 test #21 - ktrace
 154 test #22 - getitimer, setitimer, sigaction, sigpending, sigprocmask, sigsuspend, sigwait
 155 test #23 - acct
 156 test #24 - ioctl
 157 test #25 - chroot
 158 test #26 - madvise, mincore, minherit, mlock, mlock, mmap, mprotect, msync, munmap
 159 test #27 - getpgrp, getpgid, getsid, setpgid, setpgrp, setsid
 160 test #28 - fcntl
 161 test #29 - getlogin, setlogin
 162 test #30 - getpriority, setpriority
 163 test #31 - futimes, gettimeofday, settimeofday, utimes
 164 test #32 - rename, stat
 165 test #33 - flock
 166 test #34 - mkfifo, read, write
 167 test #35 - quotactl
 168 test #36 - getrlimit, setrlimit
 169 test #37 - getattrlist, getdirentries, getdirentriesattr, setattrlist
 170 test #38 - exchangedata
 171 test #39 - searchfs
 172 test #40 - sem_close, sem_open, sem_post, sem_trywait, sem_unlink, sem_wait
 173 test #41 - semctl, semget, semop
 174 test #42 - shm_open, shm_unlink
 175 test #43 - shmat, shmctl, shmdt, shmget
 176 test #44 - fgetxattr, flistxattr, fremovexattr, fsetxattr, getxattr, listxattr, removexattr, setxattr
 177 test #45 - aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, fcntl, lio_listio
 178 test #46 - kevent, kqueue
 179 test #47 - msgctl, msgget, msgrcv, msgsnd
 180
 181 End testing - Tue Dec 13 15:57:08 2005