tools/tests/xnu_quick_test/README

   1 xnu_quick_test - this tool will do a quick test of every (well, to be
   2 honest most) system calls we support in xnu.
   3
   4 WARNING - this is not meant to be a full regression test of all the
   5 system calls.  The intent is to have a quick test of each system call that
   6 can be run very easily and quickly when a new kernel is built.
   7
   8 This tool is meant to grow as we find xnu problems that could have be
   9 caught before we submit to a build train.  So please add more tests and
  10 make the existing ones better.  Some of the original tests are nothing
  11 more than place holders and quite lame.  Just keep in mind that the tool
  12 should run as fast as possible.  If it gets too slow then most people
  13 will stop running it.
  14
  15 LP64 testing tip - when adding or modifying tests, keep in mind the
  16 variants in the LP64 world.  If xnu gets passed a structure the varies in
  17 size between 32 and 64-bit processes, try to test that a field in the
  18 structure contains valid data.  For example, if we know foo structure
  19 looks like:
  20 struct foo {
  21              int             an_int;
  22              long    a_long;
  23              int             another_int;
  24 }
  25 And if we know what another_int should contain then test for the known
  26 value since it's offset will vary depending on whether the calling process
  27 is 32 or 64 bits.
  28
  29 NOTE - we have several workarounds and test exceptions for some
  30 outstanding bugs in xnu.  All the workarounds are marked with "todo" and
  31 some comments noting the radar number of the offending bug.  Do a search
  32 for "todo" in the source files for this project to locate which tests have
  33 known failures.   And please tag any new exceptions you find with "todo"
  34 in the comment and the radar number of the bug.
  35
  36 Building:
  37 xnu_quick_test is built automatically by BNI for both Mac (10.9 and later), and
  38 iOS (7 and later) trains, and is delivered on AppleInternal builds in
  39 /AppleInternal/CoreOS/xnu_quick_test. It is built as part of the xnu_quick_test
  40 build alias, so you can also find a copy on ~rc at:
  41 ~rc/Software/$RELEASE/Updates/$RELEASEVERSION/Roots/xnu_quick_test/AppleInternal/CoreOS/xnu_quick_test.
  42
  43 Alternatively you can build it yourself using make like so:
  44 SDKROOT=/path/to/sdk make
  45
  46 For example:
  47 # build for Mac, current OS
  48 SDKROOT=/ make
  49 # build for iOS
  50 SDKROOT=`xcodebuild -sdk iphoneos.internal -version Path` make
  51
  52 By default xnu builds all-way fat, but you can restrict this by explicitly
  53 specifying architectures like so:
  54 # build for only armv7 and armv7s
  55 SDKROOT=`xcodebuild -sdk iphoneos.internal -version Path` make ARCH="armv7 armv7s"
  56
  57 There are four defines which you can use at the compile line to build variants.
  58 DEBUG
  59        turn on additional printfs
  60 CONFORMANCE_TESTS_IN_XNU
  61        when conformance tests are in xnu, set this to 1
  62 TEST_SYSTEM_CALLS
  63        test system calls (doesn't compile; a different bug)
  64 RUN_UNDER_TESTBOTS
  65        when running under testbots, set this to 1
  66 by default, all four are set to 0, i.e. disabled.  To build, export
  67 MORECFLAGS with the values you want set, e.g.
  68        export MORECFLAGS="-D DEBUG=1 -D CONFORMANCE_TESTS_IN_XNU=1"
  69
  70 todo:
  71 -- have folks with area expertise fix lame tests
  72    (most of the networking related tests are pretty lame)
  73 -- mach system calls support
  74
  75
  76 USAGE:  xnu_quick_test -target TARGET_PATH
  77
  78          -f[ailures] MAX_FAILS_ALLOWED   # number of test cases that may fail before we give up.  defaults to 0
  79          -l[ist]                         # list all the tests this tool performs
  80          -r[un] 1, 3, 10 - 19            # run specific tests.  enter individual test numbers and/or range of numbers.  use -list to list tests.
  81          -s[kip]                         # skip setuid tests
  82          -t[arget] TARGET_PATH           # path to directory where tool will create test files.  defaults to "/tmp/"
  83          -testbot                        # output results in CoreOS TestBot compatible format
  84
  85 examples:
  86 --- Place all test files and directories at the root of volume "test_vol" ---
  87 xnu_quick_test -t /Volumes/test_vol/
  88
  89 --- Run the tool for tests 10 thru 15, test 18 and test 20 ---
  90 xnu_quick_test -r 10-15, 18, 20
  91
  92
  93 --- example of running the tool to list all the tests it currently supports ---
  94 xnu_quick_test -l
  95 List of all tests this tool performs...
  96  1         syscall
  97  2         fork, wait4, exit
  98  3         fsync, ftruncate, lseek, pread, pwrite, read, readv, truncate, write, writev
  99  4         close, fpathconf, fstat, open, pathconf
 100  5         link, stat, unlink
 101  6         chdir, fchdir
 102  7         access, chmod, fchmod
 103  8         chown, fchown, lchown, lstat, readlink, symlink
 104  9         fstatfs, getattrlist, getfsstat, statfs
 105  10        getpid, getppid, pipe
 106  11        getauid, gettid, getuid, geteuid, issetugid, setaudit_addr, seteuid, settid, settid_with_pid, setuid
 107  12        mkdir, rmdir, umask
 108  13        mknod, sync
 109  14        fsync, getsockopt, poll, select, setsockopt, socketpair
 110  15        accept, bind, connect, getpeername, getsockname, listen, socket, recvmsg, sendmsg, sendto, sendfile
 111  16        chflags, fchflags
 112  17        kill, vfork, execve
 113  18        getegid, getgid, getgroups, setegid, setgid, setgroups
 114  19        dup, dup2, getdtablesize
 115  20        getrusage, profil
 116  21        getitimer, setitimer, sigaction, sigpending, sigprocmask, sigsuspend, sigwait
 117  22        acct
 118  23        ioctl
 119  24        chroot
 120  25        madvise, mincore, minherit, mlock, mlock, mmap, mprotect, msync, munmap
 121  26        getpgrp, getpgid, getsid, setpgid, setpgrp, setsid
 122  27        fcntl
 123  28        getlogin, setlogin
 124  29        getpriority, setpriority
 125  30        futimes, gettimeofday, settimeofday, utimes
 126  31        rename, stat
 127  32        flock
 128  33        mkfifo, read, write
 129  34        quotactl
 130  35        getrlimit, setrlimit
 131  36        getattrlist, getdirentries, getdirentriesattr, setattrlist
 132  37        exchangedata
 133  38        searchfs
 134  39        sem_close, sem_open, sem_post, sem_trywait, sem_unlink, sem_wait
 135  40        semctl, semget, semop
 136  41        shm_open, shm_unlink
 137  42        shmat, shmctl, shmdt, shmget
 138  43        fgetxattr, flistxattr, fremovexattr, fsetxattr, getxattr, listxattr, removexattr, setxattr
 139  44        aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, fcntl, lio_listio
 140  45        kevent, kqueue
 141  46        msgctl, msgget, msgrcv, msgsnd
 142
 143
 144 --- example of a full test run ---
 145 cottje% ./BUILD/dst/xnu_quick_test
 146 created test directory at "/tmp/xnu_quick_test-1660251855"
 147 Will allow 0 failures before testing is aborted
 148
 149 Begin testing - Tue Dec 13 15:56:50 2005
 150
 151 test #1 - syscall
 152 test #2 - fork, wait4, exit
 153 test #3 - fsync, ftruncate, lseek, pread, pwrite, read, readv, truncate, write, writev
 154 test #4 - close, fpathconf, fstat, open, pathconf
 155 test #5 - link, stat, unlink
 156 test #6 - chdir, fchdir
 157 test #7 - access, chmod, fchmod
 158 test #8 - chown, fchown, lchown, lstat, readlink, symlink
 159 test #9 - fstatfs, getattrlist, getfsstat, statfs
 160 test #10 - getpid, getppid, pipe
 161 test #11 - getauid, gettid, getuid, geteuid, issetugid, setauid, seteuid, settid, settid_with_pid, setuid
 162 test #12 - mkdir, rmdir, umask
 163 test #13 - mknod, sync
 164 test #14 - fsync, getsockopt, poll, select, setsockopt, socketpair
 165 test #15 - accept, bind, connect, getpeername, getsockname, listen, socket, recvmsg, sendmsg, sendto
 166 test #16 - chflags, fchflags
 167 test #17 - kill, vfork, execve
 168 test #18 - getegid, getgid, getgroups, setegid, setgid, setgroups
 169 test #19 - dup, dup2, getdtablesize
 170 test #20 - getrusage, profil
 171 test #21 - ktrace
 172 test #22 - getitimer, setitimer, sigaction, sigpending, sigprocmask, sigsuspend, sigwait
 173 test #23 - acct
 174 test #24 - ioctl
 175 test #25 - chroot
 176 test #26 - madvise, mincore, minherit, mlock, mlock, mmap, mprotect, msync, munmap
 177 test #27 - getpgrp, getpgid, getsid, setpgid, setpgrp, setsid
 178 test #28 - fcntl
 179 test #29 - getlogin, setlogin
 180 test #30 - getpriority, setpriority
 181 test #31 - futimes, gettimeofday, settimeofday, utimes
 182 test #32 - rename, stat
 183 test #33 - flock
 184 test #34 - mkfifo, read, write
 185 test #35 - quotactl
 186 test #36 - getrlimit, setrlimit
 187 test #37 - getattrlist, getdirentries, getdirentriesattr, setattrlist
 188 test #38 - exchangedata
 189 test #39 - searchfs
 190 test #40 - sem_close, sem_open, sem_post, sem_trywait, sem_unlink, sem_wait
 191 test #41 - semctl, semget, semop
 192 test #42 - shm_open, shm_unlink
 193 test #43 - shmat, shmctl, shmdt, shmget
 194 test #44 - fgetxattr, flistxattr, fremovexattr, fsetxattr, getxattr, listxattr, removexattr, setxattr
 195 test #45 - aio_cancel, aio_error, aio_read, aio_return, aio_suspend, aio_write, fcntl, lio_listio
 196 test #46 - kevent, kqueue
 197 test #47 - msgctl, msgget, msgrcv, msgsnd
 198
 199 End testing - Tue Dec 13 15:57:08 2005