xnu-4570.20.62.tar.gz

[apple/xnu.git] / tools / tests / libMicro / AppleReadMe

Mac OS X specific notes

*** Instructions before Starting libMicro ***

# Disable Open directory and LDAP using Directory Utility app
# Turn off airport
# Turn off spotlight. In terminal, execute the following:
  launchctl unload /System/Library/LaunchDaemons/com.apple.metadata.mds.plist
# Turn off Time Machine in System Preferences
# Wait at least 2 minutes after boot to desktop for boot cache to settle down
   
*** Make and run quickstart ***

	make
	./bench >output.txt
runs the libMicro test suite excluding the lmbench tests and gives you a text file named output.txt with the results of one run.
	

	./multiview output1.txt output2.txt >compare.html
gives you a html file comparing two runs.

*** To run libMicro testsuite with stepper disabled ***
*** For Desktop use coreos_bench script***

To get a more consistent result of libMicro benchmark run, we need to disable the 
stepper to prevent it from causing wide variations in results. See rdar://6243819 
for details.

So to run libMicro test suite with stepper disabled, use 'coreos_bench' script 
instead of 'bench' script.  

For example:
./coreos_bench > output.txt
runs the libMicro test suite excluding the lmbench tests and gives you a text file named output.txt with the results of one run, with stepper disabled.
 
Note: 
1) We need '/usr/local/bin/pstates' to disable the stepper. Install AppleInternal package 
which provides '/usr/local/bin/pstates'.

2) 'coreos_bench' script is used exactly like the 'bench' script. All the usage examples for 
'bench' script in this readme file also holds true for 'coreos_bench' script. 





*** Makefile ***

The Makefile invokes Makefile.Darwin which invokes Makefile.com.Darwin.
Just invoke make, with options if necessary, and everything should
build correctly. The binaries are placed in a directory called
bin-ARCH where ARCH is the default or specified when building via
the ARCH flag.

Note:
1) The binaries of apple added tests are placed in a directory called
   apple/bin-ARCH

2) All the binaries under bin-ARCH and apple/bin-ARCH are code signed
   during build.
 
options for invoking Makefile are:
ARCH 		defaults to  i386

        to build fat/multi architecture, specify 
		make ARCH=fat
	the makefile will automatically build with ARCH_FLAG="-arch i386 -arch x86_64" and put the results in bin-fat

        to build for ARM architecture,
        first set an environment variable 'SDKROOT' to point to iPhone internal sdk
	For example:
		$export SDKROOT="/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.Internal.sdk/"
	to build use:
                make ARCH=ARM_ARCH where ARM_ARCH can be for e.g. armv6, armv7 
        this will put the results in bin-ARM_ARCH

	to build with only two of the architectures see below

ARCH_FLAG 	defaults to -arch $(ARCH)
	to build fat/multi architecture, specify 
		make ARCH_FLAG="-arch i386" ARCH=fat
	this will put the results in bin-fat

OPT_FLAG 	defaults to  -g
	to build optimized, specify make OPT_FLAG=-Os

SEMOP_FLAG 	defaults to  -DUSE_SEMOP
	to eliminate SEMOP usage, specify make SEMOP_FLAG=
	this is needed on some lower-end systems (e.g. M63)

These can be combined, e.g.
	make ARCH=i386 SEMOP_FLAG=

*** Before running benchmarks ***

The shell script create_stuff should be run before any benchmarking

this script takes care of raising the process limits which would
otherwise cause several of the tests to fail - if not you will see:
	Running:           pipe_pst1
	fork: Resource temporarily unavailable
in your stderr during the runs. After you run create_stuff, the
system then needs to be rebooted.

*** running the benchmarks ***

The shell script "bench" will run all the benchmarks, or you can
pass it a parameter to run a single benchmark, e.g.

*** To run libMicro on the embedded platform, use "embd_bench" script. 'embd_bench' script is used exactly like the 'bench' script. All the usage examples for 
'bench' script in this readme file also holds true for 'embd_bench' script. ***

	bench lmbench_bw_unix

By default the script will run only the libMicro testsuite excluding the lmbench tests. 
To run the libmicro testsuite with the lmbench tests included, just pass the -l parameter. e.g,
	
	bench -l 
To run only the lmbench testsuite 

	bench lmbench

To display the usage, just do
	bench -h

Watch for:
	# WARNINGS
	#     Quantization error likely;increase batch size (-B option) 4X to avoid.
in the output
To see an example run the supplied testbench script

Add or adjust the -B parameter for any benchmark that fails. The
Quantization error will refer to the benchmark preceding the error,
not the one following...

A typical run:
	$ make clean
	$ make
	$ ./create_stuff
	$ ./bench > output1
	Running:              getpid
	 for      0.13353 seconds
	Running:             getppid
	 for      3.65609 seconds
	Running:              getenv
	 for      0.20924 seconds
	Running:            getenvT2
	 for      0.37437 seconds
	Running:        gettimeofday
	 for      0.58077 seconds
	etc...

Use the supplied multiview script to compare runs like:

multiview output1 output2 > compare.html
open compare.html (safari launches)
will show output2 results as a percentage change from the output1 results

*** Adding additional benchmark tests ***

Look at the sample file trivial.c.  This demonstrates how to do
argument passing, the flow of control of a benchmark, etc. for the
trivial case.  The tests starting with "lmbench_" were ported from
the lmbench suite, so they might be good examples as well.

*** A note regarding future changes in bench.sh script ***
coreos_bench.sh script is almost identical to bench.sh script, except that it
has additional code to disable the stepper during libmicro benchmark run. 

In future, if bench.sh script is modified, make sure the changes reflect
in coreos_bench.sh script also.

*** Things to do ***

* port the rest of the lmbench benchmarks into this framework

* create website that will allow easy ability to compare many builds
across many machines with historical repository of runs

* document better how to write a benchmark for this framework
(started in trivial.c)

* check this into xnu/test

* create new benchmarks

*** Leopard notes ***

Due to rdar://4654956 and its original, rdar://2588252 you cannot
run these tests on Leopard without removing the cascade_lockf test.
There may be other tests which panic a Leopard system.

*** benchDS notes ***

From rdar://problem/7468995 add the ability to benchmark the key APIs
for server daemons.  In particular, a test binary is added for each of:

	ODQueryCreateWithNode()  (standard User, Groups, and Hosts records)
	getaddrinfo()  (hosts and ports)
	mbr_check_service_membership()
	mbr_check_membership()
	getpwnam()
	getpwuid()
	getgrgid()
	getpwent()
	getgrent()
	getgrnam()

The script benchDS is provided to run a standard set of tests presuming
that the tests are run by root on a system configured with an OD binding.
The OD server (local or remote) must have a set of accounts created with
od_acount_create shell script.  This script must also be run as root, 
and passed a single argument of the number of users to create.  It creates
od_test_{1..N}, and all belong to a ds_test_group1(gid 1211).  In addition,
ds_test_group2(gid 1212) is created which has no users as members.  User ids are
set sequentially from 5000.  In order to administer the OD server, it assumes
user 'diradmin' and password 'admin' are the OD admin.

Also, these tests consult the APIs listed, which can be run against the local
account info, or even Active Directory.

Thus, the quick recipe is:
	Install X Server
	Enable OD, and create directory admin user 'diradmin' with password 'admin'
	As root run:  od_account_create 1000
	Now run the test, as root:  ./benchDS 1000 > output-file


In addition, od_account_delete 1000 will delete the 1000 users created with od_account_create.