]> git.saurik.com Git - apple/xnu.git/blame - iokit/Kernel/IOHibernateIO.cpp
projects / apple / xnu.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
xnu-1699.32.7.tar.gz
[apple/xnu.git] / iokit / Kernel / IOHibernateIO.cpp
CommitLineData
3a60a9f5 1/*
b0d623f7 2 * Copyright (c) 2004-2008 Apple Computer, Inc. All rights reserved.
3a60a9f5 3 *
2d21ac55 4 * @APPLE_OSREFERENCE_LICENSE_HEADER_START@
3a60a9f5 5 *
2d21ac55
A		 6 * This file contains Original Code and/or Modifications of Original Code
7 * as defined in and that are subject to the Apple Public Source License
8 * Version 2.0 (the 'License'). You may not use this file except in
9 * compliance with the License. The rights granted to you under the License
10 * may not be used to create, or enable the creation or redistribution of,
11 * unlawful or unlicensed copies of an Apple operating system, or to
12 * circumvent, violate, or enable the circumvention or violation of, any
13 * terms of an Apple operating system software license agreement.
8f6c56a5 14 *
2d21ac55
A		 15 * Please obtain a copy of the License at
16 * http://www.opensource.apple.com/apsl/ and read it before using this file.
17 *
18 * The Original Code and all software distributed under the License are
19 * distributed on an 'AS IS' basis, WITHOUT WARRANTY OF ANY KIND, EITHER
8f6c56a5
A		 20 * EXPRESS OR IMPLIED, AND APPLE HEREBY DISCLAIMS ALL SUCH WARRANTIES,
21 * INCLUDING WITHOUT LIMITATION, ANY WARRANTIES OF MERCHANTABILITY,
2d21ac55
A		 22 * FITNESS FOR A PARTICULAR PURPOSE, QUIET ENJOYMENT OR NON-INFRINGEMENT.
23 * Please see the License for the specific language governing rights and
24 * limitations under the License.
8f6c56a5 25 *
2d21ac55 26 * @APPLE_OSREFERENCE_LICENSE_HEADER_END@
3a60a9f5
A		 27 */
28
29
30/*
31
32Sleep:
33
34- PMRootDomain calls IOHibernateSystemSleep() before system sleep
35(devices awake, normal execution context)
36- IOHibernateSystemSleep opens the hibernation file (or partition) at the bsd level,
37 grabs its extents and searches for a polling driver willing to work with that IOMedia.
38 The BSD code makes an ioctl to the storage driver to get the partition base offset to
39 the disk, and other ioctls to get the transfer constraints
40 If successful, the file is written to make sure its initially not bootable (in case of
41 later failure) and nvram set to point to the first block of the file. (Has to be done
42 here so blocking is possible in nvram support).
43 hibernate_setup() in osfmk is called to allocate page bitmaps for all dram, and
44 page out any pages it wants to (currently zero, but probably some percentage of memory).
45 Its assumed just allocating pages will cause the VM system to naturally select the best
46 pages for eviction. It also copies processor flags needed for the restore path and sets
47 a flag in the boot processor proc info.
48 gIOHibernateState = kIOHibernateStateHibernating.
49- Regular sleep progresses - some drivers may inspect the root domain property
50 kIOHibernateStateKey to modify behavior. The platform driver saves state to memory
51 as usual but leaves motherboard I/O on.
52- Eventually the platform calls ml_ppc_sleep() in the shutdown context on the last cpu,
53 at which point memory is ready to be saved. mapping_hibernate_flush() is called to get
54 all ppc RC bits out of the hash table and caches into the mapping structures.
55- hibernate_write_image() is called (still in shutdown context, no blocking or preemption).
56 hibernate_page_list_setall() is called to get a bitmap of dram pages that need to be saved.
57 All pages are assumed to be saved (as part of the wired image) unless explicitly subtracted
58 by hibernate_page_list_setall(), avoiding having to find arch dependent low level bits.
59 The image header and block list are written. The header includes the second file extent so
60 only the header block is needed to read the file, regardless of filesystem.
61 The kernel section "__HIB" is written uncompressed to the image. This section of code and data
62 (only) is used to decompress the image during wake/boot.
63 Some additional pages are removed from the bitmaps - the buffers used for hibernation.
64 The bitmaps are written to the image.
65 More areas are removed from the bitmaps (after they have been written to the image) - the
66 section "__HIB" pages and interrupt stack.
67 Each wired page is compressed and written and then each non-wired page. Compression and
68 disk writes are in parallel.
69 The image header is written to the start of the file and the polling driver closed.
70 The machine powers down (or sleeps).
71
72Boot/Wake:
73
74- BootX sees the boot-image nvram variable containing the device and block number of the image,
75 reads the header and if the signature is correct proceeds. The boot-image variable is cleared.
76- BootX reads the portion of the image used for wired pages, to memory. Its assumed this will fit
77 in the OF memory environment, and the image is decrypted. There is no decompression in BootX,
78 that is in the kernel's __HIB section.
79- BootX copies the "__HIB" section to its correct position in memory, quiesces and calls its entry
80 hibernate_kernel_entrypoint(), passing the location of the image in memory. Translation is off,
81 only code & data in that section is safe to call since all the other wired pages are still
82 compressed in the image.
83- hibernate_kernel_entrypoint() removes pages occupied by the raw image from the page bitmaps.
84 It uses the bitmaps to work out which pages can be uncompressed from the image to their final
85 location directly, and copies those that can't to interim free pages. When the image has been
86 completed, the copies are uncompressed, overwriting the wired image pages.
87 hibernate_restore_phys_page() (in osfmk since its arch dependent, but part of the "__HIB" section)
88 is used to get pages into place for 64bit.
89- the reset vector is called (at least on ppc), the kernel proceeds on a normal wake, with some
90 changes conditional on the per proc flag - before VM is turned on the boot cpu, all mappings
91 are removed from the software strutures, and the hash table is reinitialized.
92- After the platform CPU init code is called, hibernate_machine_init() is called to restore the rest
93 of memory, using the polled mode driver, before other threads can run or any devices are turned on.
94 This reduces the memory usage for BootX and allows decompression in parallel with disk reads,
95 for the remaining non wired pages.
96- The polling driver is closed down and regular wake proceeds. When the kernel calls iokit to wake
97 (normal execution context) hibernate_teardown() in osmfk is called to release any memory, the file
98 is closed via bsd.
99
100Polled Mode I/O:
101
102IOHibernateSystemSleep() finds a polled mode interface to the ATA controller via a property in the
103registry, specifying an object of calls IOPolledInterface.
104
105Before the system goes to sleep it searches from the IOMedia object (could be a filesystem or
106partition) that the image is going to live, looking for polled interface properties. If it finds
107one the IOMedia object is passed to a "probe" call for the interface to accept or reject. All the
108interfaces found are kept in an ordered list.
109
110There is an Open/Close pair of calls made to each of the interfaces at various stages since there are
111few different contexts things happen in:
112
113- there is an Open/Close (Preflight) made before any part of the system has slept (I/O is all
114up and running) and after wake - this is safe to allocate memory and do anything. The device
115ignores sleep requests from that point since its a waste of time if it goes to sleep and
116immediately wakes back up for the image write.
117
118- there is an Open/Close (BeforeSleep) pair made around the image write operations that happen
119immediately before sleep. These can't block or allocate memory - the I/O system is asleep apart
120from the low level bits (motherboard I/O etc). There is only one thread running. The close can be
121used to flush and set the disk to sleep.
122
123- there is an Open/Close (AfterSleep) pair made around the image read operations that happen
124immediately after sleep. These can't block or allocate memory. This is happening after the platform
125expert has woken the low level bits of the system, but most of the I/O system has not. There is only
126one thread running.
127
128For the actual I/O, all the ops are with respect to a single IOMemoryDescriptor that was passed
129(prepared) to the Preflight Open() call. There is a read/write op, buffer offset to the IOMD for
130the data, an offset to the disk and length (block aligned 64 bit numbers), and completion callback.
131Each I/O is async but only one is ever outstanding. The polled interface has a checkForWork call
132that is called for the hardware to check for events, and complete the I/O via the callback.
133The hibernate path uses the same transfer constraints the regular cluster I/O path in BSD uses
134to restrict I/O ops.
135*/
136
137#include <sys/systm.h>
138
139#include <IOKit/IOWorkLoop.h>
140#include <IOKit/IOCommandGate.h>
141#include <IOKit/IOTimerEventSource.h>
142#include <IOKit/IOPlatformExpert.h>
143#include <IOKit/IOKitDebug.h>
144#include <IOKit/IOTimeStamp.h>
145#include <IOKit/pwr_mgt/RootDomain.h>
146#include <IOKit/pwr_mgt/IOPMPrivate.h>
147#include <IOKit/IOMessage.h>
148#include <IOKit/IODeviceTreeSupport.h>
149#include <IOKit/IOBSD.h>
150#include "RootDomainUserClient.h"
151#include <IOKit/pwr_mgt/IOPowerConnection.h>
152#include "IOPMPowerStateQueue.h"
153#include <IOKit/IOBufferMemoryDescriptor.h>
6d2010ae 154#include <IOKit/AppleKeyStoreInterface.h>
3a60a9f5
A		 155#include <crypto/aes.h>
156
157#include <sys/uio.h>
158#include <sys/conf.h>
159#include <sys/stat.h>
160#include <sys/fcntl.h> // (FWRITE, ...)
3a60a9f5 161#include <sys/sysctl.h>
6d2010ae 162#include <sys/kdebug.h>
3a60a9f5
A		 163
164#include <IOKit/IOHibernatePrivate.h>
165#include <IOKit/IOPolledInterface.h>
166#include <IOKit/IONVRAM.h>
167#include "IOHibernateInternal.h"
6d2010ae 168#include <libkern/WKdm.h>
3a60a9f5 169#include "IOKitKernelInternal.h"
6d2010ae
A		 170#include <pexpert/device_tree.h>
171
172#include <machine/pal_routines.h>
173#include <machine/pal_hibernate.h>
174
175extern "C" addr64_t kvtophys(vm_offset_t va);
3a60a9f5
A		 176
177/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
178
0b4c1975 179extern unsigned int save_kdebug_enable;
3a60a9f5
A		 180extern uint32_t gIOHibernateState;
181uint32_t gIOHibernateMode;
182static char gIOHibernateBootSignature[256+1];
183static char gIOHibernateFilename[MAXPATHLEN+1];
184static uint32_t gIOHibernateFreeRatio = 0; // free page target (percent)
185uint32_t gIOHibernateFreeTime = 0*1000; // max time to spend freeing pages (ms)
186
187static IODTNVRAM * gIOOptionsEntry;
188static IORegistryEntry * gIOChosenEntry;
b0d623f7 189#if defined(__i386__) || defined(__x86_64__)
0c530ab8 190static const OSSymbol * gIOCreateEFIDevicePathSymbol;
060df5ea
A		 191static const OSSymbol * gIOHibernateRTCVariablesKey;
192static const OSSymbol * gIOHibernateBoot0082Key;
193static const OSSymbol * gIOHibernateBootNextKey;
194static OSData * gIOHibernateBoot0082Data;
195static OSData * gIOHibernateBootNextData;
196static OSObject * gIOHibernateBootNextSave;
0c530ab8 197#endif
3a60a9f5
A		 198
199static IOPolledFileIOVars gFileVars;
200static IOHibernateVars gIOHibernateVars;
201static struct kern_direct_file_io_ref_t * gIOHibernateFileRef;
202static hibernate_cryptvars_t gIOHibernateCryptWakeContext;
6d2010ae
A		 203static hibernate_graphics_t _hibernateGraphics;
204static hibernate_graphics_t * gIOHibernateGraphicsInfo = &_hibernateGraphics;
3a60a9f5
A		 205
206/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
207
208enum { kXPRamAudioVolume = 8 };
209enum { kDefaultIOSize = 128 * 1024 };
210enum { kVideoMapSize = 32 * 1024 * 1024 };
211
212#ifndef kIOMediaPreferredBlockSizeKey
213#define kIOMediaPreferredBlockSizeKey "Preferred Block Size"
214#endif
215
216#ifndef kIOBootPathKey
217#define kIOBootPathKey "bootpath"
218#endif
219#ifndef kIOSelectedBootDeviceKey
220#define kIOSelectedBootDeviceKey "boot-device"
221#endif
222
223
224enum { kIOHibernateMinPollersNeeded = 2 };
225
226/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
227
228// copy from phys addr to MD
229
230static IOReturn
231IOMemoryDescriptorWriteFromPhysical(IOMemoryDescriptor * md,
232 IOByteCount offset, addr64_t bytes, IOByteCount length)
233{
234 addr64_t srcAddr = bytes;
235 IOByteCount remaining;
236
237 remaining = length = min(length, md->getLength() - offset);
238 while (remaining) { // (process another target segment?)
239 addr64_t dstAddr64;
240 IOByteCount dstLen;
241
b0d623f7 242 dstAddr64 = md->getPhysicalSegment(offset, &dstLen, kIOMemoryMapperNone);
3a60a9f5
A		 243 if (!dstAddr64)
244 break;
245
246 // Clip segment length to remaining
247 if (dstLen > remaining)
248 dstLen = remaining;
249
250#if 1
251 bcopy_phys(srcAddr, dstAddr64, dstLen);
252#else
253 copypv(srcAddr, dstAddr64, dstLen,
254 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
255#endif
256 srcAddr += dstLen;
257 offset += dstLen;
258 remaining -= dstLen;
259 }
260
261 assert(!remaining);
262
263 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
264}
265
266// copy from MD to phys addr
267
268static IOReturn
269IOMemoryDescriptorReadToPhysical(IOMemoryDescriptor * md,
270 IOByteCount offset, addr64_t bytes, IOByteCount length)
271{
272 addr64_t dstAddr = bytes;
273 IOByteCount remaining;
274
275 remaining = length = min(length, md->getLength() - offset);
276 while (remaining) { // (process another target segment?)
277 addr64_t srcAddr64;
278 IOByteCount dstLen;
279
b0d623f7 280 srcAddr64 = md->getPhysicalSegment(offset, &dstLen, kIOMemoryMapperNone);
3a60a9f5
A		 281 if (!srcAddr64)
282 break;
283
284 // Clip segment length to remaining
285 if (dstLen > remaining)
286 dstLen = remaining;
287
288#if 1
289 bcopy_phys(srcAddr64, dstAddr, dstLen);
290#else
291 copypv(srcAddr, dstAddr64, dstLen,
292 cppvPsnk | cppvFsnk | cppvNoRefSrc | cppvNoModSnk | cppvKmap);
293#endif
294 dstAddr += dstLen;
295 offset += dstLen;
296 remaining -= dstLen;
297 }
298
299 assert(!remaining);
300
301 return remaining ? kIOReturnUnderrun : kIOReturnSuccess;
302}
303
304/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
305
306void
307hibernate_set_page_state(hibernate_page_list_t * page_list, hibernate_page_list_t * page_list_wired,
308 vm_offset_t ppnum, vm_offset_t count, uint32_t kind)
309{
310 count += ppnum;
311 switch (kind)
312 {
313 case kIOHibernatePageStateUnwiredSave:
314 // unwired save
315 for (; ppnum < count; ppnum++)
316 {
317 hibernate_page_bitset(page_list, FALSE, ppnum);
318 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
319 }
320 break;
321 case kIOHibernatePageStateWiredSave:
322 // wired save
323 for (; ppnum < count; ppnum++)
324 {
325 hibernate_page_bitset(page_list, FALSE, ppnum);
326 hibernate_page_bitset(page_list_wired, FALSE, ppnum);
327 }
328 break;
329 case kIOHibernatePageStateFree:
330 // free page
331 for (; ppnum < count; ppnum++)
332 {
333 hibernate_page_bitset(page_list, TRUE, ppnum);
334 hibernate_page_bitset(page_list_wired, TRUE, ppnum);
335 }
336 break;
337 default:
338 panic("hibernate_set_page_state");
339 }
340}
341
342static vm_offset_t
0c530ab8 343hibernate_page_list_iterate(hibernate_page_list_t * list, vm_offset_t * pPage)
3a60a9f5 344{
0c530ab8
A		 345 uint32_t page = *pPage;
346 uint32_t count;
347 hibernate_bitmap_t * bitmap;
21362eb3 348
0c530ab8
A		 349 while ((bitmap = hibernate_page_bitmap_pin(list, &page)))
350 {
351 count = hibernate_page_bitmap_count(bitmap, TRUE, page);
352 if (!count)
353 break;
354 page += count;
355 if (page <= bitmap->last_page)
356 break;
357 }
6601e61a 358
0c530ab8
A		 359 *pPage = page;
360 if (bitmap)
361 count = hibernate_page_bitmap_count(bitmap, FALSE, page);
362 else
363 count = 0;
3a60a9f5
A		 364
365 return (count);
366}
367
368/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
369
370static IOReturn
371IOHibernatePollerProbe(IOPolledFileIOVars * vars, IOService * target)
372{
373 IOReturn err = kIOReturnError;
374 int32_t idx;
375 IOPolledInterface * poller;
376
377 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
378 {
379 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
380 err = poller->probe(target);
381 if (err)
382 {
383 HIBLOG("IOPolledInterface::probe[%d] 0x%x\n", idx, err);
384 break;
385 }
386 }
387
388 return (err);
389}
390
391static IOReturn
392IOHibernatePollerOpen(IOPolledFileIOVars * vars, uint32_t state, IOMemoryDescriptor * md)
393{
394 IOReturn err = kIOReturnError;
395 int32_t idx;
396 IOPolledInterface * poller;
397
398 for (idx = vars->pollers->getCount() - 1; idx >= 0; idx--)
399 {
400 poller = (IOPolledInterface *) vars->pollers->getObject(idx);
401 err = poller->open(state, md);
402 if (err)
403 {
404 HIBLOG("IOPolledInterface::open[%d] 0x%x\n", idx, err);
405 break;
406 }
407 }
408
409 return (err);
410}
411
412static IOReturn
413IOHibernatePollerClose(IOPolledFileIOVars * vars, uint32_t state)
414{
415 IOReturn err = kIOReturnError;
416 int32_t idx;
417 IOPolledInterface * poller;
418
419 for (idx = 0;
420 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
421 idx++)
422 {
423 err = poller->close(state);
424 if (err)
425 HIBLOG("IOPolledInterface::close[%d] 0x%x\n", idx, err);
426 }
427
428 return (err);
429}
430
431static void
432IOHibernatePollerIOComplete(void * target,
433 void * parameter,
434 IOReturn status,
435 UInt64 actualByteCount)
436{
437 IOPolledFileIOVars * vars = (IOPolledFileIOVars *) parameter;
438
439 vars->ioStatus = status;
440}
441
442static IOReturn
443IOHibernatePollerIO(IOPolledFileIOVars * vars,
444 uint32_t operation, uint32_t bufferOffset,
445 uint64_t deviceOffset, uint64_t length)
446{
447
448 IOReturn err = kIOReturnError;
449 IOPolledInterface * poller;
450 IOPolledCompletion completion;
451
452 completion.target = 0;
453 completion.action = &IOHibernatePollerIOComplete;
454 completion.parameter = vars;
455
456 vars->ioStatus = -1;
457
458 poller = (IOPolledInterface *) vars->pollers->getObject(0);
459 err = poller->startIO(operation, bufferOffset, deviceOffset + vars->block0, length, completion);
460 if (err)
461 HIBLOG("IOPolledInterface::startIO[%d] 0x%x\n", 0, err);
462
463 return (err);
464}
465
466static IOReturn
2d21ac55 467IOHibernatePollerIODone(IOPolledFileIOVars * vars, bool abortable)
3a60a9f5 468{
2d21ac55
A		 469 IOReturn err = kIOReturnSuccess;
470 int32_t idx = 0;
3a60a9f5
A		 471 IOPolledInterface * poller;
472
473 while (-1 == vars->ioStatus)
474 {
2d21ac55
A		 475 for (idx = 0;
476 (poller = (IOPolledInterface *) vars->pollers->getObject(idx));
3a60a9f5
A		 477 idx++)
478 {
2d21ac55
A		 479 IOReturn newErr;
480 newErr = poller->checkForWork();
481 if ((newErr == kIOReturnAborted) && !abortable)
482 newErr = kIOReturnSuccess;
483 if (kIOReturnSuccess == err)
484 err = newErr;
3a60a9f5
A		 485 }
486 }
487
0b4c1975
A		 488 if ((kIOReturnSuccess == err) && abortable && hibernate_should_abort())
489 {
490 err = kIOReturnAborted;
491 HIBLOG("IOPolledInterface::checkForWork sw abort\n");
492 }
493
2d21ac55
A		 494 if (err)
495 {
496 HIBLOG("IOPolledInterface::checkForWork[%d] 0x%x\n", idx, err);
497 }
498 else
499 {
500 err = vars->ioStatus;
501 if (kIOReturnSuccess != err)
502 HIBLOG("IOPolledInterface::ioStatus 0x%x\n", err);
503 }
3a60a9f5 504
2d21ac55 505 return (err);
3a60a9f5
A		 506}
507
508IOReturn
509IOPolledInterface::checkAllForWork(void)
510{
511 IOReturn err = kIOReturnNotReady;
512 int32_t idx;
513 IOPolledInterface * poller;
514
515 IOHibernateVars * vars = &gIOHibernateVars;
516
517 if (!vars->fileVars || !vars->fileVars->pollers)
518 return (err);
519
520 for (idx = 0;
521 (poller = (IOPolledInterface *) vars->fileVars->pollers->getObject(idx));
522 idx++)
523 {
524 err = poller->checkForWork();
525 if (err)
526 HIBLOG("IOPolledInterface::checkAllForWork[%d] 0x%x\n", idx, err);
527 }
528
529 return (err);
530}
531
532/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
533
534struct _OpenFileContext
535{
536 OSData * extents;
537 uint64_t size;
538};
539
540static void
541file_extent_callback(void * ref, uint64_t start, uint64_t length)
542{
543 _OpenFileContext * ctx = (_OpenFileContext *) ref;
544 IOPolledFileExtent extent;
545
546 extent.start = start;
547 extent.length = length;
548
549 ctx->extents->appendBytes(&extent, sizeof(extent));
550 ctx->size += length;
551}
552
6d2010ae
A		 553static IOService *
554IOCopyMediaForDev(dev_t device)
555{
556 OSDictionary * matching;
557 OSNumber * num;
558 OSIterator * iter;
559 IOService * result = 0;
560
561 matching = IOService::serviceMatching("IOMedia");
562 if (!matching)
563 return (0);
564 do
565 {
566 num = OSNumber::withNumber(major(device), 32);
567 if (!num)
568 break;
569 matching->setObject(kIOBSDMajorKey, num);
570 num->release();
571 num = OSNumber::withNumber(minor(device), 32);
572 if (!num)
573 break;
574 matching->setObject(kIOBSDMinorKey, num);
575 num->release();
576 if (!num)
577 break;
578 iter = IOService::getMatchingServices(matching);
579 if (iter)
580 {
581 result = (IOService *) iter->getNextObject();
582 result->retain();
583 iter->release();
584 }
585 }
586 while (false);
587 matching->release();
588
589 return (result);
590}
591
3a60a9f5
A		 592IOReturn
593IOPolledFileOpen( const char * filename, IOBufferMemoryDescriptor * ioBuffer,
594 IOPolledFileIOVars ** fileVars, OSData ** fileExtents,
6d2010ae 595 OSData ** imagePath, uint8_t * volumeCryptKey)
3a60a9f5
A		 596{
597 IOReturn err = kIOReturnError;
598 IOPolledFileIOVars * vars;
599 _OpenFileContext ctx;
600 OSData * extentsData;
601 OSNumber * num;
6d2010ae
A		 602 IOService * part = 0;
603 OSString * keyUUID = 0;
604 OSString * keyStoreUUID = 0;
605 dev_t block_dev;
3a60a9f5
A		 606 dev_t hibernate_image_dev;
607 uint64_t maxiobytes;
608
609 vars = &gFileVars;
610 do
611 {
612 HIBLOG("sizeof(IOHibernateImageHeader) == %ld\n", sizeof(IOHibernateImageHeader));
613 if (sizeof(IOHibernateImageHeader) != 512)
614 continue;
615
616 vars->io = false;
617 vars->buffer = (uint8_t *) ioBuffer->getBytesNoCopy();
618 vars->bufferHalf = 0;
619 vars->bufferOffset = 0;
620 vars->bufferSize = ioBuffer->getLength() >> 1;
621
622 extentsData = OSData::withCapacity(32);
623
624 ctx.extents = extentsData;
625 ctx.size = 0;
626 vars->fileRef = kern_open_file_for_direct_io(filename,
627 &file_extent_callback, &ctx,
6d2010ae 628 &block_dev,
3a60a9f5
A		 629 &hibernate_image_dev,
630 &vars->block0,
0b4c1975 631 &maxiobytes,
6d2010ae
A		 632 &vars->flags,
633 0, (caddr_t) gIOHibernateCurrentHeader,
634 sizeof(IOHibernateImageHeader));
3a60a9f5
A		 635 if (!vars->fileRef)
636 {
637 err = kIOReturnNoSpace;
638 break;
639 }
b0d623f7 640 gIOHibernateFileRef = vars->fileRef;
0b4c1975
A		 641
642 if (kIOHibernateModeSSDInvert & gIOHibernateMode)
6d2010ae 643 vars->flags ^= kIOHibernateOptionSSD;
0b4c1975
A		 644
645 HIBLOG("Opened file %s, size %qd, partition base 0x%qx, maxio %qx ssd %d\n", filename, ctx.size,
6d2010ae 646 vars->block0, maxiobytes, kIOHibernateOptionSSD & vars->flags);
3a60a9f5
A		 647 if (ctx.size < 1*1024*1024) // check against image size estimate!
648 {
649 err = kIOReturnNoSpace;
650 break;
651 }
652
653 if (maxiobytes < vars->bufferSize)
654 vars->bufferSize = maxiobytes;
655
656 vars->extentMap = (IOPolledFileExtent *) extentsData->getBytesNoCopy();
2d21ac55 657
6d2010ae
A		 658 part = IOCopyMediaForDev(block_dev);
659 if (!part)
660 break;
661
662 err = part->callPlatformFunction(PLATFORM_FUNCTION_GET_MEDIA_ENCRYPTION_KEY_UUID, false,
663 (void *) &keyUUID, (void *) &keyStoreUUID, NULL, NULL);
664 if ((kIOReturnSuccess == err) && keyUUID && keyStoreUUID)
665 {
666// IOLog("got volume key %s\n", keyStoreUUID->getCStringNoCopy());
667 uuid_t volumeKeyUUID;
668 aks_volume_key_t vek;
669 static IOService * sKeyStore;
670 static const OSSymbol * sAKSGetKey;
671
672 if (!sAKSGetKey)
673 sAKSGetKey = OSSymbol::withCStringNoCopy(AKS_PLATFORM_FUNCTION_GETKEY);
674 if (!sKeyStore)
675 sKeyStore = (IOService *) IORegistryEntry::fromPath(AKS_SERVICE_PATH, gIOServicePlane);
676 if (sKeyStore)
677 err = uuid_parse(keyStoreUUID->getCStringNoCopy(), volumeKeyUUID);
678 else
679 err = kIOReturnNoResources;
680 if (kIOReturnSuccess == err)
681 err = sKeyStore->callPlatformFunction(sAKSGetKey, true, volumeKeyUUID, &vek, NULL, NULL);
682 if (kIOReturnSuccess != err)
683 IOLog("volume key err 0x%x\n", err);
684 else
685 {
686 size_t bytes = (kIOHibernateAESKeySize / 8);
687 if (vek.key.keybytecount < bytes)
688 bytes = vek.key.keybytecount;
689 bcopy(&vek.key.keybytes[0], volumeCryptKey, bytes);
690 }
691 bzero(&vek, sizeof(vek));
692 }
693 part->release();
694
695 part = IOCopyMediaForDev(hibernate_image_dev);
696 if (!part)
697 break;
698
3a60a9f5
A		 699 IORegistryEntry * next;
700 IORegistryEntry * child;
701 OSData * data;
702
3a60a9f5
A		 703 vars->pollers = OSArray::withCapacity(4);
704 if (!vars->pollers)
705 break;
706
707 vars->blockSize = 512;
708 next = part;
709 do
710 {
711 IOPolledInterface * poller;
c0fea474
A		 712 OSObject * obj;
713
714 obj = next->getProperty(kIOPolledInterfaceSupportKey);
715 if (kOSBooleanFalse == obj)
716 {
717 vars->pollers->flushCollection();
718 break;
719 }
720 else if ((poller = OSDynamicCast(IOPolledInterface, obj)))
3a60a9f5
A		 721 vars->pollers->setObject(poller);
722 if ((num = OSDynamicCast(OSNumber, next->getProperty(kIOMediaPreferredBlockSizeKey))))
723 vars->blockSize = num->unsigned32BitValue();
724 child = next;
725 }
726 while ((next = child->getParentEntry(gIOServicePlane))
727 && child->isParent(next, gIOServicePlane, true));
728
729 HIBLOG("hibernate image major %d, minor %d, blocksize %ld, pollers %d\n",
6d2010ae 730 major(hibernate_image_dev), minor(hibernate_image_dev), (long)vars->blockSize, vars->pollers->getCount());
3a60a9f5
A		 731 if (vars->pollers->getCount() < kIOHibernateMinPollersNeeded)
732 continue;
733
734 err = IOHibernatePollerProbe(vars, (IOService *) part);
735 if (kIOReturnSuccess != err)
736 break;
737
738 err = IOHibernatePollerOpen(vars, kIOPolledPreflightState, ioBuffer);
739 if (kIOReturnSuccess != err)
740 break;
741
742 *fileVars = vars;
743 *fileExtents = extentsData;
744
745 // make imagePath
3a60a9f5 746
0c530ab8 747 if ((extentsData->getLength() >= sizeof(IOPolledFileExtent)))
3a60a9f5 748 {
6d2010ae 749 char str2[24 + sizeof(uuid_string_t) + 2];
0c530ab8 750
b0d623f7 751#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 752 if (!gIOCreateEFIDevicePathSymbol)
753 gIOCreateEFIDevicePathSymbol = OSSymbol::withCString("CreateEFIDevicePath");
754
6d2010ae
A		 755 if (keyUUID)
756 snprintf(str2, sizeof(str2), "%qx:%s",
757 vars->extentMap[0].start, keyUUID->getCStringNoCopy());
758 else
759 snprintf(str2, sizeof(str2), "%qx", vars->extentMap[0].start);
0c530ab8
A		 760
761 err = IOService::getPlatform()->callPlatformFunction(
762 gIOCreateEFIDevicePathSymbol, false,
6d2010ae
A		 763 (void *) part, (void *) str2,
764 (void *) (uintptr_t) true, (void *) &data);
0c530ab8
A		 765#else
766 char str1[256];
767 int len = sizeof(str1);
768
769 if (!part->getPath(str1, &len, gIODTPlane))
770 err = kIOReturnNotFound;
771 else
772 {
2d21ac55 773 snprintf(str2, sizeof(str2), ",%qx", vars->extentMap[0].start);
0c530ab8
A		 774 // (strip the plane name)
775 char * tail = strchr(str1, ':');
776 if (!tail)
777 tail = str1 - 1;
778 data = OSData::withBytes(tail + 1, strlen(tail + 1));
779 data->appendBytes(str2, strlen(str2));
780 }
781#endif
782 if (kIOReturnSuccess == err)
3a60a9f5 783 *imagePath = data;
0c530ab8
A		 784 else
785 HIBLOG("error 0x%x getting path\n", err);
3a60a9f5
A		 786 }
787 }
788 while (false);
789
790 if (kIOReturnSuccess != err)
791 {
792 HIBLOG("error 0x%x opening hibernation file\n", err);
793 if (vars->fileRef)
b0d623f7 794 {
7ddcb079 795 kern_close_file_for_direct_io(vars->fileRef, 0, 0, 0, 0, 0);
b0d623f7
A		 796 gIOHibernateFileRef = vars->fileRef = NULL;
797 }
3a60a9f5
A		 798 }
799
800 if (part)
801 part->release();
802
803 return (err);
804}
805
806IOReturn
807IOPolledFileClose( IOPolledFileIOVars * vars )
808{
809 if (vars->pollers)
810 {
811 IOHibernatePollerClose(vars, kIOPolledPostflightState);
812 vars->pollers->release();
813 }
814
3a60a9f5
A		 815 bzero(vars, sizeof(IOPolledFileIOVars));
816
817 return (kIOReturnSuccess);
818}
819
820static IOReturn
821IOPolledFileSeek(IOPolledFileIOVars * vars, uint64_t position)
822{
823 IOPolledFileExtent * extentMap;
824
825 extentMap = vars->extentMap;
826
827 vars->position = position;
828
829 while (position >= extentMap->length)
830 {
831 position -= extentMap->length;
832 extentMap++;
833 }
834
835 vars->currentExtent = extentMap;
836 vars->extentRemaining = extentMap->length - position;
837 vars->extentPosition = vars->position - position;
838
839 if (vars->bufferSize <= vars->extentRemaining)
840 vars->bufferLimit = vars->bufferSize;
841 else
842 vars->bufferLimit = vars->extentRemaining;
843
844 return (kIOReturnSuccess);
845}
846
847static IOReturn
848IOPolledFileWrite(IOPolledFileIOVars * vars,
849 const uint8_t * bytes, IOByteCount size,
850 hibernate_cryptvars_t * cryptvars)
851{
852 IOReturn err = kIOReturnSuccess;
853 IOByteCount copy;
854 bool flush = false;
855
856 do
857 {
858 if (!bytes && !size)
859 {
860 // seek to end of block & flush
861 size = vars->position & (vars->blockSize - 1);
862 if (size)
863 size = vars->blockSize - size;
864 flush = true;
865 // use some garbage for the fill
866 bytes = vars->buffer + vars->bufferOffset;
867 }
868
869 copy = vars->bufferLimit - vars->bufferOffset;
870 if (copy > size)
871 copy = size;
872 else
873 flush = true;
874
875 if (bytes)
876 {
877 bcopy(bytes, vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
878 bytes += copy;
879 }
880 else
881 bzero(vars->buffer + vars->bufferHalf + vars->bufferOffset, copy);
882
883 size -= copy;
884 vars->bufferOffset += copy;
885 vars->position += copy;
886
887 if (flush && vars->bufferOffset)
888 {
889 uint64_t offset = (vars->position - vars->bufferOffset
890 - vars->extentPosition + vars->currentExtent->start);
891 uint32_t length = (vars->bufferOffset);
892
2d21ac55 893#if CRYPTO
0b4c1975
A		 894 if (cryptvars && vars->encryptStart
895 && (vars->position > vars->encryptStart)
896 && ((vars->position - length) < vars->encryptEnd))
3a60a9f5 897 {
6d2010ae
A		 898 AbsoluteTime startTime, endTime;
899
7ddcb079 900 uint64_t encryptLen, encryptStart;
3a60a9f5
A		 901 encryptLen = vars->position - vars->encryptStart;
902 if (encryptLen > length)
903 encryptLen = length;
904 encryptStart = length - encryptLen;
0b4c1975
A		 905 if (vars->position > vars->encryptEnd)
906 encryptLen -= (vars->position - vars->encryptEnd);
907
6d2010ae
A		 908 clock_get_uptime(&startTime);
909
3a60a9f5
A		 910 // encrypt the buffer
911 aes_encrypt_cbc(vars->buffer + vars->bufferHalf + encryptStart,
912 &cryptvars->aes_iv[0],
913 encryptLen / AES_BLOCK_SIZE,
914 vars->buffer + vars->bufferHalf + encryptStart,
915 &cryptvars->ctx.encrypt);
6d2010ae
A		 916
917 clock_get_uptime(&endTime);
918 ADD_ABSOLUTETIME(&vars->cryptTime, &endTime);
919 SUB_ABSOLUTETIME(&vars->cryptTime, &startTime);
920 vars->cryptBytes += encryptLen;
921
3a60a9f5
A		 922 // save initial vector for following encrypts
923 bcopy(vars->buffer + vars->bufferHalf + encryptStart + encryptLen - AES_BLOCK_SIZE,
924 &cryptvars->aes_iv[0],
925 AES_BLOCK_SIZE);
926 }
2d21ac55 927#endif /* CRYPTO */
3a60a9f5
A		 928
929 if (vars->io)
930 {
2d21ac55 931 err = IOHibernatePollerIODone(vars, true);
3a60a9f5
A		 932 if (kIOReturnSuccess != err)
933 break;
934 }
935
936if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
937//if (length != vars->bufferSize) HIBLOG("short write of %qx ends@ %qx\n", length, offset + length);
938
939 err = IOHibernatePollerIO(vars, kIOPolledWrite, vars->bufferHalf, offset, length);
940 if (kIOReturnSuccess != err)
941 break;
942 vars->io = true;
943
944 vars->extentRemaining -= vars->bufferOffset;
945 if (!vars->extentRemaining)
946 {
947 vars->currentExtent++;
948 vars->extentRemaining = vars->currentExtent->length;
949 vars->extentPosition = vars->position;
950 if (!vars->extentRemaining)
951 {
952 err = kIOReturnOverrun;
953 break;
954 }
955 }
956
957 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
958 vars->bufferOffset = 0;
959 if (vars->bufferSize <= vars->extentRemaining)
960 vars->bufferLimit = vars->bufferSize;
961 else
962 vars->bufferLimit = vars->extentRemaining;
963
964 flush = false;
965 }
966 }
967 while (size);
968
969 return (err);
970}
971
972static IOReturn
973IOPolledFileRead(IOPolledFileIOVars * vars,
974 uint8_t * bytes, IOByteCount size,
975 hibernate_cryptvars_t * cryptvars)
976{
977 IOReturn err = kIOReturnSuccess;
978 IOByteCount copy;
979
980// bytesWritten += size;
981
982 do
983 {
984 copy = vars->bufferLimit - vars->bufferOffset;
985 if (copy > size)
986 copy = size;
987
988 if (bytes)
989 {
990 bcopy(vars->buffer + vars->bufferHalf + vars->bufferOffset, bytes, copy);
991 bytes += copy;
992 }
993 size -= copy;
994 vars->bufferOffset += copy;
995// vars->position += copy;
996
6d2010ae 997 if ((vars->bufferOffset == vars->bufferLimit) && (vars->position < vars->readEnd))
3a60a9f5
A		 998 {
999 if (vars->io)
1000 {
2d21ac55 1001 err = IOHibernatePollerIODone(vars, false);
3a60a9f5
A		 1002 if (kIOReturnSuccess != err)
1003 break;
1004 }
1005 else
1006 cryptvars = 0;
1007
1008if (vars->position & (vars->blockSize - 1)) HIBLOG("misaligned file pos %qx\n", vars->position);
1009
6d2010ae 1010 vars->position += vars->lastRead;
3a60a9f5 1011 vars->extentRemaining -= vars->lastRead;
6d2010ae 1012 vars->bufferLimit = vars->lastRead;
3a60a9f5
A		 1013
1014 if (!vars->extentRemaining)
1015 {
1016 vars->currentExtent++;
1017 vars->extentRemaining = vars->currentExtent->length;
1018 vars->extentPosition = vars->position;
1019 if (!vars->extentRemaining)
1020 {
1021 err = kIOReturnOverrun;
1022 break;
1023 }
1024 }
1025
36401178
A		 1026 uint64_t length;
1027 uint64_t lastReadLength = vars->lastRead;
3a60a9f5
A		 1028 uint64_t offset = (vars->position
1029 - vars->extentPosition + vars->currentExtent->start);
36401178
A		 1030 if (vars->extentRemaining <= vars->bufferSize)
1031 length = vars->extentRemaining;
1032 else
1033 length = vars->bufferSize;
6d2010ae
A		 1034 if ((length + vars->position) > vars->readEnd)
1035 length = vars->readEnd - vars->position;
3a60a9f5 1036
6d2010ae
A		 1037 vars->lastRead = length;
1038 if (length)
1039 {
3a60a9f5 1040//if (length != vars->bufferSize) HIBLOG("short read of %qx ends@ %qx\n", length, offset + length);
6d2010ae
A		 1041 err = IOHibernatePollerIO(vars, kIOPolledRead, vars->bufferHalf, offset, length);
1042 if (kIOReturnSuccess != err)
1043 break;
1044 vars->io = true;
1045 }
3a60a9f5
A		 1046
1047 vars->bufferHalf = vars->bufferHalf ? 0 : vars->bufferSize;
1048 vars->bufferOffset = 0;
1049
2d21ac55 1050#if CRYPTO
3a60a9f5
A		 1051 if (cryptvars)
1052 {
1053 uint8_t thisVector[AES_BLOCK_SIZE];
6d2010ae
A		 1054 AbsoluteTime startTime, endTime;
1055
3a60a9f5
A		 1056 // save initial vector for following decrypts
1057 bcopy(&cryptvars->aes_iv[0], &thisVector[0], AES_BLOCK_SIZE);
36401178 1058 bcopy(vars->buffer + vars->bufferHalf + lastReadLength - AES_BLOCK_SIZE,
3a60a9f5 1059 &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
6d2010ae 1060
3a60a9f5 1061 // decrypt the buffer
6d2010ae
A		 1062 clock_get_uptime(&startTime);
1063
3a60a9f5
A		 1064 aes_decrypt_cbc(vars->buffer + vars->bufferHalf,
1065 &thisVector[0],
36401178 1066 lastReadLength / AES_BLOCK_SIZE,
3a60a9f5
A		 1067 vars->buffer + vars->bufferHalf,
1068 &cryptvars->ctx.decrypt);
6d2010ae
A		 1069
1070 clock_get_uptime(&endTime);
1071 ADD_ABSOLUTETIME(&vars->cryptTime, &endTime);
1072 SUB_ABSOLUTETIME(&vars->cryptTime, &startTime);
1073 vars->cryptBytes += lastReadLength;
3a60a9f5 1074 }
b0d623f7 1075#endif /* CRYPTO */
3a60a9f5
A		 1076 }
1077 }
1078 while (size);
1079
1080 return (err);
1081}
1082
1083/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1084
1085IOReturn
1086IOHibernateSystemSleep(void)
1087{
1088 IOReturn err;
1089 OSData * data;
1090 OSObject * obj;
1091 OSString * str;
0b4c1975 1092 bool dsSSD;
3a60a9f5
A		 1093
1094 IOHibernateVars * vars = &gIOHibernateVars;
1095
1096 if (vars->fileVars && vars->fileVars->fileRef)
1097 // already on the way down
1098 return (kIOReturnSuccess);
1099
1100 gIOHibernateState = kIOHibernateStateInactive;
1101
b0d623f7
A		 1102 gIOHibernateDebugFlags = 0;
1103 if (kIOLogHibernate & gIOKitDebug)
1104 gIOHibernateDebugFlags |= kIOHibernateDebugRestoreLogs;
1105
0b4c1975
A		 1106 if (IOService::getPMRootDomain()->getHibernateSettings(
1107 &gIOHibernateMode, &gIOHibernateFreeRatio, &gIOHibernateFreeTime))
6d2010ae 1108 {
3a60a9f5
A		 1109 if (kIOHibernateModeSleep & gIOHibernateMode)
1110 // default to discard clean for safe sleep
1111 gIOHibernateMode ^= (kIOHibernateModeDiscardCleanInactive
1112 | kIOHibernateModeDiscardCleanActive);
6d2010ae 1113 }
3a60a9f5 1114
3a60a9f5
A		 1115 if ((obj = IOService::getPMRootDomain()->copyProperty(kIOHibernateFileKey)))
1116 {
1117 if ((str = OSDynamicCast(OSString, obj)))
2d21ac55
A		 1118 strlcpy(gIOHibernateFilename, str->getCStringNoCopy(),
1119 sizeof(gIOHibernateFilename));
3a60a9f5
A		 1120 obj->release();
1121 }
1122
1123 if (!gIOHibernateMode || !gIOHibernateFilename[0])
1124 return (kIOReturnUnsupported);
1125
1126 HIBLOG("hibernate image path: %s\n", gIOHibernateFilename);
1127
2d21ac55 1128
3a60a9f5
A		 1129 do
1130 {
0c530ab8
A		 1131 vars->srcBuffer = IOBufferMemoryDescriptor::withOptions(kIODirectionOutIn,
1132 4 * page_size, page_size);
1133 vars->ioBuffer = IOBufferMemoryDescriptor::withOptions(kIODirectionOutIn,
1134 2 * kDefaultIOSize, page_size);
3a60a9f5 1135
6d2010ae
A		 1136 vars->handoffBuffer = IOBufferMemoryDescriptor::withOptions(kIODirectionOutIn,
1137 ptoa_64(gIOHibernateHandoffPageCount), page_size);
1138
1139 if (!vars->srcBuffer || !vars->ioBuffer || !vars->handoffBuffer)
3a60a9f5
A		 1140 {
1141 err = kIOReturnNoMemory;
1142 break;
1143 }
1144
6d2010ae
A		 1145 // open & invalidate the image file
1146 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderInvalidSignature;
3a60a9f5 1147 err = IOPolledFileOpen(gIOHibernateFilename, vars->ioBuffer,
6d2010ae
A		 1148 &vars->fileVars, &vars->fileExtents, &data,
1149 &vars->volumeCryptKey[0]);
3a60a9f5
A		 1150 if (KERN_SUCCESS != err)
1151 {
1152 HIBLOG("IOPolledFileOpen(%x)\n", err);
1153 break;
1154 }
3a60a9f5
A		 1155
1156 bzero(gIOHibernateCurrentHeader, sizeof(IOHibernateImageHeader));
b0d623f7 1157 gIOHibernateCurrentHeader->debugFlags = gIOHibernateDebugFlags;
6d2010ae 1158 dsSSD = ((0 != (kIOHibernateOptionSSD & vars->fileVars->flags))
0b4c1975 1159 && (kOSBooleanTrue == IOService::getPMRootDomain()->getProperty(kIOPMDeepSleepEnabledKey)));
0b4c1975
A		 1160 if (dsSSD)
1161 {
1162 gIOHibernateCurrentHeader->options |=
1163 kIOHibernateOptionSSD
1164 | kIOHibernateOptionColor;
6d2010ae
A		 1165
1166#if defined(__i386__) || defined(__x86_64__)
1167 if (!uuid_is_null(vars->volumeCryptKey) &&
1168 (kOSBooleanTrue != IOService::getPMRootDomain()->getProperty(kIOPMDestroyFVKeyOnStandbyKey)))
1169 {
1170 uintptr_t smcVars[2];
1171 smcVars[0] = sizeof(vars->volumeCryptKey);
1172 smcVars[1] = (uintptr_t)(void *) &vars->volumeCryptKey[0];
1173
1174 IOService::getPMRootDomain()->setProperty(kIOHibernateSMCVariablesKey, smcVars, sizeof(smcVars));
1175 bzero(smcVars, sizeof(smcVars));
1176 }
1177#endif
0b4c1975
A		 1178 }
1179 else
1180 {
1181 gIOHibernateCurrentHeader->options |= kIOHibernateOptionProgress;
1182 }
1183
3a60a9f5 1184 boolean_t encryptedswap;
0b4c1975
A		 1185 AbsoluteTime startTime, endTime;
1186 uint64_t nsec;
1187
1188 clock_get_uptime(&startTime);
3a60a9f5
A		 1189 err = hibernate_setup(gIOHibernateCurrentHeader,
1190 gIOHibernateFreeRatio, gIOHibernateFreeTime,
0b4c1975 1191 dsSSD,
6d2010ae 1192 &vars->page_list, &vars->page_list_wired, &vars->page_list_pal, &encryptedswap);
0b4c1975
A		 1193 clock_get_uptime(&endTime);
1194 SUB_ABSOLUTETIME(&endTime, &startTime);
1195 absolutetime_to_nanoseconds(endTime, &nsec);
1196 HIBLOG("hibernate_setup(%d) took %qd ms\n", err, nsec / 1000000ULL);
1197
3a60a9f5 1198 if (KERN_SUCCESS != err)
3a60a9f5 1199 break;
3a60a9f5 1200
6d2010ae 1201 if (encryptedswap || !uuid_is_null(vars->volumeCryptKey))
3a60a9f5
A		 1202 gIOHibernateMode ^= kIOHibernateModeEncrypt;
1203
0b4c1975
A		 1204 if (kIOHibernateOptionProgress & gIOHibernateCurrentHeader->options)
1205 {
1206 vars->videoAllocSize = kVideoMapSize;
1207 if (KERN_SUCCESS != kmem_alloc_pageable(kernel_map, &vars->videoMapping, vars->videoAllocSize))
1208 vars->videoMapping = 0;
1209 }
3a60a9f5
A		 1210
1211 // generate crypt keys
1212 for (uint32_t i = 0; i < sizeof(vars->wiredCryptKey); i++)
1213 vars->wiredCryptKey[i] = random();
1214 for (uint32_t i = 0; i < sizeof(vars->cryptKey); i++)
1215 vars->cryptKey[i] = random();
1216
1217 // set nvram
1218
1219 IORegistryEntry * regEntry;
1220 if (!gIOOptionsEntry)
1221 {
1222 regEntry = IORegistryEntry::fromPath("/options", gIODTPlane);
1223 gIOOptionsEntry = OSDynamicCast(IODTNVRAM, regEntry);
1224 if (regEntry && !gIOOptionsEntry)
1225 regEntry->release();
1226 }
1227 if (!gIOChosenEntry)
1228 gIOChosenEntry = IORegistryEntry::fromPath("/chosen", gIODTPlane);
1229
1230 if (gIOOptionsEntry)
1231 {
1232 const OSSymbol * sym;
3a60a9f5
A		 1233
1234 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKey);
1235 if (sym)
1236 {
1237 gIOOptionsEntry->setProperty(sym, data);
1238 sym->release();
1239 }
1240 data->release();
1241
b0d623f7 1242#if defined(__i386__) || defined(__x86_64__)
0c530ab8
A		 1243 struct AppleRTCHibernateVars
1244 {
1245 uint8_t signature[4];
1246 uint32_t revision;
1247 uint8_t booterSignature[20];
1248 uint8_t wiredCryptKey[16];
1249 };
1250 AppleRTCHibernateVars rtcVars;
1251
1252 rtcVars.signature[0] = 'A';
1253 rtcVars.signature[1] = 'A';
1254 rtcVars.signature[2] = 'P';
1255 rtcVars.signature[3] = 'L';
1256 rtcVars.revision = 1;
1257 bcopy(&vars->wiredCryptKey[0], &rtcVars.wiredCryptKey[0], sizeof(rtcVars.wiredCryptKey));
1258 if (gIOHibernateBootSignature[0])
1259 {
1260 char c;
1261 uint8_t value = 0;
1262 for (uint32_t i = 0;
1263 (c = gIOHibernateBootSignature[i]) && (i < (sizeof(rtcVars.booterSignature) << 1));
1264 i++)
1265 {
1266 if (c >= 'a')
1267 c -= 'a' - 10;
1268 else if (c >= 'A')
1269 c -= 'A' - 10;
1270 else if (c >= '0')
1271 c -= '0';
1272 else
1273 continue;
1274 value = (value << 4) | c;
1275 if (i & 1)
1276 rtcVars.booterSignature[i >> 1] = value;
1277 }
1278 }
1279 data = OSData::withBytes(&rtcVars, sizeof(rtcVars));
1280 if (data)
2d21ac55 1281 {
060df5ea
A		 1282 if (!gIOHibernateRTCVariablesKey)
1283 gIOHibernateRTCVariablesKey = OSSymbol::withCStringNoCopy(kIOHibernateRTCVariablesKey);
1284 if (gIOHibernateRTCVariablesKey)
1285 IOService::getPMRootDomain()->setProperty(gIOHibernateRTCVariablesKey, data);
1286
1287 if( gIOOptionsEntry )
1288 {
1289 if( gIOHibernateMode & kIOHibernateModeSwitch )
1290 {
1291 const OSSymbol *sym;
1292 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootSwitchVarsKey);
1293 if( sym )
2d21ac55 1294 {
060df5ea
A		 1295 gIOOptionsEntry->setProperty(sym, data); /* intentional insecure backup of rtc boot vars */
1296 sym->release();
2d21ac55 1297 }
060df5ea
A		 1298 }
1299 }
2d21ac55 1300
060df5ea 1301 data->release();
0c530ab8
A		 1302 }
1303 if (gIOChosenEntry)
1304 {
1305 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty(kIOHibernateMachineSignatureKey));
1306 if (data)
1307 gIOHibernateCurrentHeader->machineSignature = *((UInt32 *)data->getBytesNoCopy());
060df5ea
A		 1308 {
1309 // set BootNext
1310
1311 if (!gIOHibernateBoot0082Data)
1312 {
1313 data = OSDynamicCast(OSData, gIOChosenEntry->getProperty("boot-device-path"));
1314 if (data)
1315 {
1316 // AppleNVRAM_EFI_LOAD_OPTION
1317 struct {
1318 uint32_t Attributes;
1319 uint16_t FilePathLength;
1320 uint16_t Desc;
1321 } loadOptionHeader;
1322 loadOptionHeader.Attributes = 1;
1323 loadOptionHeader.FilePathLength = data->getLength();
1324 loadOptionHeader.Desc = 0;
1325 gIOHibernateBoot0082Data = OSData::withCapacity(sizeof(loadOptionHeader) + loadOptionHeader.FilePathLength);
1326 if (gIOHibernateBoot0082Data)
1327 {
1328 gIOHibernateBoot0082Data->appendBytes(&loadOptionHeader, sizeof(loadOptionHeader));
1329 gIOHibernateBoot0082Data->appendBytes(data);
1330 }
1331 }
1332 }
1333 if (!gIOHibernateBoot0082Key)
1334 gIOHibernateBoot0082Key = OSSymbol::withCString("8BE4DF61-93CA-11D2-AA0D-00E098032B8C:Boot0082");
1335 if (!gIOHibernateBootNextKey)
1336 gIOHibernateBootNextKey = OSSymbol::withCString("8BE4DF61-93CA-11D2-AA0D-00E098032B8C:BootNext");
1337 if (!gIOHibernateBootNextData)
1338 {
1339 uint16_t bits = 0x0082;
1340 gIOHibernateBootNextData = OSData::withBytes(&bits, sizeof(bits));
1341 }
1342 if (gIOHibernateBoot0082Key && gIOHibernateBoot0082Data && gIOHibernateBootNextKey && gIOHibernateBootNextData)
1343 {
1344 gIOHibernateBootNextSave = gIOOptionsEntry->copyProperty(gIOHibernateBootNextKey);
1345 gIOOptionsEntry->setProperty(gIOHibernateBoot0082Key, gIOHibernateBoot0082Data);
1346 gIOOptionsEntry->setProperty(gIOHibernateBootNextKey, gIOHibernateBootNextData);
1347 }
1348 }
0c530ab8 1349 }
b0d623f7 1350#else /* !i386 && !x86_64 */
3a60a9f5
A		 1351 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1352 {
1353 data = OSData::withBytes(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1354 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootImageKeyKey);
1355 if (sym && data)
1356 gIOOptionsEntry->setProperty(sym, data);
1357 if (sym)
1358 sym->release();
1359 if (data)
1360 data->release();
2d21ac55 1361 if (false && gIOHibernateBootSignature[0])
3a60a9f5
A		 1362 {
1363 data = OSData::withCapacity(16);
1364 sym = OSSymbol::withCStringNoCopy(kIOHibernateBootSignatureKey);
1365 if (sym && data)
1366 {
1367 char c;
0c530ab8 1368 uint8_t value = 0;
3a60a9f5
A		 1369 for (uint32_t i = 0; (c = gIOHibernateBootSignature[i]); i++)
1370 {
1371 if (c >= 'a')
1372 c -= 'a' - 10;
1373 else if (c >= 'A')
1374 c -= 'A' - 10;
1375 else if (c >= '0')
1376 c -= '0';
1377 else
1378 continue;
1379 value = (value << 4) | c;
1380 if (i & 1)
1381 data->appendBytes(&value, sizeof(value));
1382 }
1383 gIOOptionsEntry->setProperty(sym, data);
1384 }
1385 if (sym)
1386 sym->release();
1387 if (data)
1388 data->release();
1389 }
1390 }
3a60a9f5
A		 1391 if (!vars->haveFastBoot)
1392 {
1393 // set boot volume to zero
1394 IODTPlatformExpert * platform = OSDynamicCast(IODTPlatformExpert, IOService::getPlatform());
1395 if (platform && (kIOReturnSuccess == platform->readXPRAM(kXPRamAudioVolume,
1396 &vars->saveBootAudioVolume, sizeof(vars->saveBootAudioVolume))))
1397 {
1398 uint8_t newVolume;
1399 newVolume = vars->saveBootAudioVolume & 0xf8;
1400 platform->writeXPRAM(kXPRamAudioVolume,
1401 &newVolume, sizeof(newVolume));
1402 }
1403 }
b0d623f7 1404#endif /* !i386 && !x86_64 */
3a60a9f5
A		 1405 }
1406 // --
1407
1408 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderSignature;
1409 gIOHibernateState = kIOHibernateStateHibernating;
1410 }
1411 while (false);
1412
1413 return (err);
1414}
1415
1416/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1417
2d21ac55
A		 1418DECLARE_IOHIBERNATEPROGRESSALPHA
1419
1420static void
1421ProgressInit(hibernate_graphics_t * display, uint8_t * screen, uint8_t * saveunder, uint32_t savelen)
1422{
1423 uint32_t rowBytes, pixelShift;
1424 uint32_t x, y;
1425 int32_t blob;
1426 uint32_t alpha, in, color, result;
1427 uint8_t * out;
1428 uint32_t saveindex[kIOHibernateProgressCount] = { 0 };
1429
1430 rowBytes = display->rowBytes;
1431 pixelShift = display->depth >> 4;
1432 if (pixelShift < 1) return;
1433
1434 screen += ((display->width
1435 - kIOHibernateProgressCount * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << (pixelShift - 1))
1436 + (display->height - kIOHibernateProgressOriginY - kIOHibernateProgressHeight) * rowBytes;
1437
1438 for (y = 0; y < kIOHibernateProgressHeight; y++)
1439 {
1440 out = screen + y * rowBytes;
1441 for (blob = 0; blob < kIOHibernateProgressCount; blob++)
1442 {
1443 color = blob ? kIOHibernateProgressDarkGray : kIOHibernateProgressMidGray;
1444 for (x = 0; x < kIOHibernateProgressWidth; x++)
1445 {
1446 alpha = gIOHibernateProgressAlpha[y][x];
1447 result = color;
1448 if (alpha)
1449 {
1450 if (0xff != alpha)
1451 {
1452 if (1 == pixelShift)
1453 {
1454 in = *((uint16_t *)out) & 0x1f; // 16
1455 in = (in << 3) | (in >> 2);
1456 }
1457 else
1458 in = *((uint32_t *)out) & 0xff; // 32
1459 saveunder[blob * kIOHibernateProgressSaveUnderSize + saveindex[blob]++] = in;
1460 result = ((255 - alpha) * in + alpha * result + 0xff) >> 8;
1461 }
1462 if (1 == pixelShift)
1463 {
1464 result >>= 3;
1465 *((uint16_t *)out) = (result << 10) | (result << 5) | result; // 16
1466 }
1467 else
1468 *((uint32_t *)out) = (result << 16) | (result << 8) | result; // 32
1469 }
1470 out += (1 << pixelShift);
1471 }
1472 out += (kIOHibernateProgressSpacing << pixelShift);
1473 }
1474 }
1475}
1476
1477
1478static void
1479ProgressUpdate(hibernate_graphics_t * display, uint8_t * screen, int32_t firstBlob, int32_t select)
1480{
1481 uint32_t rowBytes, pixelShift;
1482 uint32_t x, y;
1483 int32_t blob, lastBlob;
1484 uint32_t alpha, in, color, result;
1485 uint8_t * out;
1486 uint32_t saveindex[kIOHibernateProgressCount] = { 0 };
1487
1488 pixelShift = display->depth >> 4;
1489 if (pixelShift < 1)
1490 return;
1491
1492 rowBytes = display->rowBytes;
1493
1494 screen += ((display->width
1495 - kIOHibernateProgressCount * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << (pixelShift - 1))
1496 + (display->height - kIOHibernateProgressOriginY - kIOHibernateProgressHeight) * rowBytes;
1497
1498 lastBlob = (select < kIOHibernateProgressCount) ? select : (kIOHibernateProgressCount - 1);
1499
1500 screen += (firstBlob * (kIOHibernateProgressWidth + kIOHibernateProgressSpacing)) << pixelShift;
1501
1502 for (y = 0; y < kIOHibernateProgressHeight; y++)
1503 {
1504 out = screen + y * rowBytes;
1505 for (blob = firstBlob; blob <= lastBlob; blob++)
1506 {
1507 color = (blob < select) ? kIOHibernateProgressLightGray : kIOHibernateProgressMidGray;
1508 for (x = 0; x < kIOHibernateProgressWidth; x++)
1509 {
1510 alpha = gIOHibernateProgressAlpha[y][x];
1511 result = color;
1512 if (alpha)
1513 {
1514 if (0xff != alpha)
1515 {
1516 in = display->progressSaveUnder[blob][saveindex[blob]++];
1517 result = ((255 - alpha) * in + alpha * result + 0xff) / 255;
1518 }
1519 if (1 == pixelShift)
1520 {
1521 result >>= 3;
1522 *((uint16_t *)out) = (result << 10) | (result << 5) | result; // 16
1523 }
1524 else
1525 *((uint32_t *)out) = (result << 16) | (result << 8) | result; // 32
1526 }
1527 out += (1 << pixelShift);
1528 }
1529 out += (kIOHibernateProgressSpacing << pixelShift);
1530 }
1531 }
1532}
1533
1534/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1535
3a60a9f5
A		 1536IOReturn
1537IOHibernateSystemHasSlept(void)
1538{
1539 IOHibernateVars * vars = &gIOHibernateVars;
2d21ac55
A		 1540 OSObject * obj;
1541 OSData * data;
1542
1543 obj = IOService::getPMRootDomain()->copyProperty(kIOHibernatePreviewBufferKey);
1544 vars->previewBuffer = OSDynamicCast(IOMemoryDescriptor, obj);
1545 if (obj && !vars->previewBuffer)
1546 obj->release();
3a60a9f5 1547
2d21ac55
A		 1548 vars->consoleMapping = NULL;
1549 if (vars->previewBuffer && (kIOReturnSuccess != vars->previewBuffer->prepare()))
3a60a9f5 1550 {
2d21ac55
A		 1551 vars->previewBuffer->release();
1552 vars->previewBuffer = 0;
1553 }
3a60a9f5 1554
0b4c1975
A		 1555 if ((kIOHibernateOptionProgress & gIOHibernateCurrentHeader->options)
1556 && vars->previewBuffer
1557 && (data = OSDynamicCast(OSData,
2d21ac55
A		 1558 IOService::getPMRootDomain()->getProperty(kIOHibernatePreviewActiveKey))))
1559 {
1560 UInt32 flags = *((UInt32 *)data->getBytesNoCopy());
b0d623f7 1561 HIBPRINT("kIOHibernatePreviewActiveKey %08lx\n", (long)flags);
2d21ac55
A		 1562
1563 IOService::getPMRootDomain()->removeProperty(kIOHibernatePreviewActiveKey);
1564
1565 if (kIOHibernatePreviewUpdates & flags)
1566 {
1567 PE_Video consoleInfo;
1568 hibernate_graphics_t * graphicsInfo = gIOHibernateGraphicsInfo;
1569
1570 IOService::getPlatform()->getConsoleInfo(&consoleInfo);
1571
1572 graphicsInfo->width = consoleInfo.v_width;
1573 graphicsInfo->height = consoleInfo.v_height;
1574 graphicsInfo->rowBytes = consoleInfo.v_rowBytes;
1575 graphicsInfo->depth = consoleInfo.v_depth;
1576 vars->consoleMapping = (uint8_t *) consoleInfo.v_baseAddr;
1577
1578 HIBPRINT("video %p %d %d %d\n",
0b4c1975
A		 1579 vars->consoleMapping, graphicsInfo->depth,
1580 graphicsInfo->width, graphicsInfo->height);
2d21ac55
A		 1581 if (vars->consoleMapping)
1582 ProgressInit(graphicsInfo, vars->consoleMapping,
1583 &graphicsInfo->progressSaveUnder[0][0], sizeof(graphicsInfo->progressSaveUnder));
1584 }
3a60a9f5 1585 }
2d21ac55 1586
3a60a9f5
A		 1587 if (gIOOptionsEntry)
1588 gIOOptionsEntry->sync();
1589
1590 return (kIOReturnSuccess);
1591}
1592
1593/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1594
6d2010ae
A		 1595static DeviceTreeNode *
1596MergeDeviceTree(DeviceTreeNode * entry, IORegistryEntry * regEntry)
1597{
1598 DeviceTreeNodeProperty * prop;
1599 DeviceTreeNode * child;
1600 IORegistryEntry * childRegEntry;
1601 const char * nameProp;
1602 unsigned int propLen, idx;
1603
1604 prop = (DeviceTreeNodeProperty *) (entry + 1);
1605 for (idx = 0; idx < entry->nProperties; idx++)
1606 {
1607 if (regEntry && (0 != strcmp("name", prop->name)))
1608 {
1609 regEntry->setProperty((const char *) prop->name, (void *) (prop + 1), prop->length);
1610// HIBPRINT("%s: %s, %d\n", regEntry->getName(), prop->name, prop->length);
1611 }
1612 prop = (DeviceTreeNodeProperty *) (((uintptr_t)(prop + 1)) + ((prop->length + 3) & ~3));
1613 }
1614
1615 child = (DeviceTreeNode *) prop;
1616 for (idx = 0; idx < entry->nChildren; idx++)
1617 {
1618 if (kSuccess != DTGetProperty(child, "name", (void **) &nameProp, &propLen))
1619 panic("no name");
1620 childRegEntry = regEntry ? regEntry->childFromPath(nameProp, gIODTPlane) : NULL;
1621// HIBPRINT("%s == %p\n", nameProp, childRegEntry);
1622 child = MergeDeviceTree(child, childRegEntry);
1623 }
1624 return (child);
1625}
1626
3a60a9f5
A		 1627IOReturn
1628IOHibernateSystemWake(void)
1629{
1630 IOHibernateVars * vars = &gIOHibernateVars;
1631
1632 hibernate_teardown(vars->page_list, vars->page_list_wired);
1633
1634 if (vars->videoMapping)
1635 {
1636 if (vars->videoMapSize)
1637 // remove mappings
1638 IOUnmapPages(kernel_map, vars->videoMapping, vars->videoMapSize);
1639 if (vars->videoAllocSize)
1640 // dealloc range
b0d623f7 1641 kmem_free(kernel_map, trunc_page(vars->videoMapping), vars->videoAllocSize);
3a60a9f5
A		 1642 }
1643
1644 if (vars->previewBuffer)
1645 {
1646 vars->previewBuffer->release();
1647 vars->previewBuffer = 0;
1648 }
1649
0b4c1975
A		 1650 if (kIOHibernateStateWakingFromHibernate == gIOHibernateState)
1651 {
1652 IOService::getPMRootDomain()->setProperty(kIOHibernateOptionsKey,
1653 gIOHibernateCurrentHeader->options, 32);
1654 }
1655 else
1656 {
1657 IOService::getPMRootDomain()->removeProperty(kIOHibernateOptionsKey);
1658 }
1659
1660 if ((kIOHibernateStateWakingFromHibernate == gIOHibernateState)
1661 && (kIOHibernateGfxStatusUnknown != gIOHibernateGraphicsInfo->gfxStatus))
1662 {
1663 IOService::getPMRootDomain()->setProperty(kIOHibernateGfxStatusKey,
1664 &gIOHibernateGraphicsInfo->gfxStatus,
1665 sizeof(gIOHibernateGraphicsInfo->gfxStatus));
1666 }
1667 else
1668 {
1669 IOService::getPMRootDomain()->removeProperty(kIOHibernateGfxStatusKey);
1670 }
1671
1672
3a60a9f5
A		 1673 if (vars->fileVars)
1674 {
1675 IOPolledFileClose(vars->fileVars);
1676 }
1677
1678 // invalidate nvram properties - (gIOOptionsEntry != 0) => nvram was touched
1679
b0d623f7 1680#if defined(__i386__) || defined(__x86_64__)
060df5ea 1681 IOService::getPMRootDomain()->removeProperty(gIOHibernateRTCVariablesKey);
6d2010ae 1682 IOService::getPMRootDomain()->removeProperty(kIOHibernateSMCVariablesKey);
2d21ac55
A		 1683
1684 /*
1685 * Hibernate variable is written to NVRAM on platforms in which RtcRam
1686 * is not backed by coin cell. Remove Hibernate data from NVRAM.
1687 */
1688 if (gIOOptionsEntry) {
2d21ac55 1689
060df5ea
A		 1690 if (gIOHibernateRTCVariablesKey) {
1691 if (gIOOptionsEntry->getProperty(gIOHibernateRTCVariablesKey)) {
1692 gIOOptionsEntry->removeProperty(gIOHibernateRTCVariablesKey);
1693 }
1694 }
1695
1696 if (gIOHibernateBootNextKey)
1697 {
1698 if (gIOHibernateBootNextSave)
1699 {
1700 gIOOptionsEntry->setProperty(gIOHibernateBootNextKey, gIOHibernateBootNextSave);
1701 gIOHibernateBootNextSave->release();
1702 gIOHibernateBootNextSave = NULL;
2d21ac55 1703 }
060df5ea
A		 1704 else
1705 gIOOptionsEntry->removeProperty(gIOHibernateBootNextKey);
1706 }
1707 gIOOptionsEntry->sync();
2d21ac55 1708 }
0c530ab8 1709#endif
3a60a9f5
A		 1710
1711 if (vars->srcBuffer)
1712 vars->srcBuffer->release();
1713 if (vars->ioBuffer)
1714 vars->ioBuffer->release();
6d2010ae 1715 bzero(&gIOHibernateHandoffPages[0], gIOHibernateHandoffPageCount * sizeof(gIOHibernateHandoffPages[0]));
7ddcb079 1716 if (vars->handoffBuffer && (kIOHibernateStateWakingFromHibernate == gIOHibernateState))
6d2010ae
A		 1717 {
1718 IOHibernateHandoff * handoff;
1719 bool done = false;
1720 for (handoff = (IOHibernateHandoff *) vars->handoffBuffer->getBytesNoCopy();
1721 !done;
1722 handoff = (IOHibernateHandoff *) &handoff->data[handoff->bytecount])
1723 {
7ddcb079 1724 HIBPRINT("handoff %p, %x, %x\n", handoff, handoff->type, handoff->bytecount);
6d2010ae
A		 1725 uint8_t * data = &handoff->data[0];
1726 switch (handoff->type)
1727 {
1728 case kIOHibernateHandoffTypeEnd:
1729 done = true;
1730 break;
1731
1732 case kIOHibernateHandoffTypeDeviceTree:
1733 MergeDeviceTree((DeviceTreeNode *) data, IOService::getServiceRoot());
1734 break;
1735
1736 case kIOHibernateHandoffTypeKeyStore:
1737#if defined(__i386__) || defined(__x86_64__)
1738 {
1739 IOBufferMemoryDescriptor *
1740 md = IOBufferMemoryDescriptor::withBytes(data, handoff->bytecount, kIODirectionOutIn);
1741 if (md)
1742 {
1743 IOSetKeyStoreData(md);
1744 }
1745 }
1746#endif
1747 break;
1748
1749 default:
1750 done = (kIOHibernateHandoffType != (handoff->type & 0xFFFF0000));
1751 break;
1752 }
1753 }
1754 vars->handoffBuffer->release();
1755 }
3a60a9f5
A		 1756 if (vars->fileExtents)
1757 vars->fileExtents->release();
1758
1759 bzero(vars, sizeof(*vars));
1760
1761// gIOHibernateState = kIOHibernateStateInactive; // leave it for post wake code to see
1762
1763 return (kIOReturnSuccess);
1764}
1765
1766IOReturn
1767IOHibernateSystemPostWake(void)
1768{
1769 if (gIOHibernateFileRef)
1770 {
6d2010ae 1771 // invalidate & close the image file
0c530ab8 1772 gIOHibernateCurrentHeader->signature = kIOHibernateHeaderInvalidSignature;
6d2010ae
A		 1773 kern_close_file_for_direct_io(gIOHibernateFileRef,
1774 0, (caddr_t) gIOHibernateCurrentHeader,
7ddcb079
A		 1775 sizeof(IOHibernateImageHeader),
1776 sizeof(IOHibernateImageHeader),
1777 gIOHibernateCurrentHeader->imageSize);
3a60a9f5
A		 1778 gIOHibernateFileRef = 0;
1779 }
1780 return (kIOReturnSuccess);
1781}
1782
1783/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1784
2d21ac55 1785SYSCTL_STRING(_kern, OID_AUTO, hibernatefile,
6d2010ae 1786 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN | CTLFLAG_LOCKED,
2d21ac55
A		 1787 gIOHibernateFilename, sizeof(gIOHibernateFilename), "");
1788SYSCTL_STRING(_kern, OID_AUTO, bootsignature,
6d2010ae 1789 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN | CTLFLAG_LOCKED,
2d21ac55
A		 1790 gIOHibernateBootSignature, sizeof(gIOHibernateBootSignature), "");
1791SYSCTL_UINT(_kern, OID_AUTO, hibernatemode,
6d2010ae 1792 CTLFLAG_RW | CTLFLAG_NOAUTO | CTLFLAG_KERN | CTLFLAG_LOCKED,
2d21ac55
A		 1793 &gIOHibernateMode, 0, "");
1794
3a60a9f5
A		 1795void
1796IOHibernateSystemInit(IOPMrootDomain * rootDomain)
1797{
1798 OSData * data = OSData::withBytesNoCopy(&gIOHibernateState, sizeof(gIOHibernateState));
1799 if (data)
1800 {
1801 rootDomain->setProperty(kIOHibernateStateKey, data);
1802 data->release();
1803 }
1804
2d21ac55 1805 if (PE_parse_boot_argn("hfile", gIOHibernateFilename, sizeof(gIOHibernateFilename)))
3a60a9f5
A		 1806 gIOHibernateMode = kIOHibernateModeOn;
1807 else
1808 gIOHibernateFilename[0] = 0;
1809
3a60a9f5 1810 sysctl_register_oid(&sysctl__kern_hibernatefile);
3a60a9f5 1811 sysctl_register_oid(&sysctl__kern_bootsignature);
3a60a9f5
A		 1812 sysctl_register_oid(&sysctl__kern_hibernatemode);
1813}
1814
2d21ac55 1815
3a60a9f5
A		 1816/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1817
1818static void
1819hibernate_setup_for_wake(void)
1820{
3a60a9f5
A		 1821}
1822
1823/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
1824
0c530ab8
A		 1825#define C_ASSERT(e) typedef char __C_ASSERT__[(e) ? 1 : -1]
1826
0b4c1975
A		 1827static bool
1828no_encrypt_page(vm_offset_t ppnum)
1829{
1830 if (pmap_is_noencrypt((ppnum_t)ppnum) == TRUE)
1831 {
1832 return true;
1833 }
1834 return false;
1835}
1836
1837uint32_t wired_pages_encrypted = 0;
1838uint32_t dirty_pages_encrypted = 0;
1839uint32_t wired_pages_clear = 0;
1840
6d2010ae
A		 1841static void
1842hibernate_pal_callback(void *vars_arg, vm_offset_t addr)
1843{
1844 IOHibernateVars *vars = (IOHibernateVars *)vars_arg;
1845 /* Make sure it's not in either of the save lists */
1846 hibernate_set_page_state(vars->page_list, vars->page_list_wired, atop_64(addr), 1, kIOHibernatePageStateFree);
1847
1848 /* Set it in the bitmap of pages owned by the PAL */
1849 hibernate_page_bitset(vars->page_list_pal, TRUE, atop_64(addr));
1850}
1851
1852static struct hibernate_cryptvars_t *local_cryptvars;
1853
1854extern "C" int
1855hibernate_pal_write(void *buffer, size_t size)
1856{
1857 IOHibernateVars * vars = &gIOHibernateVars;
1858
1859 IOReturn err = IOPolledFileWrite(vars->fileVars, (const uint8_t *)buffer, size, local_cryptvars);
1860 if (kIOReturnSuccess != err) {
1861 kprintf("epic hibernate fail! %d\n", err);
1862 return err;
1863 }
1864
1865 return 0;
1866}
1867
1868
2d21ac55 1869extern "C" uint32_t
3a60a9f5
A		 1870hibernate_write_image(void)
1871{
1872 IOHibernateImageHeader * header = gIOHibernateCurrentHeader;
1873 IOHibernateVars * vars = &gIOHibernateVars;
1874 IOPolledFileExtent * fileExtents;
1875
0c530ab8
A		 1876 C_ASSERT(sizeof(IOHibernateImageHeader) == 512);
1877
3a60a9f5
A		 1878 uint32_t pageCount, pagesDone;
1879 IOReturn err;
0b4c1975
A		 1880 vm_offset_t ppnum, page;
1881 IOItemCount count;
3a60a9f5
A		 1882 uint8_t * src;
1883 uint8_t * data;
1884 IOByteCount pageCompressedSize;
1885 uint64_t compressedSize, uncompressedSize;
1886 uint64_t image1Size = 0;
1887 uint32_t bitmap_size;
0b4c1975 1888 bool iterDone, pollerOpen, needEncrypt;
3a60a9f5
A		 1889 uint32_t restore1Sum, sum, sum1, sum2;
1890 uint32_t tag;
1891 uint32_t pageType;
1892 uint32_t pageAndCount[2];
6d2010ae
A		 1893 addr64_t phys64;
1894 IOByteCount segLen;
3a60a9f5
A		 1895
1896 AbsoluteTime startTime, endTime;
6d2010ae 1897 AbsoluteTime allTime, compTime;
0b4c1975 1898 uint64_t compBytes;
3a60a9f5
A		 1899 uint64_t nsec;
1900 uint32_t lastProgressStamp = 0;
1901 uint32_t progressStamp;
2d21ac55 1902 uint32_t blob, lastBlob = (uint32_t) -1L;
3a60a9f5
A		 1903
1904 hibernate_cryptvars_t _cryptvars;
1905 hibernate_cryptvars_t * cryptvars = 0;
1906
0b4c1975
A		 1907 wired_pages_encrypted = 0;
1908 dirty_pages_encrypted = 0;
1909 wired_pages_clear = 0;
1910
3a60a9f5
A		 1911 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
1912 return (false /* sleep */ );
1913
0b4c1975
A		 1914 if (kIOHibernateModeSleep & gIOHibernateMode)
1915 kdebug_enable = save_kdebug_enable;
1916
1917 KERNEL_DEBUG_CONSTANT(IOKDBG_CODE(DBG_HIBERNATE, 1) | DBG_FUNC_START, 0, 0, 0, 0, 0);
6d2010ae 1918 IOService::getPMRootDomain()->tracePoint(kIOPMTracePointHibernate);
0b4c1975 1919
3a60a9f5
A		 1920 restore1Sum = sum1 = sum2 = 0;
1921
6d2010ae
A		 1922 hibernate_pal_prepare();
1923
2d21ac55 1924#if CRYPTO
3a60a9f5
A		 1925 // encryption data. "iv" is the "initial vector".
1926 if (kIOHibernateModeEncrypt & gIOHibernateMode)
1927 {
1928 static const unsigned char first_iv[AES_BLOCK_SIZE]
1929 = { 0xa3, 0x63, 0x65, 0xa9, 0x0b, 0x71, 0x7b, 0x1c,
1930 0xdf, 0x9e, 0x5f, 0x32, 0xd7, 0x61, 0x63, 0xda };
1931
1932 cryptvars = &gIOHibernateCryptWakeContext;
1933 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
1934 aes_encrypt_key(vars->cryptKey,
1935 kIOHibernateAESKeySize,
1936 &cryptvars->ctx.encrypt);
1937 aes_decrypt_key(vars->cryptKey,
1938 kIOHibernateAESKeySize,
1939 &cryptvars->ctx.decrypt);
1940
1941 cryptvars = &_cryptvars;
1942 bzero(cryptvars, sizeof(hibernate_cryptvars_t));
6d2010ae
A		 1943 for (pageCount = 0; pageCount < sizeof(vars->wiredCryptKey); pageCount++)
1944 vars->wiredCryptKey[pageCount] ^= vars->volumeCryptKey[pageCount];
1945 bzero(&vars->volumeCryptKey[0], sizeof(vars->volumeCryptKey));
3a60a9f5
A		 1946 aes_encrypt_key(vars->wiredCryptKey,
1947 kIOHibernateAESKeySize,
1948 &cryptvars->ctx.encrypt);
1949
1950 bcopy(&first_iv[0], &cryptvars->aes_iv[0], AES_BLOCK_SIZE);
1951 bzero(&vars->wiredCryptKey[0], sizeof(vars->wiredCryptKey));
1952 bzero(&vars->cryptKey[0], sizeof(vars->cryptKey));
6d2010ae
A		 1953
1954 local_cryptvars = cryptvars;
3a60a9f5 1955 }
2d21ac55 1956#endif /* CRYPTO */
3a60a9f5
A		 1957
1958 hibernate_setup_for_wake();
1959
1960 hibernate_page_list_setall(vars->page_list,
1961 vars->page_list_wired,
6d2010ae 1962 vars->page_list_pal,
3a60a9f5
A		 1963 &pageCount);
1964
1965 HIBLOG("hibernate_page_list_setall found pageCount %d\n", pageCount);
1966
1967 fileExtents = (IOPolledFileExtent *) vars->fileExtents->getBytesNoCopy();
1968
1969#if 0
1970 count = vars->fileExtents->getLength() / sizeof(IOPolledFileExtent);
1971 for (page = 0; page < count; page++)
1972 {
1973 HIBLOG("fileExtents[%d] %qx, %qx (%qx)\n", page,
1974 fileExtents[page].start, fileExtents[page].length,
1975 fileExtents[page].start + fileExtents[page].length);
1976 }
1977#endif
1978
0b4c1975 1979 needEncrypt = (0 != (kIOHibernateModeEncrypt & gIOHibernateMode));
3a60a9f5 1980 AbsoluteTime_to_scalar(&compTime) = 0;
6d2010ae 1981 compBytes = 0;
3a60a9f5
A		 1982
1983 clock_get_uptime(&allTime);
b0d623f7
A		 1984 IOService::getPMRootDomain()->pmStatsRecordEvent(
1985 kIOPMStatsHibernateImageWrite | kIOPMStatsEventStartFlag, allTime);
3a60a9f5
A		 1986
1987 do
1988 {
1989 compressedSize = 0;
1990 uncompressedSize = 0;
3a60a9f5
A		 1991
1992 IOPolledFileSeek(vars->fileVars, sizeof(IOHibernateImageHeader));
1993
1994 HIBLOG("IOHibernatePollerOpen, ml_get_interrupts_enabled %d\n",
1995 ml_get_interrupts_enabled());
1996 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledBeforeSleepState, vars->ioBuffer);
1997 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
1998 pollerOpen = (kIOReturnSuccess == err);
1999 if (!pollerOpen)
2000 break;
2001
2002 // copy file block extent list if larger than header
2003
2004 count = vars->fileExtents->getLength();
2005 if (count > sizeof(header->fileExtentMap))
2006 {
2007 count -= sizeof(header->fileExtentMap);
2008 err = IOPolledFileWrite(vars->fileVars,
2009 ((uint8_t *) &fileExtents[0]) + sizeof(header->fileExtentMap), count, cryptvars);
2010 if (kIOReturnSuccess != err)
2011 break;
2012 }
2013
b0d623f7
A		 2014 uintptr_t hibernateBase;
2015 uintptr_t hibernateEnd;
2016
6d2010ae 2017 hibernateBase = HIB_BASE; /* Defined in PAL headers */
b0d623f7
A		 2018
2019 hibernateEnd = (sectHIBB + sectSizeHIB);
6d2010ae 2020
3a60a9f5 2021 // copy out restore1 code
6d2010ae
A		 2022
2023 for (count = 0;
2024 (phys64 = vars->handoffBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
2025 count += segLen)
2026 {
2027 for (pagesDone = 0; pagesDone < atop_32(segLen); pagesDone++)
2028 {
2029 gIOHibernateHandoffPages[atop_32(count) + pagesDone] = atop_64(phys64) + pagesDone;
2030 }
2031 }
2032
2033 page = atop_32(kvtophys(hibernateBase));
2034 count = atop_32(round_page(hibernateEnd) - hibernateBase);
2035 header->restore1CodePhysPage = page;
2036 header->restore1CodeVirt = hibernateBase;
3a60a9f5 2037 header->restore1PageCount = count;
b0d623f7
A		 2038 header->restore1CodeOffset = ((uintptr_t) &hibernate_machine_entrypoint) - hibernateBase;
2039 header->restore1StackOffset = ((uintptr_t) &gIOHibernateRestoreStackEnd[0]) - 64 - hibernateBase;
3a60a9f5
A		 2040
2041 // sum __HIB sect, with zeros for the stack
b0d623f7 2042 src = (uint8_t *) trunc_page(hibernateBase);
3a60a9f5
A		 2043 for (page = 0; page < count; page++)
2044 {
2045 if ((src < &gIOHibernateRestoreStack[0]) || (src >= &gIOHibernateRestoreStackEnd[0]))
6d2010ae 2046 restore1Sum += hibernate_sum_page(src, header->restore1CodeVirt + page);
3a60a9f5 2047 else
0b4c1975 2048 restore1Sum += 0x00000000;
3a60a9f5
A		 2049 src += page_size;
2050 }
2051 sum1 = restore1Sum;
2052
2053 // write the __HIB sect, with zeros for the stack
2054
b0d623f7
A		 2055 src = (uint8_t *) trunc_page(hibernateBase);
2056 count = ((uintptr_t) &gIOHibernateRestoreStack[0]) - trunc_page(hibernateBase);
3a60a9f5
A		 2057 if (count)
2058 {
2059 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
2060 if (kIOReturnSuccess != err)
2061 break;
2062 }
2063 err = IOPolledFileWrite(vars->fileVars,
2064 (uint8_t *) 0,
2065 &gIOHibernateRestoreStackEnd[0] - &gIOHibernateRestoreStack[0],
2066 cryptvars);
2067 if (kIOReturnSuccess != err)
2068 break;
2069 src = &gIOHibernateRestoreStackEnd[0];
b0d623f7 2070 count = round_page(hibernateEnd) - ((uintptr_t) src);
3a60a9f5
A		 2071 if (count)
2072 {
2073 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
2074 if (kIOReturnSuccess != err)
2075 break;
2076 }
2077
2078 // write the preview buffer
2079
2d21ac55 2080 if (vars->previewBuffer)
3a60a9f5
A		 2081 {
2082 ppnum = 0;
2083 count = 0;
2084 do
2085 {
b0d623f7 2086 phys64 = vars->previewBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone);
3a60a9f5
A		 2087 pageAndCount[0] = atop_64(phys64);
2088 pageAndCount[1] = atop_32(segLen);
2089 err = IOPolledFileWrite(vars->fileVars,
2090 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
2091 cryptvars);
2092 if (kIOReturnSuccess != err)
2093 break;
2094 count += segLen;
2095 ppnum += sizeof(pageAndCount);
2096 }
2097 while (phys64);
2098 if (kIOReturnSuccess != err)
2099 break;
2100
b0d623f7 2101 src = (uint8_t *) vars->previewBuffer->getPhysicalSegment(0, NULL, _kIOMemorySourceSegment);
2d21ac55 2102 count = vars->previewBuffer->getLength();
3a60a9f5
A		 2103
2104 header->previewPageListSize = ppnum;
2105 header->previewSize = count + ppnum;
2106
2107 for (page = 0; page < count; page += page_size)
0b4c1975
A		 2108 {
2109 phys64 = vars->previewBuffer->getPhysicalSegment(page, NULL, kIOMemoryMapperNone);
2110 sum1 += hibernate_sum_page(src + page, atop_64(phys64));
2111 }
3a60a9f5
A		 2112 err = IOPolledFileWrite(vars->fileVars, src, count, cryptvars);
2113 if (kIOReturnSuccess != err)
2114 break;
2115 }
2116
2117 // mark areas for no save
2118
2119 for (count = 0;
b0d623f7 2120 (phys64 = vars->ioBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 2121 count += segLen)
2122 {
2123 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
2124 atop_64(phys64), atop_32(segLen),
2125 kIOHibernatePageStateFree);
2126 pageCount -= atop_32(segLen);
2127 }
2128
2129 for (count = 0;
b0d623f7 2130 (phys64 = vars->srcBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 2131 count += segLen)
2132 {
2133 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
2134 atop_64(phys64), atop_32(segLen),
2135 kIOHibernatePageStateFree);
2136 pageCount -= atop_32(segLen);
2137 }
2138
2139 // copy out bitmap of pages available for trashing during restore
2140
2141 bitmap_size = vars->page_list_wired->list_size;
2142 src = (uint8_t *) vars->page_list_wired;
2143 err = IOPolledFileWrite(vars->fileVars, src, bitmap_size, cryptvars);
2144 if (kIOReturnSuccess != err)
2145 break;
2146
2147 // mark more areas for no save, but these are not available
2148 // for trashing during restore
0c530ab8
A		 2149
2150 hibernate_page_list_set_volatile(vars->page_list, vars->page_list_wired, &pageCount);
3a60a9f5 2151
6d2010ae
A		 2152
2153 page = atop_32(KERNEL_IMAGE_TO_PHYS(hibernateBase));
2154 count = atop_32(round_page(KERNEL_IMAGE_TO_PHYS(hibernateEnd))) - page;
3a60a9f5
A		 2155 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
2156 page, count,
2157 kIOHibernatePageStateFree);
2158 pageCount -= count;
3a60a9f5
A		 2159
2160 if (vars->previewBuffer) for (count = 0;
b0d623f7 2161 (phys64 = vars->previewBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
3a60a9f5
A		 2162 count += segLen)
2163 {
2164 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
2165 atop_64(phys64), atop_32(segLen),
2166 kIOHibernatePageStateFree);
2167 pageCount -= atop_32(segLen);
2168 }
2169
6d2010ae
A		 2170 for (count = 0;
2171 (phys64 = vars->handoffBuffer->getPhysicalSegment(count, &segLen, kIOMemoryMapperNone));
2172 count += segLen)
2173 {
2174 hibernate_set_page_state(vars->page_list, vars->page_list_wired,
2175 atop_64(phys64), atop_32(segLen),
2176 kIOHibernatePageStateFree);
2177 pageCount -= atop_32(segLen);
2178 }
2179
2180 (void)hibernate_pal_callback;
2181
3a60a9f5
A		 2182 src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();
2183
6d2010ae
A		 2184 pagesDone = 0;
2185 lastBlob = 0;
3a60a9f5
A		 2186
2187 HIBLOG("writing %d pages\n", pageCount);
2188
0b4c1975
A		 2189 enum
2190 // pageType
2191 {
2192 kWired = 0x02,
2193 kEncrypt = 0x01,
2194 kWiredEncrypt = kWired | kEncrypt,
2195 kWiredClear = kWired,
2196 kUnwiredEncrypt = kEncrypt
2197 };
2198
2199 for (pageType = kWiredEncrypt; pageType >= kUnwiredEncrypt; pageType--)
3a60a9f5 2200 {
0b4c1975 2201 if (needEncrypt && (kEncrypt & pageType))
3a60a9f5 2202 {
7ddcb079 2203 vars->fileVars->encryptStart = (vars->fileVars->position & ~(((uint64_t)AES_BLOCK_SIZE) - 1));
0b4c1975
A		 2204 vars->fileVars->encryptEnd = UINT64_MAX;
2205 HIBLOG("encryptStart %qx\n", vars->fileVars->encryptStart);
2206
2207 if (kUnwiredEncrypt == pageType)
3a60a9f5 2208 {
0b4c1975
A		 2209 // start unwired image
2210 bcopy(&cryptvars->aes_iv[0],
2211 &gIOHibernateCryptWakeContext.aes_iv[0],
2212 sizeof(cryptvars->aes_iv));
2213 cryptvars = &gIOHibernateCryptWakeContext;
3a60a9f5 2214 }
0b4c1975
A		 2215 }
2216 for (iterDone = false, ppnum = 0; !iterDone; )
2217 {
2218 count = hibernate_page_list_iterate((kWired & pageType)
2219 ? vars->page_list_wired : vars->page_list,
2220 &ppnum);
2221// kprintf("[%d](%x : %x)\n", pageType, ppnum, count);
2222 iterDone = !count;
3a60a9f5 2223
0b4c1975 2224 if (count && (kWired & pageType) && needEncrypt)
3a60a9f5 2225 {
0b4c1975
A		 2226 uint32_t checkIndex;
2227 for (checkIndex = 0;
2228 (checkIndex < count)
2229 && (((kEncrypt & pageType) == 0) == no_encrypt_page(ppnum + checkIndex));
2230 checkIndex++)
2231 {}
2232 if (!checkIndex)
2233 {
2234 ppnum++;
2235 continue;
2236 }
2237 count = checkIndex;
3a60a9f5
A		 2238 }
2239
0b4c1975
A		 2240 switch (pageType)
2241 {
2242 case kWiredEncrypt: wired_pages_encrypted += count; break;
2243 case kWiredClear: wired_pages_clear += count; break;
2244 case kUnwiredEncrypt: dirty_pages_encrypted += count; break;
2245 }
2246
2247 if (iterDone && (kWiredEncrypt == pageType)) {/* not yet end of wired list */}
3a60a9f5 2248 else
3a60a9f5 2249 {
0b4c1975
A		 2250 pageAndCount[0] = ppnum;
2251 pageAndCount[1] = count;
2252 err = IOPolledFileWrite(vars->fileVars,
2253 (const uint8_t *) &pageAndCount, sizeof(pageAndCount),
2254 cryptvars);
2255 if (kIOReturnSuccess != err)
2256 break;
3a60a9f5 2257 }
0b4c1975
A		 2258
2259 for (page = ppnum; page < (ppnum + count); page++)
3a60a9f5 2260 {
0b4c1975
A		 2261 err = IOMemoryDescriptorWriteFromPhysical(vars->srcBuffer, 0, ptoa_64(page), page_size);
2262 if (err)
2263 {
2264 HIBLOG("IOMemoryDescriptorWriteFromPhysical %d [%ld] %x\n", __LINE__, (long)page, err);
2265 break;
2266 }
2267
2268 sum = hibernate_sum_page(src, page);
2269 if (kWired & pageType)
2270 sum1 += sum;
2271 else
2272 sum2 += sum;
2273
2274 clock_get_uptime(&startTime);
2275
2276 pageCompressedSize = WKdm_compress ((WK_word*) src, (WK_word*) (src + page_size), PAGE_SIZE_IN_WORDS);
2277
3a60a9f5 2278 clock_get_uptime(&endTime);
0b4c1975
A		 2279 ADD_ABSOLUTETIME(&compTime, &endTime);
2280 SUB_ABSOLUTETIME(&compTime, &startTime);
2281 compBytes += page_size;
2282
2283 if (kIOHibernateModeEncrypt & gIOHibernateMode)
2284 pageCompressedSize = (pageCompressedSize + AES_BLOCK_SIZE - 1) & ~(AES_BLOCK_SIZE - 1);
2285
2286 if (pageCompressedSize > page_size)
3a60a9f5 2287 {
0b4c1975
A		 2288// HIBLOG("------------lose: %d\n", pageCompressedSize);
2289 pageCompressedSize = page_size;
2290 }
2291
2292 if (pageCompressedSize != page_size)
2293 data = (src + page_size);
2294 else
2295 data = src;
2296
2297 tag = pageCompressedSize | kIOHibernateTagSignature;
2298 err = IOPolledFileWrite(vars->fileVars, (const uint8_t *) &tag, sizeof(tag), cryptvars);
2299 if (kIOReturnSuccess != err)
2300 break;
2301
2302 err = IOPolledFileWrite(vars->fileVars, data, (pageCompressedSize + 3) & ~3, cryptvars);
2303 if (kIOReturnSuccess != err)
2304 break;
2305
2306 compressedSize += pageCompressedSize;
2307 if (pageCompressedSize)
2308 uncompressedSize += page_size;
2309 pagesDone++;
2310
2311 if (vars->consoleMapping && (0 == (1023 & pagesDone)))
2312 {
2313 blob = ((pagesDone * kIOHibernateProgressCount) / pageCount);
2314 if (blob != lastBlob)
2315 {
2316 ProgressUpdate(gIOHibernateGraphicsInfo, vars->consoleMapping, lastBlob, blob);
2317 lastBlob = blob;
2318 }
2319 }
2320 if (0 == (8191 & pagesDone))
2321 {
2322 clock_get_uptime(&endTime);
2323 SUB_ABSOLUTETIME(&endTime, &allTime);
2324 absolutetime_to_nanoseconds(endTime, &nsec);
2325 progressStamp = nsec / 750000000ULL;
2326 if (progressStamp != lastProgressStamp)
2327 {
2328 lastProgressStamp = progressStamp;
2329 HIBPRINT("pages %d (%d%%)\n", pagesDone, (100 * pagesDone) / pageCount);
2330 }
3a60a9f5
A		 2331 }
2332 }
0b4c1975
A		 2333 if (kIOReturnSuccess != err)
2334 break;
2335 ppnum = page;
3a60a9f5 2336 }
0b4c1975 2337
3a60a9f5
A		 2338 if (kIOReturnSuccess != err)
2339 break;
0b4c1975
A		 2340
2341 if ((kEncrypt & pageType))
3a60a9f5 2342 {
7ddcb079 2343 vars->fileVars->encryptEnd = ((vars->fileVars->position + 511) & ~511ULL);
0b4c1975
A		 2344 HIBLOG("encryptEnd %qx\n", vars->fileVars->encryptEnd);
2345 }
2346
2347 if (kWiredEncrypt != pageType)
2348 {
2349 // end of image1/2 - fill to next block
3a60a9f5
A		 2350 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
2351 if (kIOReturnSuccess != err)
2352 break;
0b4c1975
A		 2353 }
2354 if (kWiredClear == pageType)
2355 {
7ddcb079
A		 2356 // enlarge wired image for test
2357// err = IOPolledFileWrite(vars->fileVars, 0, 0x60000000, cryptvars);
2358
0b4c1975
A		 2359 // end wired image
2360 header->encryptStart = vars->fileVars->encryptStart;
2361 header->encryptEnd = vars->fileVars->encryptEnd;
3a60a9f5 2362 image1Size = vars->fileVars->position;
7ddcb079 2363 HIBLOG("image1Size 0x%qx, encryptStart1 0x%qx, End1 0x%qx\n",
0b4c1975 2364 image1Size, header->encryptStart, header->encryptEnd);
3a60a9f5
A		 2365 }
2366 }
3a60a9f5
A		 2367 if (kIOReturnSuccess != err)
2368 break;
2369
2370 // Header:
2371
2372 header->imageSize = vars->fileVars->position;
2373 header->image1Size = image1Size;
2374 header->bitmapSize = bitmap_size;
2375 header->pageCount = pageCount;
3a60a9f5
A		 2376
2377 header->restore1Sum = restore1Sum;
2378 header->image1Sum = sum1;
2379 header->image2Sum = sum2;
2380
2381 count = vars->fileExtents->getLength();
2382 if (count > sizeof(header->fileExtentMap))
2383 {
2384 header->fileExtentMapSize = count;
2385 count = sizeof(header->fileExtentMap);
2386 }
2387 else
2388 header->fileExtentMapSize = sizeof(header->fileExtentMap);
2389 bcopy(&fileExtents[0], &header->fileExtentMap[0], count);
0b4c1975 2390
6d2010ae
A		 2391 header->deviceBase = vars->fileVars->block0;
2392
3a60a9f5
A		 2393 IOPolledFileSeek(vars->fileVars, 0);
2394 err = IOPolledFileWrite(vars->fileVars,
2395 (uint8_t *) header, sizeof(IOHibernateImageHeader),
2396 cryptvars);
2397 if (kIOReturnSuccess != err)
2398 break;
2399 err = IOPolledFileWrite(vars->fileVars, 0, 0, cryptvars);
2400 if (kIOReturnSuccess != err)
2401 break;
2d21ac55 2402 err = IOHibernatePollerIODone(vars->fileVars, true);
3a60a9f5
A		 2403 if (kIOReturnSuccess != err)
2404 break;
2405 }
2406 while (false);
2407
2408 clock_get_uptime(&endTime);
b0d623f7
A		 2409
2410 IOService::getPMRootDomain()->pmStatsRecordEvent(
2411 kIOPMStatsHibernateImageWrite | kIOPMStatsEventStopFlag, endTime);
2412
3a60a9f5
A		 2413 SUB_ABSOLUTETIME(&endTime, &allTime);
2414 absolutetime_to_nanoseconds(endTime, &nsec);
2415 HIBLOG("all time: %qd ms, ",
2416 nsec / 1000000ULL);
2417
2418 absolutetime_to_nanoseconds(compTime, &nsec);
6d2010ae
A		 2419 HIBLOG("comp bytes: %qd time: %qd ms %qd Mb/s, ",
2420 compBytes,
2421 nsec / 1000000ULL,
2422 nsec ? (((compBytes * 1000000000ULL) / 1024 / 1024) / nsec) : 0);
3a60a9f5 2423
6d2010ae
A		 2424 absolutetime_to_nanoseconds(vars->fileVars->cryptTime, &nsec);
2425 HIBLOG("crypt bytes: %qd time: %qd ms %qd Mb/s, ",
2426 vars->fileVars->cryptBytes,
2427 nsec / 1000000ULL,
2428 nsec ? (((vars->fileVars->cryptBytes * 1000000000ULL) / 1024 / 1024) / nsec) : 0);
3a60a9f5
A		 2429
2430 HIBLOG("\nimage %qd, uncompressed %qd (%d), compressed %qd (%d%%), sum1 %x, sum2 %x\n",
2431 header->imageSize,
2432 uncompressedSize, atop_32(uncompressedSize), compressedSize,
8f6c56a5 2433 uncompressedSize ? ((int) ((compressedSize * 100ULL) / uncompressedSize)) : 0,
3a60a9f5
A		 2434 sum1, sum2);
2435
0b4c1975
A		 2436 HIBLOG("wired_pages_encrypted %d, wired_pages_clear %d, dirty_pages_encrypted %d\n",
2437 wired_pages_encrypted, wired_pages_clear, dirty_pages_encrypted);
2438
c910b4d9
A		 2439 if (vars->fileVars->io)
2440 (void) IOHibernatePollerIODone(vars->fileVars, false);
2441
3a60a9f5
A		 2442 if (pollerOpen)
2443 IOHibernatePollerClose(vars->fileVars, kIOPolledBeforeSleepState);
2444
2d21ac55
A		 2445 if (vars->consoleMapping)
2446 ProgressUpdate(gIOHibernateGraphicsInfo,
2447 vars->consoleMapping, 0, kIOHibernateProgressCount);
2448
3a60a9f5
A		 2449 HIBLOG("hibernate_write_image done(%x)\n", err);
2450
2451 // should we come back via regular wake, set the state in memory.
2452 gIOHibernateState = kIOHibernateStateInactive;
2453
0b4c1975
A		 2454 KERNEL_DEBUG_CONSTANT(IOKDBG_CODE(DBG_HIBERNATE, 1) | DBG_FUNC_END,
2455 wired_pages_encrypted, wired_pages_clear, dirty_pages_encrypted, 0, 0);
2456
2d21ac55
A		 2457 if (kIOReturnSuccess == err)
2458 {
2459 if (kIOHibernateModeSleep & gIOHibernateMode)
2460 {
2461 return (kIOHibernatePostWriteSleep);
2462 }
2463 else if(kIOHibernateModeRestart & gIOHibernateMode)
2464 {
2465 return (kIOHibernatePostWriteRestart);
2466 }
2467 else
2468 {
2469 /* by default, power down */
2470 return (kIOHibernatePostWriteHalt);
2471 }
2472 }
2473 else if (kIOReturnAborted == err)
2474 {
2475 return (kIOHibernatePostWriteWake);
2476 }
3a60a9f5 2477 else
3a60a9f5 2478 {
2d21ac55
A		 2479 /* on error, sleep */
2480 return (kIOHibernatePostWriteSleep);
3a60a9f5
A		 2481 }
2482}
2483
2484/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
2485
2486extern "C" void
2487hibernate_machine_init(void)
2488{
2489 IOReturn err;
2490 uint32_t sum;
2491 uint32_t pagesDone;
0b4c1975 2492 uint32_t pagesRead = 0;
6d2010ae 2493 AbsoluteTime startTime, compTime;
3a60a9f5 2494 AbsoluteTime allTime, endTime;
6d2010ae 2495 uint64_t compBytes;
3a60a9f5
A		 2496 uint64_t nsec;
2497 uint32_t lastProgressStamp = 0;
2498 uint32_t progressStamp;
2499 uint64_t progressZeroPosition = 0;
2500 uint32_t blob, lastBlob = (uint32_t) -1L;
2501 hibernate_cryptvars_t * cryptvars = 0;
2502
2503 IOHibernateVars * vars = &gIOHibernateVars;
2504
2505 if (!vars->fileVars || !vars->fileVars->pollers || !vars->fileExtents)
2506 return;
2507
3a60a9f5
A		 2508 sum = gIOHibernateCurrentHeader->actualImage1Sum;
2509 pagesDone = gIOHibernateCurrentHeader->actualUncompressedPages;
2510
2511 HIBLOG("hibernate_machine_init: state %d, image pages %d, sum was %x, image1Size %qx, conflictCount %d, nextFree %x\n",
2512 gIOHibernateState, pagesDone, sum, gIOHibernateCurrentHeader->image1Size,
2513 gIOHibernateCurrentHeader->conflictCount, gIOHibernateCurrentHeader->nextFree);
2514
2515 if (kIOHibernateStateWakingFromHibernate != gIOHibernateState)
2516 {
2517 HIBLOG("regular wake\n");
2518 return;
2519 }
2520
2521 HIBPRINT("diag %x %x %x %x\n",
2522 gIOHibernateCurrentHeader->diag[0], gIOHibernateCurrentHeader->diag[1],
6d2010ae 2523 gIOHibernateCurrentHeader->diag[2], gIOHibernateCurrentHeader->diag[3]);
3a60a9f5 2524
0b4c1975 2525 HIBPRINT("video %x %d %d %d status %x\n",
3a60a9f5 2526 gIOHibernateGraphicsInfo->physicalAddress, gIOHibernateGraphicsInfo->depth,
0b4c1975 2527 gIOHibernateGraphicsInfo->width, gIOHibernateGraphicsInfo->height, gIOHibernateGraphicsInfo->gfxStatus);
3a60a9f5 2528
0c530ab8
A		 2529 if ((kIOHibernateModeDiscardCleanActive | kIOHibernateModeDiscardCleanInactive) & gIOHibernateMode)
2530 hibernate_page_list_discard(vars->page_list);
2531
2532 boot_args *args = (boot_args *) PE_state.bootArgs;
2533
6d2010ae
A		 2534 cryptvars = (kIOHibernateModeEncrypt & gIOHibernateMode) ? &gIOHibernateCryptWakeContext : 0;
2535
2536 if (gIOHibernateCurrentHeader->handoffPageCount > gIOHibernateHandoffPageCount)
2537 panic("handoff overflow");
2538
2539 IOHibernateHandoff * handoff;
2540 bool done = false;
2541 bool foundCryptData = false;
2542
2543 for (handoff = (IOHibernateHandoff *) vars->handoffBuffer->getBytesNoCopy();
2544 !done;
2545 handoff = (IOHibernateHandoff *) &handoff->data[handoff->bytecount])
2546 {
2547// HIBPRINT("handoff %p, %x, %x\n", handoff, handoff->type, handoff->bytecount);
2548 uint8_t * data = &handoff->data[0];
2549 switch (handoff->type)
2550 {
2551 case kIOHibernateHandoffTypeEnd:
2552 done = true;
2553 break;
2554
2555 case kIOHibernateHandoffTypeGraphicsInfo:
2556 bcopy(data, gIOHibernateGraphicsInfo, sizeof(*gIOHibernateGraphicsInfo));
2557 break;
2558
2559 case kIOHibernateHandoffTypeCryptVars:
2560 if (cryptvars)
2561 {
2562 hibernate_cryptwakevars_t *
2563 wakevars = (hibernate_cryptwakevars_t *) &handoff->data[0];
2564 bcopy(&wakevars->aes_iv[0], &cryptvars->aes_iv[0], sizeof(cryptvars->aes_iv));
2565 }
2566 foundCryptData = true;
2567 bzero(data, handoff->bytecount);
2568 break;
2569
2570 case kIOHibernateHandoffTypeMemoryMap:
2571 hibernate_newruntime_map(data, handoff->bytecount,
2572 gIOHibernateCurrentHeader->systemTableOffset);
2573 break;
2574
2575 case kIOHibernateHandoffTypeDeviceTree:
2576 {
2577// DTEntry chosen = NULL;
2578// HIBPRINT("DTLookupEntry %d\n", DTLookupEntry((const DTEntry) data, "/chosen", &chosen));
2579 }
2580 break;
2581
2582 default:
2583 done = (kIOHibernateHandoffType != (handoff->type & 0xFFFF0000));
2584 break;
2585 }
2586 }
2587 if (cryptvars && !foundCryptData)
2588 panic("hibernate handoff");
2589
0c530ab8
A		 2590 if (vars->videoMapping
2591 && gIOHibernateGraphicsInfo->physicalAddress
2592 && (args->Video.v_baseAddr == gIOHibernateGraphicsInfo->physicalAddress))
3a60a9f5
A		 2593 {
2594 vars->videoMapSize = round_page(gIOHibernateGraphicsInfo->height
2595 * gIOHibernateGraphicsInfo->rowBytes);
2596 IOMapPages(kernel_map,
2597 vars->videoMapping, gIOHibernateGraphicsInfo->physicalAddress,
2598 vars->videoMapSize, kIOMapInhibitCache );
2599 }
2600
0c530ab8 2601 uint8_t * src = (uint8_t *) vars->srcBuffer->getBytesNoCopy();
3a60a9f5
A		 2602 uint32_t decoOffset;
2603
2604 clock_get_uptime(&allTime);
6d2010ae
A		 2605 AbsoluteTime_to_scalar(&compTime) = 0;
2606 compBytes = 0;
3a60a9f5
A		 2607
2608 HIBLOG("IOHibernatePollerOpen(), ml_get_interrupts_enabled %d\n", ml_get_interrupts_enabled());
2609 err = IOHibernatePollerOpen(vars->fileVars, kIOPolledAfterSleepState, 0);
2610 HIBLOG("IOHibernatePollerOpen(%x)\n", err);
2611
2612 if (gIOHibernateCurrentHeader->previewSize)
2613 progressZeroPosition = gIOHibernateCurrentHeader->previewSize
2614 + gIOHibernateCurrentHeader->fileExtentMapSize
2615 - sizeof(gIOHibernateCurrentHeader->fileExtentMap)
2616 + ptoa_64(gIOHibernateCurrentHeader->restore1PageCount);
2617
2618 IOPolledFileSeek(vars->fileVars, gIOHibernateCurrentHeader->image1Size);
2619
2d21ac55 2620 if (vars->videoMapSize)
3a60a9f5
A		 2621 {
2622 lastBlob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
2623 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
2624 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, 0, lastBlob);
2625 }
2626
3a60a9f5
A		 2627 // kick off the read ahead
2628 vars->fileVars->io = false;
2629 vars->fileVars->bufferHalf = 0;
2630 vars->fileVars->bufferLimit = 0;
2631 vars->fileVars->lastRead = 0;
6d2010ae 2632 vars->fileVars->readEnd = gIOHibernateCurrentHeader->imageSize;
3a60a9f5 2633 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
6d2010ae
A		 2634 vars->fileVars->cryptBytes = 0;
2635 AbsoluteTime_to_scalar(&vars->fileVars->cryptTime) = 0;
3a60a9f5 2636
6d2010ae 2637 err = IOPolledFileRead(vars->fileVars, 0, 0, cryptvars);
3a60a9f5
A		 2638 vars->fileVars->bufferOffset = vars->fileVars->bufferLimit;
2639 // --
2640
2641 HIBLOG("hibernate_machine_init reading\n");
2642
2643 uint32_t * header = (uint32_t *) src;
2644 sum = 0;
2645
6d2010ae 2646 while (kIOReturnSuccess == err)
3a60a9f5
A		 2647 {
2648 unsigned int count;
2649 unsigned int page;
2650 uint32_t tag;
2651 vm_offset_t ppnum, compressedSize;
2652
36401178
A		 2653 err = IOPolledFileRead(vars->fileVars, src, 8, cryptvars);
2654 if (kIOReturnSuccess != err)
2655 break;
3a60a9f5
A		 2656
2657 ppnum = header[0];
2658 count = header[1];
2659
2660// HIBPRINT("(%x, %x)\n", ppnum, count);
2661
2662 if (!count)
2663 break;
2664
2665 for (page = 0; page < count; page++)
2666 {
36401178
A		 2667 err = IOPolledFileRead(vars->fileVars, (uint8_t *) &tag, 4, cryptvars);
2668 if (kIOReturnSuccess != err)
2669 break;
3a60a9f5
A		 2670
2671 compressedSize = kIOHibernateTagLength & tag;
36401178
A		 2672 if (kIOHibernateTagSignature != (tag & ~kIOHibernateTagLength))
2673 {
2674 err = kIOReturnIPCError;
2675 break;
2676 }
2677
3a60a9f5
A		 2678 if (!compressedSize)
2679 {
2680 ppnum++;
2681 pagesDone++;
2682 continue;
2683 }
2684
36401178
A		 2685 err = IOPolledFileRead(vars->fileVars, src, (compressedSize + 3) & ~3, cryptvars);
2686 if (kIOReturnSuccess != err)
2687 break;
2688
2689 if (compressedSize < page_size)
3a60a9f5
A		 2690 {
2691 decoOffset = page_size;
6d2010ae
A		 2692
2693 clock_get_uptime(&startTime);
3a60a9f5 2694 WKdm_decompress((WK_word*) src, (WK_word*) (src + decoOffset), PAGE_SIZE_IN_WORDS);
6d2010ae
A		 2695 clock_get_uptime(&endTime);
2696 ADD_ABSOLUTETIME(&compTime, &endTime);
2697 SUB_ABSOLUTETIME(&compTime, &startTime);
2698
2699 compBytes += page_size;
3a60a9f5
A		 2700 }
2701 else
2702 decoOffset = 0;
2703
0b4c1975 2704 sum += hibernate_sum_page((src + decoOffset), ppnum);
3a60a9f5
A		 2705
2706 err = IOMemoryDescriptorReadToPhysical(vars->srcBuffer, decoOffset, ptoa_64(ppnum), page_size);
2707 if (err)
36401178 2708 {
b0d623f7 2709 HIBLOG("IOMemoryDescriptorReadToPhysical [%ld] %x\n", (long)ppnum, err);
36401178
A		 2710 break;
2711 }
3a60a9f5
A		 2712
2713 ppnum++;
2714 pagesDone++;
0b4c1975 2715 pagesRead++;
3a60a9f5 2716
2d21ac55 2717 if (vars->videoMapSize && (0 == (1023 & pagesDone)))
3a60a9f5
A		 2718 {
2719 blob = ((vars->fileVars->position - progressZeroPosition) * kIOHibernateProgressCount)
2720 / (gIOHibernateCurrentHeader->imageSize - progressZeroPosition);
2721 if (blob != lastBlob)
2722 {
2723 ProgressUpdate(gIOHibernateGraphicsInfo, (uint8_t *) vars->videoMapping, lastBlob, blob);
2724 lastBlob = blob;
2725 }
2726 }
2727
2728 if (0 == (8191 & pagesDone))
2729 {
2730 clock_get_uptime(&endTime);
2731 SUB_ABSOLUTETIME(&endTime, &allTime);
2732 absolutetime_to_nanoseconds(endTime, &nsec);
2733 progressStamp = nsec / 750000000ULL;
2734 if (progressStamp != lastProgressStamp)
2735 {
2736 lastProgressStamp = progressStamp;
2737 HIBPRINT("pages %d (%d%%)\n", pagesDone,
2738 (100 * pagesDone) / gIOHibernateCurrentHeader->pageCount);
2739 }
2740 }
2741 }
2742 }
7ddcb079
A		 2743 if ((kIOReturnSuccess == err) && (pagesDone == gIOHibernateCurrentHeader->actualUncompressedPages))
2744 err = kIOReturnLockedRead;
3a60a9f5 2745
36401178
A		 2746 if (kIOReturnSuccess != err)
2747 panic("Hibernate restore error %x", err);
2748
3a60a9f5
A		 2749 gIOHibernateCurrentHeader->actualImage2Sum = sum;
2750
2751 if (vars->fileVars->io)
2d21ac55 2752 (void) IOHibernatePollerIODone(vars->fileVars, false);
3a60a9f5
A		 2753
2754 err = IOHibernatePollerClose(vars->fileVars, kIOPolledAfterSleepState);
2755
2d21ac55 2756 if (vars->videoMapSize)
3a60a9f5
A		 2757 ProgressUpdate(gIOHibernateGraphicsInfo,
2758 (uint8_t *) vars->videoMapping, 0, kIOHibernateProgressCount);
2759
2760 clock_get_uptime(&endTime);
b0d623f7
A		 2761
2762 IOService::getPMRootDomain()->pmStatsRecordEvent(
2763 kIOPMStatsHibernateImageRead | kIOPMStatsEventStartFlag, allTime);
2764 IOService::getPMRootDomain()->pmStatsRecordEvent(
2765 kIOPMStatsHibernateImageRead | kIOPMStatsEventStopFlag, endTime);
2766
3a60a9f5
A		 2767 SUB_ABSOLUTETIME(&endTime, &allTime);
2768 absolutetime_to_nanoseconds(endTime, &nsec);
2769
6d2010ae 2770 HIBLOG("hibernate_machine_init pagesDone %d sum2 %x, time: %qd ms, ",
3a60a9f5 2771 pagesDone, sum, nsec / 1000000ULL);
6d2010ae
A		 2772
2773 absolutetime_to_nanoseconds(compTime, &nsec);
2774 HIBLOG("comp bytes: %qd time: %qd ms %qd Mb/s, ",
2775 compBytes,
2776 nsec / 1000000ULL,
2777 nsec ? (((compBytes * 1000000000ULL) / 1024 / 1024) / nsec) : 0);
2778
2779 absolutetime_to_nanoseconds(vars->fileVars->cryptTime, &nsec);
2780 HIBLOG("crypt bytes: %qd time: %qd ms %qd Mb/s\n",
2781 vars->fileVars->cryptBytes,
2782 nsec / 1000000ULL,
2783 nsec ? (((vars->fileVars->cryptBytes * 1000000000ULL) / 1024 / 1024) / nsec) : 0);
2784
2785 KERNEL_DEBUG_CONSTANT(IOKDBG_CODE(DBG_HIBERNATE, 2) | DBG_FUNC_NONE, pagesRead, pagesDone, 0, 0, 0);
3a60a9f5
A		 2786}
2787
2788/* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * */
mirror of XNU